Research Report

The Use of a Tactile-Vision Sensory Substitution System as an Augmentative Tool for Individuals with Visual Impairments

Michael D. Williams, Christopher T. Ray, Jennifer Griffith, and William De l'Aune

We acknowledge the contributions of Ms. Tina Calhoun for her day-to-day efforts as the clinical coordinator of the project, Dr. Bruce Blasch for his insights into the planning of the project, and Dr. Aimee Arnoldussen for her assistance with the technical execution of the study. Funding for this study was provided, in part, by WiCab, Inc.

The promise of novel technological strategies and solutions to assist persons with visual impairments (that is, those who are blind or have low vision) is frequently discussed and held to be widely beneficial in countless applications and daily activities. One such approach involving a tactile-vision sensory substitution modality as a mechanism to compensate for vision loss likewise holds such promise. It has been suggested that the use of a sensory substitution modality is an obvious and historically used mechanism of compensation for visual impairment and can be readily pointed to in a number of rudimentary, yet highly effective, strategies that are in use.

Perhaps the most established and successful example of a sensory substitution strategy that targets persons who are visually impaired is the use of braille. Information that is typically acquired visually through reading is, instead, acquired through the fingertips and interpreted in the brain (Bach-y-Rita & Kercel, 2003).

Another obvious sensory substitution approach that is commonly used to provide additional environmental information to travelers who are visually impaired is the use of a long cane (Bach-y-Rita & Kercel, 2003). Although a long cane has the potential to allow a variety of proprioceptive and kinesthetic information to be conveyed to a traveler (the material, slope, and elevation of the walking surface and the location and dimension of obstacles and openings along a path), it clearly meets the criteria of a sensory substitution strategy (Blasch, Wiener, & Welsh, 1997). A convincing argument can be made that adaptive strategies and techniques that are often taught to individuals with visual impairments involve a sensory substitution component.

Previous efforts to develop a portable device to aid individuals who are visually impaired to acquire printed materials and images started with the Optacon (Optical-to-Tactile-Converter), originally manufactured by Telesensory Systems. As described in an owner's manual from 1978, the Optacon was "a portable reading aid for the blind that operates by use of a miniature camera, an electronics unit and a tactile simulator array that converts printed words and images into tactile images that a blind person can feel with one finger" (Optacon Owner's Manual, 1978). Although the Optacon is no longer manufactured or commercially available, a loyal base of users of this device continue to tout the advantages and utility of this platform for acquiring written information.

The device examined in the study reported here is hardly novel from the point of view of being an augmentative tool to aid in the acquisition of environmental information, but does represent an intriguing and surprisingly intuitive interface for providing information to the user. The BrainPort vision device system--Wicab, BP-WAVE 2007--consists of a postage stamp-sized 25 × 25 electrotactile electrode array for the tongue (625 individual pinhead-sized points of contact arranged across the face of the array), a control box, a digital video camera, and a handheld controller for zoom and contrast inversion (see Figure 1).

BrainPort technology converts images that are captured through a digital camera and presents this information to the brain via electrical stimulation of the tongue through the electrotactile electrode array, augmenting normal sensory channels with this additional spatial information (Arnoldussen & Hogle, 2008). Although this particular sensory-substitution platform may at first appear unorthodox, numerous previous studies have supported the use of the tongue as a sensory channel for receiving input (Bach-y-Rita, 1967, 1972, 2005; Bach-y-Rita, Collins, Saunders, White, & Scadden, 1969; Bach-y-Rita, Kaczmarek, Tyler, & Garcia-Lara, 1998). Individuals who use this device have described the experience of the electrotactile electrode array as a feeling of effervescent bubbles mildly buzzing on the tongue. Images captured through the digital camera are converted into a two-dimensional representation and are displayed in a dynamic manner on the electrotactile electrode array in much the same way as a black- and-white image is presented on a computer monitor, with each of the 625 electrotactile points in the array serving as essentially a pixel on a screen.

Method

Participants

Thirty community-dwelling adults with self-reported total or near-total blindness were tested using the BrainPort device to determine its effectiveness in helping acquire accurate information about the shape of objects that were presented at various distances through electrotactile stimulation of the tongue. All the participants were blindfolded to ensure adequate control of any additional visual stimuli among those who retained some degree of residual vision. They were given extensive training in the use of the device, spending a minimum of three hours familiarizing themselves with the equipment. Training activities involved learning the physical components of the BrainPort, including the signal intensity control, the position of the camera, control of the camera zoom, and control of the stimulus contrast inversion. These activities also included orientation to perception of high-contrast two-dimensional objects or shapes while they held the objects in their hands, as well as identification of two-dimensional high-contrast objects or shapes (horizontal, vertical, and oblique lines; circles; and rings) that were placed against a dark backdrop. After the training protocol, each participant was tested on a series of object-recognition tasks using a variety of shapes with the BrainPort device while sitting in a stationary, fixed position at 1, 2, and 3 meters (about 3.3, 6.6, and 10 feet, respectively). All the participants were instructed to use a fixed field of view (FOV) with the device and were not permitted to adjust the camera zoom after their initial familiarization with the device. It should be noted that the FOV was variable for each participant, depending on his or her level of comfort and ability to discriminate shapes in the training phase of the protocol.

Protocol

Three randomized trials of shape-recognition tasks (five distinct target shapes--a triangle, circle, bar, square, and the letter C) at three distances (1, 2, and 3 meters) were conducted in 13 presentations for each trial, a total of 39 presentations). The triangle, circle, square, and letter C measured approximately 7 inches across and from top to bottom, and the bar shape measured 1 inch wide by 24 inches across. The two primary measures of the participants' performance that were assessed in this analysis were recognition accuracy and the latency of shape identification. This protocol was conducted over three consecutive days, resulting in 117 total shape-identification tasks. A maximum recognition latency (time ceiling) of 15 seconds was used in all the trials (if the participants reached 15 seconds without identifying an identified shape, they were forced to declare their response). The researchers relied on a digital stopwatch to measure recognition latency and recorded each response time to within a tenth of a second. Testing was performed in a controlled indoor environment with all shapes presented at 100 contrast (solid white shapes presented on a solid black background) to maximize the capture of the image by the camera lens of the BrainPort.

This study followed the tenets of the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. It was approved by the Emory University Institutional Review Board's oversight committee to ensure that all the participants' rights and protections were in order. The participants signed informed consent forms, and each received an honorarium for participating.

Results

The sample included 16 women and 14 men, with an average age of 51.5 years (SD = 9.9, age range 32-71). Of the 30 participants, 83% reported a visual impairment of seven years or longer, with 90% reporting either total blindness (no light perception) or a profound visual impairment (light perception only, hand motion). The participants reported a variety of eye diseases and retinal disorders, including optic nerve-related diagnoses (40%), glaucoma (20%), maculopathies (10%), chorioretinal atrophy (10%), retinopathy of prematurity (10%), and retinitis pigmentosa (10%). Fourteen participants (47%) were gainfully employed, and the remaining 16 participants reported involvement with school or training (10%), volunteer activities (10%), or social and recreational activities (33%). The racial composition of the sample was 60% African American and 40% white.

The participants' performance scores on recognition latency (calculated and pooled for all 39 presentations of shapes on each of the three trials) averaged approximately 12 seconds, irrespective of distance (see Table 1). A repeated-measures analysis of variance (ANOVA) that examined the impact of distance on recognition latencies across all three trials revealed no within-subject effect.

An analysis of the impact of distance on recognition accuracy was also conducted. It is not surprising that the findings revealed that accuracy declined as the distance from the target increased. Using a repeated-measures ANOVA, we found that the average rate of recognition accuracy at 1 meter (the average of all 39 presentations of shapes at 1 meter) was 57.2; at 2 meters, 49.7; and at 3 meters, 40.1 (see Table 2), revealing a significant within-subject effect (F = 13.459, p < .001). Within-subject contrast further indicated a significant linear trend (F = 20.699, p < .001), with the shorter distances associated with the highest degree of accuracy. Note that with the five target shapes, the chance of a correct response would have been 20.5%, or approximately 24 correct responses out of 117 total presentations. In fact, the participants averaged 57 correct responses over all the distances and presentations (an overall average of 48.7), resulting in a recognition accuracy rate that was significantly higher than chance (p < .001).

The association of specific characteristics of the participants (including age, gender, race, and the duration or etiology of the visual impairment) with the participants' performance was examined, fully acknowledging that the small sample and the even smaller number of individuals with specific characteristics made such explorations speculative at best. No significant associations with either accuracy or latency were uncovered when a Bonferroni corrected alpha was applied. Within the sensitivity of these exploratory analyses and small sample size, it appears that no characteristic of the participants precludes the use of this device.

The influence of the shape of the target on the participants' performance was also examined for the five shapes that were used, with the average recognition accuracy for each of the shapes calculated at the 1 meter test distance that was used for this analysis. A repeated-measures ANOVA that was used to examine the within-subjects difference in the participants' performance with the different shapes revealed a significant difference among the shapes (F = 6.53, p < .001). The order of the accuracy of the shapes was the bar (.69), the circle (.60), the triangle (.51), the square (.44), and the "C" (.43); the performance of the 30 participants on these shapes is displayed in Table 3. Pairwise comparisons indicated that the bar was significantly more accurately identified than was the "C," triangle, and square, and that the "C" was significantly less accurately identified than were the circle and the bar.

Conclusions

The participants clearly demonstrated an enhanced ability to identify target shapes accurately in all the trials and at all three distances that far exceeded the results that would be expected purely on chance. The linear relationship between accuracy and target distance that was demonstrated in these analyses is likewise an important finding and suggests that a critical target size exists in which an effective range of discrimination of targets and visual stimuli can be determined. It is notable that there appeared to be no relationship between the characteristics of the participants (including the etiology of the eye disease) and recognition accuracy or latency. However, because of the small sample (N = 30), the statistical power and the potential for Type II errors is a concern; thus, the apparent lack of association between the participants' characteristics and performance of the tasks that was demonstrated in this study has yet to be fully examined as a potentially important factor in determining the most suitable users of this technology.

One area of concern related to the use of the BrainPort that has been frequently commented on has to do with mapping the camera field of view onto the fixed resolution electrotactile tongue display. In other words, perceptual detail depends upon how the size of the visual image translates to specific "tactors" on the tongue itself. Since a digital camera image can provide a wide field of view with many pixels of information and the tongue array has many fewer analogous tactors to display, spatial averaging is required, thus reducing the perceived detail. By manipulating field of view through the use of digital zoom, however, the user can increase the perceived detail by narrowing the field of view, thus matching the camera resolution to the tongue display resolution. Although it makes intuitive sense to understand the processes of the electrotactile representation of shapes in a way that is analogous to visual systems, this process is trickier than would appear. At a fixed FOV of 10 degrees, each electrode within the 625 electrotactile tongue array covers 0.4 deg/electrode (personal communication with A. Arnoldussen, July 1, 2010), but given the variable nature of the FOV that was used by the participants in this study, an accurate control of this variable was not possible. One consideration that should not be lost on readers is the influence of the camera lens itself on the resolution capacity of the BrainPort device, and that FOV is obviously associated with variations in the zoom level. What is perhaps most germane when considering the limits of resolution of this device is the potential resolution of the human tongue to discriminate gradations of stimuli that are presented via the electrotactile tongue array. Previous research (Arnoldussen et al., 2008) suggests that the tongue can resolve two small points (each 0.167 millimeters in diameter) of electrical stimulation at 0.50-0.75 millimeters apart.

Although the purpose of this study was to test the effectiveness of this device in helping the participants acquire accurate information about the shapes of presented objects at various distances, it is inevitable that more fundamental questions about the underlying mechanisms at work in the BrainPort device or human interface will continue to be raised. Further work in these areas is under way and will undoubtedly help shed light on many of these unanswered questions. More to the point regarding the functional implications of the BrainPort device is how it may assist users in the myriad of visual-related activities that individuals encounter in daily life. It is obvious that additional testing of this device under dynamic and variable indoor and outdoor lighting environments, as well as in real-world, ecologically valid settings, is crucial to assess the functional implications of this technology. On the basis of the findings of this pilot study, it seems that the BrainPort technology represents an intriguing new twist on the historic notion of sensory substitution and certainly demonstrates potential as an augmentative strategy for individuals with significant vision loss.

References

Arnoldussen, A., & Hogle, R. (2008). The BrainPort gaze-directed vision augmentation for those with macular degeneration. In Proceedings of the 9th International Conference on Low Vision, Vision 2009, Montreal.

Arnoldussen, A., Hogle, R., Fisher, E., Lederer, S., Rosing, M., Besta, B., & Ferber, A. (2008). Spatial resolution on the tongue as applied to a prosthetic BrainPort vision device. In Proceedings of the 2008 Association for Research in Vision and Ophthalmology Annual Meeting, Fort Lauderdale, FL.

Bach-y-Rita, P. (1967). Sensory plasticity: Applications to a vision substitution system. Acta Neurologica Scandinavica, 43, 417-426.

Bach-y-Rita, P. (1972). Brain mechanisms in sensory substitution. New York: Academic Press.

Bach-y-Rita, P. (2005). Emerging concepts of brain function. Journal of Integrative Neuroscience, 4, 183-205.

Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221, 963-964.

Bach-y-Rita, P., Kaczmarek, K., Tyler, M., & Garcia-Lara, J. (1998). Form perception with a 49-point electrotactile stimulus array on the tongue. Journal of Rehabilitation Research and Development, 35, 427-430.

Bach-y-Rita, P., & Kercel, S. (2003). Sensory substitution and the human-machine interface. Trends in Cognitive Sciences, 7, 541-546.

Blasch, B. B., Wiener, W. R., & Welsh, R. L. (1997). Introduction. In Foundations of Orientation and Mobility (2nd ed., pp. 1-6). New York: American Foundation for the Blind.

OPTACON Owner's Manual: Model R1D. (1978). Mountain View, CA: Telesensory Systems.

Michael D. Williams, Ph.D., research scientist, Atlanta VA Medical Center, Rehabilitation R&D Center of Excellence (151R), 1670 Clairmont Road, Decatur, GA 30033; e-mail: <mike.williams2@va.gov>. Christopher T. Ray, Ph.D., ATC, CSCS, research scientist, Dallas VA Medical Center, 4500 South Lancaster Road, Dallas, TX 75216, and assistant professor, Department of Kinesiology, University of Texas at Arlington, Box 19259, 111 Maverick Activities Center, Arlington, TX 76019; e-mail: <chrisray@uta.edu>. Jennifer Griffith, M.A., research coordinator, Mount Sinai School of Medicine, 1 Gustave L. Levy Place, New York, NY 10029; e-mail: <jennifer.griffith@mssm.edu>. William De l'Aune, Ph.D., research scientist, Atlanta VA Medical Center, Rehabilitation R&D Center of Excellence (151R); e-mail: <dad@delaune.org>. Address correspondence to Dr. Williams.

Download braille-ready file

Download ASCII text file (ASCII files are for download only)

Previous Article | Next Article | Table of Contents

There are 0 comments on this article.

Post a Comment

JVIB, Copyright © 2012 American Foundation for the Blind. All rights reserved.

If you would like to give us feedback, please contact us at jvib@afb.net.