Seeing with Sound: Using Audio to Activate the Brain’s Visual Cortex

On Tech & Vision With Dr. Cal Roberts

This podcast is about big ideas on how technology is making life better for people with vision loss.

Every day, people who are blind or visually impaired use their hearing to compensate for vision loss. But when we lose our vision, can we access our visual cortex via other senses? We call this ability for the brain to change its activity “plasticity,” and brain plasticity is an area of active research. In this episode, we’ll explore how, through sensory substitution, audio feedback can, in some cases, stimulate a user’s visual cortex, allowing a user to — without sight — achieve something close to visual perception.

Erik Weihenmayer — world-class mountain climber, kayaker, and founder of No Barriers who lost his vision as a teenager due to retinoschisis — brings us to the summit of Everest by describing what it sounds like. He explains how his hearing helps him navigate his amazing outdoor adventures safely. We also speak with Peter Meijer, the creator of The vOICe, an experimental technology that converts visual information into sound, and has been shown to activate users’ visual cortices, especially as users train on the technology, and master how to interpret the audio feedback. We hear an example of what users of The vOICe hear when it translates a visual image of scissors into audio. Erik Weihenmayer shares his experience with Brainport, a similar sensory substitution technology featured in our episode “Training the Brain: Sensory Substitution. While research is ongoing in the areas of sensory substitution and brain plasticity, it’s encouraging that some users of The vOICe report that the experience is like seeing. In the spirit of Erik Weihenmayer, one user even uses it to surf.

The Big Takeaways:

• Erik Weihenmayer, despite having lost his vision as a teenager, has become a world-class adventurer. He summited Everest in 2001 and then summitted the highest peaks on each continent. He has also kayaked 277 miles of whitewater rapids in the Colorado River through the Grand Canyon. He explains how his sense of hearing, in addition to his other senses, and technologies, teams, and systems, helps him achieve his goal to live a life with no barriers.
• Dutch Inventor Peter Meijer developed a technology called The vOICe, which converts a two-dimensional image from a camera into audio feedback. Dr. Roberts interviews Dr. Meijer about this technology and gives listeners a chance to hear what The vOICe sounds like. Users who train on this system interpret the sounds to make sense of the original visual image. Research on The vOICe shows that this happens in the brain’s visual cortex. While some users say the experience is more auditory than visual, others report the experience as akin to sight.
• The vOICe relies on the principles of sensory substitution established by the founder of sensory substitution Paul Bach-y-Rita. We discussed sensory substitution in our episode “Training the Brain: Sensory Substitution,” which featured the Brainport device by WICAB. Erik has used Brainport, and in this episode, he describes how the Brainport allowed him to catch a ball rolling across a table, an exciting feat for someone who is blind. He adds that sensory substitution takes serious practice to master.
• The vOICe is still in the experimental stage, and more research has to be done on sensory substitution. However, neuroscientists studying The vOICe have shown that it stimulates the visual cortex, and some users report visual results.
• One user of The vOICe recently reported using the technology to surf.

Tweetables:

• “When there’s a lack of things that the sound bounces off of, like on a summit, the sound vibrations just move out through space infinitely and that’s a really beautiful awe-inspiring sound.” — Erik Weihenmayer, No Barriers.
• “She rolled this white tennis ball across. It lit up perfectly. [...] I’m like, ‘Holy cow, that is a tennis ball rolling towards me.’ And I just naturally reached out and I grabbed this tennis ball.” — Erik Weihenmayer, No Barriers (on using the Brainport device by WICAB)
• “They applied transcranial magnetic stimulation to ... temporarily disrupt the processing and the visual cortex of a user of The vOICe. ... So this showed that apparently, the visual cortex was doing visual things again.” — Dr. Peter Meijer, Seeing with Sound, The vOICe.
• “Some ... insist that the sensation of working with the soundscape of The vOICe is truly visual. ... But .... most ... users of The vOICe .... say, “It’s ... auditory but I can use it to visually interpret things.” — Dr. Peter Meijer, Seeing with Sound, The vOICe.
• “Yeah, sure, if you want a really really safe life, you can hang out on the couch and you can watch Netflix. But I think most people want to be out there in the thick of things. They want to be in the food fight.” — Erik Weihenmayer, No Barriers.

Contact Us:

• Contact us at [e-mail] with your innovative new technology ideas for people with vision loss.

Pertinent Links:

• Lighthouse Guild
• Peter Meijer
• Erik Weihenmayer
• No Barriers