Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Visual Speech Differently Restores Temporal and Spectral Speech Information in the Auditory Cortex
Poster Session B, Friday, September 12, 4:30 - 6:00 pm, Field House
Cody Cao1, James Brissenden1, Andrew Jahn1, David Brang1; 1Department of Psychology, University of Michigan
Seeing a speaker’s face facilitates accurate speech perception. Previously, research has shown that listeners make use of lipreading to restore degraded auditory speech information, but these influences fail to account for the full benefits imparted by visual speech. Prior work has hypothesized that visual speech can restore information from two additional features of speech: spectral (relative pitch) and temporal information (sound changes over time). For example, listeners can recover spectral information using speakers’ mouth width and the speaker’s lip closure helps listeners parse the temporal boundary between words. However, it remains poorly understood whether, and how, spectral and temporal information is restored in the auditory cortex. In the current study, we asked two questions: first, is visual speech integration regionally specific, where auditory temporal and spectral information is restored in separate areas, or regionally nonspecific, where the same region restores both kinds of information? Second, how does visual speech alter the spatial pattern of auditory system activity to improve audibility of speech? We hypothesized that visual speech restores the spatial pattern of auditory activity evoked by degraded auditory speech, making it more similar to the pattern of activity evoked by clear speech. We collected functional magnetic resonance imaging (fMRI) data from 64 subjects who listened to 200 sentences presented across five conditions-- auditory-alone original, auditory-alone temporally smeared, auditory-alone spectrally smeared, audio-visual temporally degraded, and audio-visual spectrally degraded. Univariate contrasts reveal that the same visual signal has different effects on auditory processing depending on the degraded auditory feature. Visual speech increased BOLD in the STG for both types of degraded speech, but differed across conditions in other regions: AV spectral recovery increased BOLD in Heschl's gyrus whereas AV temporal recovery increased BOLD in anterior STG. Second, we used Representational Similarity Analysis to compare fMRI data of audiovisual conditions to the auditory unfiltered condition. Our preliminary results from RSA analysis suggest that visual restoration of speech uses distinctly different mechanisms from auditory speech perception. To verify this finding, we plan to build a single-trial-based GLM regressor to examine if representational distance is closer between audiovisual filtered speech and original auditory speech than that of audio-alone filtered speech. Together,we show that auditory cortex uses visual speech signals to selectively recover features of the auditory signal that have been degraded.
Topic Areas: Multisensory or Sensorimotor Integration, Speech Perception