Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Modality Dominance and Integration in L2 Comprehension: An fMRI Study of Speech Rate Effects
Poster Session A, Friday, September 12, 11:00 am - 12:30 pm, Field House
This poster is part of the Sandbox Series.
Mayumi Kajiura1, Chunlin Liu2, Diego Elisandro Dardon3, Motoaki Sugiura2, Hyeonjeong Jeong2; 1Nagoya City University, Nagoya, Japan, 2Tohoku University, Sendai, Japan, 3Miyagi University of Education, Sendai, Japan
[Introduction] In second language (L2) comprehension, integrating multiple modalities—such as audio and text—has been shown to enhance understanding compared to relying on a single modality (Shaojie et al., 2022). However, some studies report that the simultaneous processing of auditory and visual inputs may lead to cognitive interference due to the limited capacity of attentional resources. The listening-dominance hypothesis (Robinson et al., 2018) suggests that, under time constraints, auditory input tends to be prioritized over visual input, even when the auditory signal is difficult to comprehend. This prioritization may delay or diminish the processing of concurrently presented visual information. Conversely, other research indicates that when auditory input becomes less reliable—such as in time-compressed or degraded speech—attention may shift toward the more stable visual modality (Seijdel, 2024). These contrasting findings imply that the effectiveness of multimodal input may vary depending on cognitive demands and input characteristics, including speech rate. The present study investigates whether L2 learners integrate auditory and visual information simultaneously, or whether they selectively prioritize one modality depending on the rate of auditory input. [Method] To examine this, we employed functional magnetic resonance imaging (fMRI) under two conditions: (1) simultaneous presentation of standard-speed English audio (easier to comprehend) and written text, and (2) simultaneous presentation of time-compressed (2× speed) audio (more difficult to comprehend) and text. Twenty-four Japanese university and graduate students learning English as a foreign language performed a semantic judgment task in both conditions. Behavioral measures included response accuracy and reaction times. Brain activity was analyzed across two key time windows: the audio presentation phase and the post-audio phase (from audio offset to participant response), during which only the visual text remained available. [Data Analysis] Behavioral performance was compared between conditions to evaluate comprehension. Whole-brain analyses identified brain regions exhibiting differential activation based on speech rate and processing phase. Repeated-measures ANOVAs were conducted to examine interaction effects between condition (standard vs. fast) and phase (during vs. after audio). Region-of-interest (ROI) analyses targeted areas involved in auditory processing (anterior superior temporal gyrus [STG]), multimodal integration (inferior frontal gyrus [IFG], angular gyrus [AG]), and executive function (dorsolateral prefrontal cortex [DLPFC]). [Expected Results] We hypothesize that under the standard-speed condition, learners will integrate auditory and visual inputs in real time, as reflected by concurrent activation of STG, IFG, and AG. In contrast, under the 2× speed condition, auditory input may initially dominate, with visual information being integrated later during the post-audio phase. This pattern may be reflected in delayed activation of IFG and AG, along with increased recruitment of DLPFC, which supports top-down control and working memory. These findings will contribute to a better understanding of how input rate and cognitive demands influence modality processing in L2 learners and offer insights for designing effective multimodal instructional materials.
Topic Areas: Multisensory or Sensorimotor Integration, Control, Selection, and Executive Processes