Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Context Affects Model-to-Brain Alignment in Language Comprehension
Poster Session C, Saturday, September 13, 11:00 am - 12:30 pm, Field House
Anuja Negi1,2, Fatma Deniz1,2; 1Technical University of Berlin, Berlin, Germany, 2Bernstein Center for Computational Neuroscience Berlin, Berlin, Germany
Recent progress in artificial neural networks for natural language processing has raised the question of whether these models represent language in ways similar to the human brain. Previous studies have shown that contextual embeddings from transformer-based language models can predict brain responses with high accuracy (Schrimpf et al., 2021; Caucheteux & King, 2022) and outperform static lexical embeddings (Jain & Huth, 2018; Caucheteux & King, 2022). However, most of this work has focused on a single type of linguistic stimulus, typically narrative texts or isolated sentences. At the same time, it is known that the amount of context in a stimulus influences both semantic representations and the signal-to-noise ratio of the brain recordings (Deniz & Tseng et al., 2023). This raises an important question: How does the alignment between language model representations and brain activity change with the amount of context (in different conditions)? In this study, we compare how well transformer-based embeddings with different context lengths predict brain responses across different types of stimuli: isolated words, isolated sentences, and narratives. For each stimulus condition, participants read words presented one-by-one while brain data were recorded using functional magnetic resonance imaging (fMRI). The stimuli are based on 11 stories from The Moth Radio Hour (used in Huth et al., 2016). Using voxelwise encoding models and the fMRI data, we investigate how language model context length affects prediction performance and whether these effects localize to specific brain regions. We extracted contextual embeddings from each layer (n=12) of the BERT model with four different context lengths: 0, 5, 10, and 20 tokens. For comparison with a non-contextual model, static word embeddings (english1000) were also extracted. Motion energy features (Nunez-Elizalde et al., 2022) were calculated to account for sensory visual information. Banded ridge regression (Nunez-Elizalde et al., 2019) was used to fit joint encoding models for each voxel, layer, participant, and condition. Model performance was measured using Pearson correlation between predicted and recorded BOLD responses on held-out data. Our results reveal a clear effect of stimulus context on the context length selectivity of voxels. For isolated words, in all voxels, static word embedding outperforms all contextual embeddings except for embeddings with 0 context length. For sentences and narratives, contextual embeddings for middle to late layers outperform static word embedding when context length is greater than 0. For voxels primarily in inferior frontal and prefrontal cortices, we observe a context length selectivity shift from shorter (5/10) to longer (20) from sentences to narratives condition. We did not find a simple match between model layers and brain regions (e.g., early layers to early brain areas and late layers to higher-order areas). Instead, the relationship depends on both the type of stimulus and how much context the model receives. This work highlights the importance of analyzing linguistic complexity explicitly when relating language models to brain data.
Topic Areas: Meaning: Lexical Semantics, Computational Approaches