Search Abstracts | Symposia | Slide Sessions | Poster Sessions
How well do language models describe the brain?: A robust reanalysis of six fMRI datasets
Poster Session B, Friday, September 12, 4:30 - 6:00 pm, Field House
James Fodor1; 1University of Melbourne
The rise of large language models (LLMs) has led to extensive interest in their potential value as cognitive models of how the brain represents language. In the past decade, dozens of studies have investigated this question using fMRI of participants reading or listening to linguistic stimuli. The resulting patterns of brain activity are typically compared to embeddings derived from LLMs and other vector-based semantics models. Most studies use an encoding paradigm, in which a regression model is trained for each voxel to predict brain activity using model embeddings. Many such studies have found robust correlations between brain activations and representations derived from LLMs (Arana et al., 2023; Hale et al., 2021; Karamolegkou, et al., 2023; Oota, Gupta, et al., 2023). However, several limitations hinder the interpretation of these results. The first is the substantial methodological variability across studies makes it difficult to compare findings and assess whether certain LLMs consistently match brain activity better than others. The second is the reliance on voxelwise encoding models, which involve fitting many regression parameters for each voxel, and have recently been shown to be susceptible to overfitting due partly to insufficiently controlling for autocorrelation of the BOLD signal (Hadidi et al. 2025). To overcome these limitations, we use a consistent framework to reanalyse six fMRI datasets with a total of 74 participants, covering a range of stimuli including written sentences (Pereira, et al., 2018, Anderson et al., 2017), audio narratives (Y. Zhang et al., 2020, Bhattasali et al. 2020), and written stories (Wehbe, et al., 2014). To reduce the risk of overfitting due to autocorrelation, we segment narrative stimuli into individual sentences and fit a general linear regression model in the same way as studies using discrete sentences. We then train a separate voxelwise regression model using cross-validation over stimuli for each participant. To further assess the robustness of these results to the chosen technique, we also performed representational similarity analysis (RSA), in which we compare the similarity structure of brain representations to the similarity structure of the model embeddings. Overall, we find significant positive correlations between every computational model and each fMRi dataset using both voxelwise encoding and RSA methods. We also find that while LLMs typically outperform older models based on static word embeddings, the magnitude of this effect is modest and somewhat inconsistent across datasets. Our results constitute the first systematic comparison between voxelwise encoding and RSA, and show that these techniques generally yield similar results in terms of the relative fit of different models. Our findings also highlight the importance of controlling for confounds like autocorrelation when evaluating semantic models against BOLD data, and cast doubt on previous claims that LLMs match brain representations substantially better than older static embedding models.
Topic Areas: Computational Approaches, Meaning: Lexical Semantics