Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Brain-Informed Fine-Tuning of Language Models Leads to More Powerful and Generalizable Models of the Brain
Poster Session E, Sunday, September 14, 11:00 am - 12:30 pm, Field House
Isil Poyraz Bilgin1,2, Marie St-Laurent2, Lune Pierre Bellec2, Leila Wehbe1; 1Carnegie Mellon University, 2University of Montreal
Recent work has shown that large language models (LLMs) can predict brain activity, as measured by fMRI, during naturalistic stimuli such as movies or stories (Wehbe et al., 2014; Schrimpf et al., 2021; Caucheteux et al., 2021). These studies typically use pre-trained LLM embeddings to model neural responses. However, the reverse direction, using brain recordings to guide the learning of LLMs, remains largely unexplored. This raises important questions: Can brain activity improve language model alignment? Does biologically informed training enhance linguistic performance? We address this gap using a large-scale fMRI dataset from six participants watching the first six seasons of the TV show Friends (Boyle et al., 2021). The data were collected in a within-subject design and preprocessed with fMRIprep. Season 3 was held out as a test set to evaluate generalization to unseen neural responses, but we also used the Movie10 dataset (Boyle et al., 2021) for additional out-of-sample generalization. Each fMRI time point was aligned with the preceding language context using transcripts and a canonical hemodynamic response function (HRF). We developed a brain-aligned raining framework using GPT-2 and compare three raining regimes: (1) a baseline model with fixed GPT-2 embeddings aligned with a linear ridge regression encoder; (2) a brain-finetuned model, where the layers of transformers were adapted using Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method and only the final six layers were finetuned using the brain data; and (3) a train-from-scratch model, where GPT-2 randomly initialized and trained solely on brain data. Our results demonstrated that the brain-finetuned model consistently outperformed both the baseline and the scratch model in predicting held-out brain responses. Importantly, performance improved as the amount of training data increased, indicating that brain signals offer a scalable and meaningful supervision signal. Specifically, scaling up to a training set of twenty hours of brain data led to a significant jump in performance in large parts of the cortex (including bilateral temporal and frontal), to a level that is not matched by the baseline linear model, whose performance reaches a drastically lower plateau at ten hours. Further, these results were generalized to different audiovisual stimuli from the Movie10 dataset. We note that having a dual loss (a brain prediction and a next-word prediction loss) appears essential to preserve or promote good perplexity scores, as evidenced by our observation that not including a next-word prediction loss leads to a bad perplexity score for both the scratch and fine-tuned models. These results indicate that the brain-prediction training signal is at least in part different from the next-word prediction training signal, hinting that there is a potential benefit of learning new representations when combining both. These findings support the hypothesis that brain activity encodes information that can influence LLM adaptation, but also highlight that it is not trivial to integrate this information constructively. Further, this work resulted in predictive models of brain activity that are far more accurate than those obtained using the established encoding methodology and that generalize well to out-of-training distribution stimuli.
Topic Areas: Computational Approaches, Meaning: Lexical Semantics