Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Mapping idiographic affective interpretations to brain activity using semantic embeddings
Poster Session E, Sunday, September 14, 11:00 am - 12:30 pm, Field House
This poster is part of the Sandbox Series.
Sushmita Sadhukha1, Eshin Jolly2, Nir Jacoby1, Younji Choi1, Benjamin T. Keller1, Tor D. Wager1, Jeremy R. Manning1, Luke J. Chang1; 1Dartmouth College, 2University of California San Diego
Emotions are critical to our everyday information processing, yet capturing moment-to-moment fluctuations in these multi-system responses poses a unique challenge. When processing complex stimuli like narratives, emotional responses integrate both exogenous information from the environment and endogenous representations shaped by individual differences in internal homeostatic states, past experiences, and future goals (Ashar et al., 2017). This integration results in highly idiosyncratic interpretations across individuals (Chang et al., 2021). Unlike sensory experiences which can be directly linked to external stimuli, emotional experiences cannot be locked to specific elements in the external world, creating a fundamental methodological challenge: asking participants to report on their internal states disrupts the ongoing experience of their internal states, yet such probing remains our only window into understanding them. As a potential solution, we propose a novel computational framework that captures’ individuals’ idiosyncratic affective interpretations during naturalistic movie viewing by combining collaborative filtering with advances in natural language processing. Participants (N=122) watched 8 emotionally engaging audiovisual stories in the form of autobiographical narratives while undergoing fMRI. Following fMRI scanning, participants were presented with transcripts of each story and asked to indicate which moments in the story affected them the most and were asked to make comments on their internal states and thoughts in these moments. We extracted semantic embeddings of each response using a BERT Sentence Transformer (384 dimensions; Reimers and Gurevych, 2019). Next, for each embedding dimension, we performed data imputation using collaborative filtering to create a dense participant-by-time-by-embedding dimension tensor using the Neighbors toolbox (Jolly et al., 2022). For each participant, we computed distance correlations (Székely et al., 2007) between time-by-time representational dissimilarity matrices (RDMs) derived from the semantic embeddings and from brain activity in each region of interest (ROI) from a k=50 whole brain parcellation (de la Vega et al., 2016). This approach relates patterns of semantic similarity in participants' affective interpretations to patterns of neural activity similarity across time points. Ultimately, by mapping these time-varying semantic representations to multivariate brain activity, we can identify which brain regions encode state transitions in emotional processing as approximated by the language model embeddings of participants' subjective responses. Across all stories, we find preliminary evidence for associations between participants’ idiosyncratic interpretations to the narratives and dynamic fluctuations in several brain regions, including the vmPFC, the lateral occipital/temporal occipital cortex, and the temporal occipital fusiform area. These results support prior work that has demonstrated that the vmPFC is involved in affective appraisals (Chang et al., 2021). Additionally, we observe story-specific regional activations that vary based on narrative content. This approach provides a proof of concept of how sparsely sampled subjective interpretations can be used to reconstruct a more complete picture of dynamic emotional experiences while minimizing disruption to the ongoing experience. While additional validation is needed, these preliminary findings have potential implications for understanding the neural basis of idiosyncratic emotional processing and the links between participants’ interpretations and the semantic content of the narratives themselves.
Topic Areas: Methods, Computational Approaches