Poster Presentation

Search Abstracts | Symposia | Slide Sessions | Poster Sessions

Multivariate fMRI Uncovers Form-Invariant and Language-Specific ASL Verb Representations in Deaf Signers

Poster Session E, Sunday, September 14, 11:00 am - 12:30 pm, Field House

Akshi Akshi¹, Miriam Hauptman¹, Matthew Sampson¹, Qi Cheng², Marina Bedny¹; ¹Johns Hopkins University, ²University of Washington

Verbs are a linguistic universal and partially neurobiologically dissociate from other word types (e.g., nouns) and visual representations of actions (Bedny et al., 2012; Papeo et al., 2019; Elli et al., 2019; Wurm et al., 2019). Previous studies in spoken languages implicate the left middle temporal gyrus (LMTG) in the representation of verb lexical-semantics (e.g., Hauptman et al., 2025). Sign languages have rich verb lexicons and morphology, the neurobiological underpinnings of which remain underexplored. American Sign Language (ASL) verbs provide an important test-case because in ASL, depicting verbs take handling classifiers. For example, the non-classifier verb form of ‘give’ is modified with a handling classifier to convey how the object is handled. Depicting classifier forms are iconic and resemble the actions they refer to (e.g., open, give, pick up), raising questions about their linguistic versus gestural nature. We used multivariate pattern analysis (MVPA) fMRI to look for abstract ASL verb representations invariant across classifier (i.e., form) variation and across sentence contexts. We scanned congenitally profoundly deaf participants (N = 15; ASL acquisition ranging from native to late) as they viewed ASL sentences featuring six action verbs (give, take, open, close, pick-up, put-down) in three grammatical structures: Subject-Verb-Object with Non-Classifier form, Object-Subject-Verb with Non-Classifier form, and Object-Subject-Verb with Classifier form (e.g. in ASL GLOSS: "MAN OPEN BOTTLE", "BOTTLE MAN OPEN".) Sentences varied in signers’ identity (man, woman), the grammatical subject (man, woman), and object (bottle, book, laptop, teapot). In a matched non-linguistic condition, participants watched meaning-matched action videos (e.g., a video of a man opening a bottle). As a low-level control, participants viewed perceptually matched but meaningless videos created by superimposing backward-played ASL and action videos that preserved motion, human bodies, and faces. Individually defined functional ROI-based MVPA (cross validated) and whole-brain searchlight analyses were conducted to decode verbs and actions. MVPA revealed largely non-overlapping decoding of verbs and actions. A language-responsive left MTG region showed above-chance decoding for ASL verbs (p<0.01) but did not decode actions depicted in videos (p>0.1). Whole-brain searchlight analyses identified different spatial distributions for ASL verbs (left MTG) and visual actions (right-lateralized posterior temporal and occipitotemporal cortex). Cross-domain decoding, i.e., training on ASL verbs and testing on actions and vice-versa, failed to identify common representations across sentences and videos. Thus, we find a neural dissociation of ASL action verbs (linguistic) and visual actions (non-linguistic). ASL verbs were successfully decoded across perceptual and grammatical variation in ASL sentences, including signer identity, grammatical subject and object, and word-order (all p’s<0.01). Most importantly, decoding was also invariant of classifier surface form, i.e., training on classifier constructions and testing on non-classifier constructions and vice-versa showed above chance decoding (p<0.001). In conclusion, classifier and non-classifier verbs in ASL are represented in language-specific regions previously associated with verb processing for spoken languages, i.e., the LMTG. ASL verb representations are invariant to surface classifier forms. Despite sharing modality and perceived iconicity with action representations, ASL verb representations are neurally language-specific.

Topic Areas: Signed Language and Gesture, Meaning: Lexical Semantics