Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Syntax modulates the neural response to prosody in continuous speech
Poster Session D, Saturday, September 13, 5:00 - 6:30 pm, Field House
Cas W. Coopmans1,2, Andrea E. Martin2,3; 1New York University, 2Donders Institute for Brain, Cognition and Behaviour, 3Max Planck Institute for Psycholinguistics
In order to understand the structure and meaning of spoken language, the human brain relies on prosody. While this idea is both formally and empirically well-established, how the brain uses prosody and syntax together remains elusive. We analyzed brain activity during naturalistic spoken language comprehension to investigate both how prosody modulates the neural encoding of syntax, and how knowledge of syntax shapes encoding of prosody. For each word in 49-minutes of Dutch audiobooks, we computed two syntactic features: the number of phrases that were opened (‘node opening’) and closed (‘node closure’). Moreover, we derived a word-based measure of prosodic boundary strength using the Wavelet Prosody Toolkit. We found a probabilistic relationship between prosody and syntax in speech: words with strong prosodic boundaries are primarily phrase-closing words, whereas weakly marked words are generally phrase-opening words. Thus, prosody probabilistically cues syntactic function, which has implications for language processing: a speaker’s prosody might be able to inform the listener when to integrate a phrase, and their syntax can inform the listener when prosody will be informative for structure building. Twenty-six Dutch-speaking participants listened to the audiobooks while their brain activity was measured with magnetoencephalography. We regressed the syntactic and prosodic features onto sensor-level MEG data using forward encoding models, which controlled for the acoustic and statistical properties of the stimuli by explicitly modeling several additional features (i.e., acoustic spectrogram, acoustic onsets, word onset, frequency, surprisal). To test the interaction between syntax and prosody, we marked each word in terms of its syntactic function (phrase-opening vs. phrase-closing) and in terms of having a strong or a weak boundary. The unique variance explained by each feature was quantified via hierarchical model comparison. Both syntactic features explain unique variance over bilateral frontal-temporal sensors, consistent with neural encoding of syntax across time. The prosodic feature instead did not improve model fit, suggesting that there might not be a stable, unitary response to prosody. In a first test for interactions between syntax and prosody in the neural response, we found that node opening effects were not modulated by prosodic marking. In contrast, node closure was predictive only for words with strong boundary marking: when words are prosodically weakly marked, the brain barely responds to a word’s integrative demands. A second test showed that the neural response to boundary strength was enhanced for phrase-closing words compared to phrase-opening words. This supports the idea that the brain responds to prosody in a heavily context-dependent way: the extent to which the brain encodes prosodic features of speech depends on the potential relevance of these features for broader goals of comprehension – i.e., establishing a structured meaning in context. In sum, speech features a probabilistic relationship between syntactic phrase closure and prosodic boundary marking. Our results show that the brain utilizes this statistical correspondence for the processing of both syntax and prosody: the brain attempts syntactic integration primarily when a word is prosodically strongly marked, and it weights prosodic marking especially heavily when this marking can be syntactically informative.
Topic Areas: Syntax and Combinatorial Semantics, Prosody