Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Generalizable Semantic Decoding from iEEG Using Deep Nonlinear CNNs
Poster Session A, Friday, September 12, 11:00 am - 12:30 pm, Field House
William L Gross1, Alex Skitowski1, Hernan G Rey1, Kunal Gupta1; 1Medical College of Wisconsin
Direct semantic decoding of concept identities from brain activity is essential for neuroprosthetic applications but remains a formidable challenge. Although motor articulation and planning (Moses, 2021) and phonological coding (Mesgarani , 2014; Pasley, 2012) have been successfully decoded to support speech synthesis, many individuals with post‑stroke aphasia cannot reliably generate these signals and thus would not benefit from those approaches. Prior work has recovered coarse semantic information via fMRI (Mitchell, 2008; Huth, 2016) and classified broad semantic categories with EEG (Rupp, 2017; Murphy, 2011), yet decoding a small, fixed set of categories does not guarantee scalability to a full vocabulary. Here, we tested whether deep nonlinear decoders can map brain activity onto a continuous semantic embedding space, capturing complex signal-meaning relationships. We recorded intracranial EEG from awake neurosurgical patients as they viewed rapid presentations of concrete concepts (e.g., “hammer,” “lion”) and made semantic category decisions. In total, 100 concepts were trained, each shown as three different images, 10 times each (3,000 trials; 500 ms ITI). We evaluated two decoder classes: 1) linear Ridge regression to assess linear decodability, and 2) a 12-layer deep convolutional neural network (CNN) to capture nonlinear patterns, each decoding continuous word2vec embeddings. Training used preprocessed, epoched data from left‑hemisphere electrodes; Ridge models received flattened channel‑by‑time vectors, while CNNs processed 2D spatiotemporal inputs. Models trained on 90% of stimuli and were tested on three held‑out conditions: 1) unseen trials of trained images, 2) novel images of trained concepts, 3) images of entirely novel concepts. Performance is the average rank-accuracy across four non‑overlapping splits (chance=50%). Both models decoded held‑out trials of trained images above chance (Ridge=59%, CNN=57%; p < 0.001) and extended to novel images of trained concepts, with the CNN performing significantly better than Ridge (64% vs. 58%; p = 0.0174), indicating stronger abstraction of semantic-level features. Critically, for entirely novel concepts, Ridge failed to exceed chance performance (52%; p = 0.1508) whereas the CNN achieved 66% (p < 0.001), demonstrating its ability to generalize mappings into continuous semantic space. These results demonstrate that deep nonlinear decoders can be trained to generalize to unseen stimuli and extend semantic decoding from discrete categories into continuous embeddings. Despite the training complexity of a deep CNN model, it performed better with only ≈25% as many parameters as the Ridge model. Such models hold promise for brain–computer interfaces that recover full‐vocabulary semantics in aphasia. Future work could explore the potential of larger architectures to refine neural-semantic mappings with larger amounts of data, advancing the field toward real‑time continuous semantic decoding.
Topic Areas: Computational Approaches, Meaning: Lexical Semantics