Anna Ivanova

The neural basis of program comprehension

Computer programming is a novel cognitive tool that has transformed modern society. However, little is known about the cognitive and neural systems that support this skill. Here, we use fMRI to test the role of two brain networks that may support program comprehension: the domain-general multiple demand system, which is known to respond during math and logic tasks, and the language system, which exhibits selective responses to linguistic signals. We conducted two separate experiments using stimuli written in two different programming languages: Python, a popular text-based programming language, and ScratchJr, a graphical programming language for kids. In both experiments, we found robust responses to code within the multiple demand system, as well as within the language system. The language system responses were primarily observed in regions within inferior frontal and middle frontal gyri. Our work establishes programming as a novel domain for studying cognitive processing in the human brain and suggests that program comprehension might draw on both domain-general and language-specific resources.

Jon Gauthier

Linking human and artificial neural representations of language

What information from an act of sentence understanding is robustly represented in the human brain? We investigate this question by comparing sentence encoding models on a brain decoding task, where the sentence that an experimental participant has seen must be predicted from the fMRI signal evoked by the sentence. We take a pre-trained BERT architecture as a baseline sentence encoding model and fine-tune it on a variety of natural language understanding (NLU) tasks, asking which lead to improvements in brain-decoding performance.

We find that none of the sentence encoding tasks tested yield significant increases in brain decoding performance. Through further task ablations and representational analyses, we find that tasks which produce syntax-light representations yield significant improvements in brain decoding performance. Our results constrain the space of NLU models that could best account for human neural representations of language, but also suggest limits on the possibility of decoding fine-grained syntactic information from fMRI human neuroimaging.