Matteo Visconti di Oleggio Castello, Ph.D.

One of the major goals of cognitive neuroscience is understanding how the brain represents information about its own internal states and about the external world. This goal can be addressed by creating encoding models that reveal the information represented explicitly in measured brain activity. Here we describe the Voxelwise Encoding Model (VEM) framework for creating encoding models with functional magnetic resonance imaging (fMRI) data. The VEM framework provides several key advantages over traditional neuroimaging approaches. First, the VEM framework is applicable to most experimental designs, from classic factorial designs to complex naturalistic experiments such as movie watching or video games. This flexibility enables researchers to study brain function across multiple domains with the same analytical approach. Second, hypotheses about functional representations are defined and tested quantitatively by extracting feature spaces from experimental stimuli or tasks. These feature spaces quantify specific types of information potentially represented in brain activity and can range from simple human-derived labels to complex features generated by deep neural networks. If a feature space can be used to linearly predict brain activity, the brain is considered to explicitly represent features within that feature space. This prediction approach provides a systematic way to test hypotheses about functional representations. Third, the VEM framework implements robust data science methods to improve model estimation and minimize false positive results. Encoding models are estimated on a training set and validated on an independent test set. Testing in an independent dataset provides direct evidence that experimental findings generalize beyond the dataset used for model estimation. Finally, voxelwise encoding models can be created in each participant’s native brain space without unnecessary information loss due to spatial averaging or template resampling required by conventional group analyses. This enables the VEM approach to reveal fine-grained functional organization in individual participants that might otherwise be ignored. In this article, we provide a comprehensive guide to the Voxelwise Encoding Model framework through all phases of research, from experimental design and data collection to the estimation and interpretation of voxelwise encoding models.

Each person experiences the world through a unique conceptual lens, shaped by personal experiences, natural variations, or disease. These individual differences have remained largely inaccessible to cognitive neuroscience and clinical neurology, limiting the development of precision medicine approaches to cognitive disorders. To overcome this limitation, here we develop a new statistical framework to measure and interpret individual differences in functional brain representations. We apply this framework to characterize how different individuals represent the same concepts. Twenty-four participants listened to narrative stories while their brain activity was measured with functional MRI (fMRI). Encoding models were used to recover how hundreds of concepts were represented in each person’s brain. Despite listening to identical stories, participants showed systematic individual differences in conceptual representations. These differences reveal person-specific biases in how concepts are represented in the brain. Individual variability was highest in regions that represent social information. Because these regions are thought to integrate sensory information with personal beliefs and experiences, the observed individual differences may reflect cognitive traits unique to each person. Our work reveals that individual differences are a systematic, measurable principle of conceptual representations in the human brain. By enabling researchers to measure and interpret differences in person-specific functional brain representations, our work establishes a new paradigm for precision neuroscience. This paradigm provides a rigorous foundation for developing fMRI applications in precision medicine to diagnose and monitor cognitive disorders.

Functional connectivity (FC) is the most popular method for recovering functional networks of brain areas with fMRI. However, because FC is defined as temporal correlations in brain activity, FC networks are confounded by noise and lack a precise functional role. To overcome these limitations, we developed model connectivity (MC). MC is defined as similarities in encoding model weights, which quantify reliable functional activity in terms of interpretable stimulus- or task-related features. To compare FC and MC, both methods were applied to a naturalistic story listening dataset. FC recovered spatially broad networks that are confounded by noise, and that lack a clear role during natural language comprehension. By contrast, MC recovered spatially localized networks that are robust to noise, and that represent distinct categories of semantic concepts. Thus, MC is a powerful data-driven approach for recovering and interpreting the functional networks that support complex cognitive processes.

The Voxelwise Encoding Model framework (VEM) is a powerful approach for functional brain mapping. In the VEM framework, features are extracted from the stimulus (or task) and used in an encoding model to predict brain activity. If the encoding model is able to predict brain activity in some part of the brain, then one may conclude that some information represented in the features is also encoded in the brain. In VEM, a separate encoding model is fitted on each spatial sample (i.e., each voxel). VEM has many benefits compared to other methods for analyzing and modeling neuroimaging data. Most importantly, VEM can use large numbers of features simultaneously, which enables the analysis of complex naturalistic stimuli and tasks. Therefore, VEM can produce high-dimensional functional maps that reflect the selectivity of each voxel to large numbers of features. Moreover, because model performance is estimated on a separate test dataset not used during fitting, VEM minimizes overfitting and inflated Type I error confounds that plague other approaches, and the results of VEM generalize to new subjects and new stimuli. Despite these benefits, VEM is still not widely used in neuroimaging, partly because no tutorials on this method are available currently. To demystify the VEM framework and ease its dissemination, this paper presents a series of hands-on tutorials accessible to novice practitioners. The VEM tutorials are based on free open-source tools and public datasets, and reproduce the analysis presented in previously published work.

Processes evoked by seeing a personally familiar face encompass recognition of visual appearance and activation of social and person knowledge. Whereas visual appearance is the same for all viewers, social and person knowledge may be more idiosyncratic. Using between-subject multivariate decoding of hyperaligned functional magnetic resonance imaging data, we investigated whether representations of personally familiar faces in different parts of the distributed neural system for face perception are shared across individuals who know the same people. We found that the identities of both personally familiar and merely visually familiar faces were decoded accurately across brains in the core system for visual processing, but only the identities of personally familiar faces could be decoded across brains in the extended system for processing nonvisual information associated with faces. Our results show that personal interactions with the same individuals lead to shared neural representations of both the seen and unseen features that distinguish their identities.

Personally familiar faces are processed more robustly and efficiently than unfamiliar faces. The human face processing system comprises a core system that analyzes the visual appearance of faces and an extended system for the retrieval of person-knowledge and other nonvisual information. We applied multivariate pattern analysis to fMRI data to investigate aspects of familiarity that are shared by all familiar identities and information that distinguishes specific face identities from each other. Both identity-independent familiarity information and face identity could be decoded in an overlapping set of areas in the core and extended systems. Representational similarity analysis revealed a clear distinction between the two systems and a subdivision of the core system into ventral, dorsal and anterior components. This study provides evidence that activity in the extended system carries information about both individual identities and personal familiarity, while clarifying and extending the organization of the core system for face perception.