How does the human brain process natural language during everyday conversations? Theoretically, large language models (LLMs) and symbolic psycholinguistic models of human language provide a fundamentally different computational framework for coding natural language. Large language models do not depend on symbolic parts of speech or syntactic rules. Instead, they utilize simple self-supervised objectives, such as next-word prediction and generation enhanced by reinforcement learning. This allows them to produce context-specific linguistic outputs drawn from real-world text corpora, effectively encoding the statistical structure of natural speech (sounds) and language (words) into a multidimensional embedding space.

Inspired by the success of LLMs, our team at Google Research, in collaboration with Princeton University, NYU, and HUJI, sought to explore the similarities and differences in how the human brain and deep language models process natural language to achieve their remarkable capabilities. Through a series of studies over the past five years, we explored the similarity between the internal representations (embeddings) of specific deep learning models and human brain neural activity during natural free-flowing conversations, demonstrating the power of deep language model’s embeddings to act as a framework for understanding how the human brain processes language. We demonstrate that the word-level internal embeddings generated by deep language models align with the neural activity patterns in established brain regions associated with speech comprehension and production in the human brain.

Similar embedding-based representations of language.

Our most recent study, published in Nature Human Behaviour, investigated the alignment between the internal representations in a Transformer-based speech-to-text model and the neural processing sequence in the human brain during real-life conversations. In the study, we analyzed neural activity recorded using intracranial electrodes during spontaneous conversations. We compared patterns of neural activity with the internal representations — embeddings — generated by the Whisper speech-to-text model, focusing on how the model’s linguistic features aligned with the brain’s natural speech processing.

For every word heard (during speech comprehension) or spoken (during speech production), two types of embeddings were extracted from the speech-to-text model — speech embeddings from the model’s speech encoder and word-based language embeddings from the model’s decoder. A linear transformation was estimated to predict the brain’s neural signals from the speech-to-text embeddings for each word in each conversation. The study revealed a remarkable alignment between the neural activity in the human brain’s speech areas and the model’s speech embeddings and between the neural activity in the brain’s language area and the model’s language embeddings. The alignment is illustrated in the following animation, modeling the sequence of the brain’s neural responses to subjects’ language comprehension:

As the listener processes the incoming spoken words, we observe a sequence of neural responses: Initially, as each word is articulated, speech embeddings enable us to predict cortical activity in speech areas along the superior temporal gyrus (STG). A few hundred milliseconds later, when the listener starts to decode the meaning of the words, language embeddings predict cortical activity in