Aligning artificial and biological computation

The idea

We record the brain during ordinary life, while people listen and talk, and ask whether an agent’s inner workings predict what the brain is doing. A close match is a measurement, not an explanation. It tells us where the two systems agree, not why. So we look at the machinery behind the match: what each system is trying to do, how it is built, how it learns, and what it learned from. Where the agent and the brain agree, we have a candidate principle of how the mind works. Where they diverge, the divergence tells us what to build next.

In practice: ECoG 24/7

Some patients with epilepsy spend a week in hospital with electrodes placed on the surface of the brain, so that doctors can find the source of their seizures. With their consent, we record from those electrodes around the clock while they simply talk: with doctors, with friends, with family. We match every word spoken and heard to the brain’s activity. For the first time, we can test a language model against the brain in the middle of real conversation, not in a lab task.

Where they align

The brain and today’s language models handle speech in the same way. Both predict each word before it arrives. Both register surprise when the prediction fails. And both hold a word’s meaning by the company it keeps, so that the same word means something different in different contexts. Even the model’s layers line up with the brain: deeper layers track later brain activity.

The match goes further. In face-to-face conversation, the model gives us a shared map on which to watch two brains at once. Meaning appears in the speaker’s brain just before each word is spoken, and the same meaning reappears in the listener’s brain just after it is heard. The model lets us follow a thought as it crosses from one mind to another.

Where they part ways

A language model holds an enormous window of text in view at once, weighing every word against every other. The brain does not. Most of the cortex integrates over seconds, less than a paragraph. And yet the brain follows a story for hours, binding each moment to what came before. It does this by another route: it compresses the past into memory and retrieves it when needed. Today’s models have nothing like this. Nor do they have the brain’s loops in time, its spiking neurons, or a learning rule that a real synapse could carry out. Each of these gaps is a hypothesis about how the brain works, and each is something we can now build.

The more closely an agent computes as the brain does, the more it becomes what we are after: not a system that merely predicts the brain, but a working model of how the brain works.