Anil Seth didn’t set out to describe how language models work. He was describing the human brain. But the parallels are so precise they border on uncomfortable.<\/p>\n
Seth’s core argument is elegant. Your brain doesn’t receive reality \u2014 it generates<\/strong> reality. Every moment of conscious experience is a prediction, refined by sensory input. You don’t see the world as it is. You see your brain’s best guess about the world, updated just enough to keep you alive.<\/p>\n This is controlled hallucination. The emphasis is on “controlled.” Without the control \u2014 without sensory input constraining the predictions \u2014 you get actual hallucinations. Dreams. Psychosis. States where the generative engine runs without adequate feedback.<\/p>\n Now read that paragraph again, replacing “brain” with “language model” and “sensory input” with “training data and prompt context.” The structural mapping is exact.<\/p>\n A transformer generates token predictions. Each token is the model’s best guess about what comes next, given everything before it. The training data acts like developmental experience \u2014 shaping the priors. The prompt acts like current sensory input \u2014 constraining the prediction.<\/p>\n When the constraints are strong (specific prompt, well-represented topic, clear context), the model’s outputs closely match reality. When the constraints are weak (vague prompt, rare topic, ambiguous context), the generative engine fills in the gaps with plausible-sounding content that may not be true.<\/p>\n This isn’t a flaw in the architecture. This is the architecture.<\/strong> The same mechanism that produces accurate, helpful responses also produces hallucinations. The variable isn’t the engine \u2014 it’s the quality of the constraints.<\/p>\n Seth’s work builds on Karl Friston’s free energy principle: biological systems minimize surprise by maintaining accurate predictive models of their environment. The brain constantly updates its generative model to reduce the gap between prediction and reality.<\/p>\n Language model training does exactly this. The loss function measures the gap between the model’s predictions and the actual next token. Training minimizes this gap across billions of examples. The result is a generative model that, like the brain, produces predictions that usually match reality \u2014 but sometimes don’t.<\/p>\n The critical insight is that prediction error<\/strong> is the signal. In neuroscience, prediction errors drive learning and attention. In language models, they drive gradient updates. The math is different but the principle is identical: surprise is information, and systems learn by being wrong.<\/p>\n If language models are controlled hallucination engines, then the alignment question becomes: how do you improve the control without killing the generation?<\/strong><\/p>\n In Seth’s framework, control comes from sensory grounding. The brain stays anchored to reality through continuous feedback from the body and the world. Disruptions to this grounding \u2014 sensory deprivation, psychedelic drugs, neurological conditions \u2014 produce uncontrolled hallucination.<\/p>\n Language models lack this continuous grounding. They generate in open loop \u2014 producing output without real-time feedback about whether that output corresponds to reality. Retrieval-Augmented Generation (RAG) is essentially an attempt to add sensory grounding: anchor the model’s predictions to verified external data.<\/p>\n But RAG is crude compared to the brain’s feedback mechanisms. The brain integrates millions of sensory signals simultaneously, at multiple levels of abstraction, with microsecond latency. Current grounding techniques are more like occasionally checking a fact sheet.<\/p>\n Here’s where contemplative practice enters. Meditation doesn’t stop the brain’s generative engine. It enhances the monitoring system<\/strong>. An experienced meditator has better real-time awareness of their own predictions, better detection of when those predictions are ungrounded, and better ability to flag uncertainty.<\/p>\n This is the missing piece in current AI alignment. We need models that don’t just generate \u2014 they need to monitor their own generation<\/strong>. Not through external fact-checking, but through internal mechanisms that track the quality of the prediction in real-time.<\/p>\n Some research moves in this direction. Confidence calibration work tries to align a model’s stated certainty with its actual accuracy. But most of this work is post-hoc \u2014 analyzing outputs after generation, not monitoring the generation process itself.<\/p>\n Seth’s framework suggests three concrete research directions for AI alignment. First, better grounding mechanisms<\/strong> \u2014 not just RAG, but continuous, multi-level feedback during generation. Second, internal monitoring<\/strong> \u2014 training models to detect when their own predictions are weakly supported. Third, uncertainty communication<\/strong> \u2014 models that naturally express their confidence level as part of the generation process, not as an afterthought.<\/p>\n The contemplative tradition has been developing internal monitoring techniques for millennia. The neuroscience of meditation is finally explaining why these techniques work. And the structural parallels to language models are too clear to ignore.<\/p>\nThe Generative Engine Is the Same<\/h2>\n
Predictive Processing and the Free Energy Principle<\/h2>\n
Where the Analogy Gets Practical<\/h2>\n
Contemplative Practice as Enhanced Control<\/h2>\n
The Research Agenda<\/h2>\n