When Vaswani et al. published “Attention Is All You Need” in 2017, they borrowed a term from cognitive science. Then the field promptly forgot everything cognitive science knows about attention. That forgetting is costing us.<\/p>\n
Attention isn’t just a computational convenience. In every major contemplative tradition, attention is the fundamental technology of transformation<\/strong>. Where you place your attention determines what you perceive, what you learn, and what you become.<\/p>\n Buddhist psychology identifies at least seven factors of attention. Directed attention (vitakka). Sustained attention (vicara). Selective attention. Open awareness. Metacognitive attention \u2014 attention to attention itself. Each of these has distinct properties, distinct training methods, and distinct effects on cognition.<\/p>\n The transformer’s attention mechanism captures maybe one of these: selective attention. The Query-Key-Value framework computes relevance scores and allocates processing resources accordingly. This is powerful, but it’s a fraction of what attention actually does in biological cognitive systems.<\/p>\n Sustained attention.<\/strong> Transformers process everything in parallel. There’s no mechanism for dwelling on something \u2014 returning to it, holding it, letting it deepen over time. Human attention can sustain focus on a single object for extended periods, and this sustained attention produces qualitatively different understanding than a single pass.<\/p>\n Metacognitive attention.<\/strong> Transformers can’t attend to their own attention. They can’t notice that they’re focusing too heavily on one part of the context, or that their attention distribution is biased. This self-monitoring capacity is what meditation systematically develops, and its absence in AI systems explains many alignment failures.<\/p>\n Intentional direction.<\/strong> Human attention can be deliberately directed based on goals, values, and context. A meditator chooses where to place attention and maintains that choice against distractions. Transformer attention is entirely reactive \u2014 determined by the learned weights and the input, with no capacity for intentional override.<\/p>\n Attentional quality.<\/strong> Not all attention is equal. Contemplative traditions distinguish between tight, constricted attention and spacious, open attention. Between effortful concentration and effortless awareness. These qualitative differences affect the output. Tight attention catches details but misses context. Open attention grasps patterns but misses specifics. The optimal cognitive system can modulate between these modes.<\/p>\n Most alignment problems are attention problems in disguise.<\/p>\n When a model focuses on the surface features of a prompt rather than the underlying intent, that’s an attention allocation failure. When it over-indexes on certain training patterns and ignores others, that’s attentional bias. When it can’t detect that its own response is drifting off-topic or becoming harmful, that’s a metacognitive attention deficit.<\/p>\n Current approaches try to fix these problems through training data and loss functions. But if the attention mechanism itself lacks the capacity for sustained focus, self-monitoring, and intentional direction, then better training data is a bandage on an architectural wound.<\/p>\n What would it look like to engineer attention mechanisms informed by contemplative science?<\/p>\n Multi-pass attention with depth.<\/strong> Instead of a single forward pass, allow the model to attend to the same content multiple times at different levels of abstraction. First pass: surface meaning. Second pass: implications. Third pass: meta-level assessment. This mimics how sustained attention in meditation progressively deepens understanding of the same object.<\/p>\n Attention monitoring layers.<\/strong> Add architectural components that attend to the attention patterns themselves. If the model’s attention is concentrated too narrowly (missing context) or too broadly (lacking specificity), these monitoring layers could trigger reprocessing. This is architectural metacognition.<\/p>\n Goal-modulated attention.<\/strong> Allow high-level task representations to modulate attention weights. If the goal is accuracy, attention should focus differently than if the goal is creativity or empathy. Contemplative practitioners do this naturally \u2014 they modulate their attentional mode based on the situation.<\/p>\n Attentional mode switching.<\/strong> Build mechanisms that allow the model to shift between focused and diffuse attention modes within a single generation. Focused for precise reasoning. Diffuse for creative connections. The optimal response often requires both.<\/p>\n The researchers who named the attention mechanism borrowed a word with 2,500 years of technical meaning. That meaning includes systematic training, qualitative modulation, metacognitive monitoring, and intentional direction. We’ve implemented the simplest possible version and achieved remarkable results. Imagine what happens when we implement the rest.<\/p>\nWhat’s Missing From Mechanical Attention<\/h2>\n
The Alignment Implications<\/h2>\n
Engineering Better Attention<\/h2>\n
Taking the Name Seriously<\/h2>\n