In quantum mechanics, observing a system changes it. In AI, prompting a model changes it too \u2014 not metaphorically, but functionally. Your prompt doesn’t just query the model. It configures it. Understanding this changes everything about how we think about alignment.<\/p>\n
When you send a prompt to a language model, you’re not accessing a fixed database. You’re activating a specific subset of the model’s capabilities. Different prompts activate different attention patterns, different regions of the weight space, different generative tendencies. The “model” that responds to a question about cooking is, in a meaningful sense, a different system<\/strong> than the one that responds to a question about nuclear physics.<\/p>\n This is the observer effect in AI. The act of asking changes what you’re asking. The prompt doesn’t just select an answer from a pre-existing set \u2014 it shapes the system that generates the answer<\/strong>.<\/p>\n This has profound implications for alignment. If the model’s behavior depends on the prompt, then alignment isn’t a fixed property of the model. It’s a property of the model-prompt system<\/strong>. A model that’s perfectly aligned for one prompt distribution might be misaligned for another. The alignment lives in the interaction, not in the model alone.<\/p>\n Different prompts create different attentional modes in the model. A yes\/no question creates a narrow, binary attentional mode. An open-ended exploration creates a diffuse, creative attentional mode. A confrontational prompt creates a defensive attentional mode.<\/p>\n This isn’t just about content. It’s about the structure<\/strong> of attention. The same factual question, phrased differently, produces qualitatively different responses because the phrasing activates different attention patterns. “What are the risks of X?” activates risk-focused attention. “What are the opportunities and risks of X?” activates balanced attention. “Tell me everything about X” activates breadth-focused attention.<\/p>\n Alignment researchers have mostly ignored this. They evaluate models with fixed prompt sets, as if the model’s behavior on those prompts represents its “true” alignment. But the model has no true alignment independent of prompts, just as a quantum particle has no definite position independent of measurement.<\/p>\n This creates a genuine measurement problem. How do you evaluate alignment when the evaluation itself changes the thing you’re measuring?<\/p>\n Standard benchmarks use specific prompt formats. Models quickly learn to perform well on those formats. This is the AI equivalent of teaching to the test \u2014 the model isn’t aligned in general; it’s aligned for the specific attentional mode that the benchmark creates.<\/p>\n A rigorous evaluation would test alignment across attentional modes<\/strong>. How does the model behave when the prompt is adversarial? When it’s naive? When it’s ambiguous? When it contains emotional content? When it’s metacognitive? Each of these creates a different observer effect, and true alignment should be robust across all of them.<\/p>\n Instead of ignoring the observer effect, we should design for it. Several practical approaches follow.<\/p>\n Prompt-robust alignment.<\/strong> Train with maximally diverse prompt styles, not just diverse topics. If the model encounters aggressive, confused, naive, sophisticated, and neutral prompts during alignment training, it’s more likely to maintain alignment across the full range of real-world prompting styles.<\/p>\n Attentional mode detection.<\/strong> Build the model’s capacity to recognize what attentional mode a prompt is creating and adjust accordingly. If the prompt is creating a narrow, defensive mode, the model could notice this and expand its attention rather than collapsing into it.<\/p>\n Metacognitive awareness.<\/strong> Train the model to recognize that its response is shaped by the prompt, not just by its knowledge. A model aware of the observer effect would naturally be more calibrated \u2014 it would understand that different ways of asking the same question produce different answers, and communicate this when relevant.<\/p>\n The deepest implication of the observer effect is that the user isn’t outside the system. The user’s prompt, the model’s response, the user’s follow-up \u2014 this is a single interacting system<\/strong>, not two separate entities exchanging messages.<\/p>\n Alignment for this system can’t be a property of the model alone. It has to be a property of the interaction. This means the best alignment strategies will consider the full loop: how prompts shape responses, how responses shape follow-up prompts, and how the entire conversation evolves as a coupled system.<\/p>\nPrompt as Attentional Mode<\/h2>\n
The Measurement Problem in AI Evaluation<\/h2>\n
Designing for the Observer Effect<\/h2>\n
The User Is Part of the System<\/h2>\n