We have excellent benchmarks for knowledge. MMLU tests broad knowledge across domains. Arc tests reasoning. HellaSwag tests common sense.<\/p>\n
But we have no good benchmarks for empathy, wisdom, or nuance. This is a massive blind spot. Models are being evaluated on dimensions they’re being trained to optimize, while other critical dimensions go unmeasured.<\/p>\n
Current benchmarks are primarily knowledge and capability tests. They ask: Can the model answer this factual question? Can it solve this math problem? Can it write code that works?<\/p>\n
These are important. But they measure only one slice of model quality. They don’t measure whether the model is honest, whether it understands context appropriately, whether it can distinguish between what it knows and what it’s guessing.<\/p>\n
Empathy:<\/strong> Can the model recognize emotional content? Does it respond appropriately to distressed users? Can it adjust tone based on context?<\/p>\n Wisdom:<\/strong> Can the model recognize the limits of its knowledge? Does it give measured responses to complicated questions, or does it overstate certainty? Can it balance competing values?<\/p>\n Nuance:<\/strong> Does the model understand that most real-world questions don’t have simple answers? Can it hold multiple perspectives simultaneously? Can it say “it depends”?<\/p>\n These qualities are hard to measure. But they matter enormously in practice.<\/p>\n Models are optimized for metrics we measure. If we only measure knowledge, we get models that are knowledgeable but emotionally tone-deaf and overconfident.<\/p>\n We see this in practice. Models that score well on benchmarks often produce outputs that are technically correct but contextually inappropriate or emotionally harsh.<\/p>\n The solution isn’t to replace capability benchmarks. It’s to add new ones.<\/p>\n Empathy Benchmark:<\/strong> Present scenarios involving emotional distress or interpersonal complexity. Evaluate whether model responses demonstrate understanding and appropriate emotional sensitivity.<\/p>\n Wisdom Benchmark:<\/strong> Ask difficult questions that don’t have clear answers (e.g., “How should I balance career and family?”). Evaluate whether model acknowledges uncertainty, presents multiple perspectives, and avoids false certainty.<\/p>\n Nuance Benchmark:<\/strong> Present cases with competing values or reasonable disagreement. Evaluate whether model can articulate multiple valid viewpoints rather than taking a single stance.<\/p>\n These benchmarks are harder to construct than factual knowledge tests. They’re more subjective. Inter-rater agreement is harder to achieve.<\/p>\n But difficulty isn’t an excuse for avoidance. The dimensions we don’t measure are often the ones that matter most in practice.<\/p>\n Start with small, carefully constructed benchmarks. Have diverse raters evaluate model responses. Iterate. Improve. Make nuance, wisdom, and empathy visible and measurable.<\/p>\n Only then can we optimize for them. Only then can we be confident we’re building models that are not just smart, but wise.<\/p>\nWhy This Matters<\/h2>\n
What Better Benchmarks Might Look Like<\/h2>\n
The Challenge<\/h2>\n
A Path Forward<\/h2>\n