Every model release comes with benchmark scores. MMLU, HumanEval, GSM8K, HellaSwag \u2014 the alphabet soup of evaluation. But which benchmarks actually predict real-world performance? And which ones are gamed so thoroughly that they’ve become meaningless? Knowing the difference saves you from deploying models that look great on paper and fail in production.<\/p>\n
MMLU (Massive Multitask Language Understanding)<\/strong> tests broad knowledge across 57 subjects. It remains useful as a rough gauge of general knowledge, though scores above 80% are increasingly unreliable due to data contamination. The variant MMLU-Pro adds harder questions and multiple-choice options to reduce gaming.<\/p>\n HumanEval and MBPP<\/strong> measure code generation ability. HumanEval asks models to write Python functions from docstrings. MBPP tests simpler programming problems. These correlate well with real coding utility, though they only test Python and relatively straightforward problems. EvalPlus extends HumanEval with additional test cases that catch models that pass the original tests through luck.<\/p>\n GSM8K<\/strong> tests grade-school math word problems. Despite the simple framing, it effectively measures multi-step reasoning. Models that score well on GSM8K tend to handle logical reasoning in other domains too. MATH extends this to competition-level mathematics for models pushing the frontier.<\/p>\n MT-Bench<\/strong> uses GPT-4 as a judge to evaluate multi-turn conversation quality. It’s imperfect \u2014 the judge has its own biases \u2014 but it captures aspects of quality that automated metrics miss: coherence across turns, instruction following, and natural dialogue flow.<\/p>\n HellaSwag<\/strong> was once challenging. Now most models score above 85%, compressing the useful signal into a narrow band. It still differentiates weak from adequate models, but tells you nothing about the difference between good and great.<\/p>\n ARC (AI2 Reasoning Challenge)<\/strong> suffers from the same ceiling effect. Both ARC-Easy and ARC-Challenge have been saturated by current models. When benchmark scores cluster between 90-95%, the differences are within noise margins.<\/p>\n TruthfulQA<\/strong> intended to measure honesty and factual accuracy. In practice, models learn the specific patterns of truthful vs. deceptive answers in the benchmark without actually becoming more truthful in general. High TruthfulQA scores don’t reliably predict fewer hallucinations in real use.<\/p>\n The biggest threat to benchmark validity is data contamination<\/strong>. When benchmark questions appear in training data, models score higher without being smarter. This happens more often than the community acknowledges. Popular benchmarks get scraped into web crawls, which get included in training sets.<\/p>\n Some contamination is accidental \u2014 the benchmark data was on the internet, and the internet is in the training set. Some is deliberate \u2014 model creators include benchmark-adjacent data to inflate scores. Either way, contaminated scores don’t predict real-world performance.<\/p>\n The defense against contamination is continual creation of new benchmarks<\/strong>. Private evaluation sets that have never been published online, dynamically generated problems, and live evaluation platforms where questions change regularly. Chatbot Arena’s ELO rating system, based on blind human preferences, is currently the most contamination-resistant evaluation.<\/p>\n Public benchmarks tell you about general capability. They don’t tell you whether a model works for your specific use case. The most important evaluation is the one you build yourself.<\/p>\n Start with 50-100 examples<\/strong> that represent your actual workload. Real customer queries, real documents to process, real code to review \u2014 whatever your application handles. Grade model outputs on the criteria that matter for your use case: accuracy, formatting, tone, completeness.<\/p>\n Use LLM-as-judge<\/strong> for scalable evaluation. Have a stronger model (or the same model with a detailed rubric) score outputs on specific dimensions. This isn’t perfect, but it’s reliable enough to compare models and track quality over time. The key is consistency \u2014 the same judge, the same rubric, the same test set across all evaluations.<\/p>\n A\/B testing with real users<\/strong> is the gold standard when feasible. Deploy two models side by side, let users interact with both (without knowing which is which), and measure preference rates and task completion. This captures everything benchmarks miss: user experience, perceived quality, and practical utility.<\/p>\n No single benchmark tells the full story. The models that score highest on public benchmarks aren’t always the best in production. The models that win Chatbot Arena rankings aren’t always the most cost-effective. The models that pass your custom evaluation aren’t always the fastest.<\/p>\n Effective model evaluation is multidimensional. Quality, speed, cost, safety, reliability under load \u2014 these all matter, and they trade off against each other. The best evaluation strategy measures what matters for your specific context and ignores the rest.<\/p>\nBenchmarks to Be Skeptical Of<\/h2>\n
The Contamination Problem<\/h2>\n
Building Your Own Evaluation<\/h2>\n
The Meta-Lesson<\/h2>\n