The AI industry reshuffles faster than anyone predicted. Companies that dominated two years ago are struggling. Startups nobody heard of are suddenly worth billions. The map changes every quarter. For those of us watching…
MMLU is saturated. HumanEval is contaminated. Most popular benchmarks have become optimization targets rather than measurement tools. When the benchmark becomes the goal, it ceases to measure what it was designed to measure. This…
You’d think experts would make the best annotators. They understand the domain deeply. They can evaluate quality with precision. They know what good looks like. But in practice, domain experts consistently produce worse training…
Most datasets train one capability at a time. Reasoning datasets train reasoning. Conversation datasets train conversation. Code datasets train code. This produces models that are good at specific tasks and mediocre at integrating capabilities….
Curation is the most undervalued skill in AI. Anyone can scrape the internet and dump it into a training pipeline. Building a dataset that actually produces a good model requires judgment, patience, and taste….
The AI industry obsesses over architecture. Transformers vs. state space models. Dense vs. mixture-of-experts. Billions vs. trillions of parameters. These choices matter. But they matter less than the data. Two models with identical architecture…
For a brief moment, prompt engineering was a field. People discovered that adding “step by step” to your query made models perform better. That asking a model to “think carefully” improved responses. That the…
We have excellent benchmarks for knowledge. MMLU tests broad knowledge across domains. Arc tests reasoning. HellaSwag tests common sense. But we have no good benchmarks for empathy, wisdom, or nuance. This is a massive…
Every time you send data to a cloud API, you’re trusting a third party with information that might be sensitive, proprietary, or confidential. Self-hosted AI offers a radically different model: run everything locally. The…
Current benchmarks measure speed, accuracy, benchmark scores. They don’t measure what really matters: nuance, the ability to hold competing perspectives, structural coherence in reasoning, the capacity for intellectual humility. As AI moves beyond pattern-matching…