There’s pressure to build big datasets. 100k pairs. 500k pairs. “More data is always better,” the thinking goes. It’s wrong.<\/p>\n
Laeka’s research shows consistent pattern: 500 high-quality pairs outperform 50,000 noisy pairs. The difference isn’t marginal. It’s 2-3x better downstream task performance.<\/p>\n
Every noisy pair introduces contradiction into your training signal. If Pair 1 says “verbose is bad” and Pair 50000 (from a different annotator) says “verbose is good,” the model learns: maybe verbose is sometimes good? The model’s confidence degrades. It stops learning clear principles.<\/p>\n
With 500 high-quality pairs, every pair reinforces the same principles. The model’s signal is clean. It learns with high confidence. This confidence transfers to novel prompts.<\/p>\n
Quality is signal. Quantity without quality is noise.<\/p>\n
Assume:<\/p>\n
500 pairs, 90% annotator agreement = 450 signal pairs, 50 noisy pairs.<\/p>\n
50,000 pairs, 60% annotator agreement = 30,000 signal pairs, 20,000 noisy pairs.<\/p>\n
The noisy pairs don’t cancel out. They accumulate. With 20,000 contradictory signals, the model learns to ignore weak signals and memorize surface patterns.<\/p>\n
With 50 contradictory signals, the model can afford to learn through them. They’re noise in signal.<\/p>\n
500 high-quality pairs:<\/p>\n
Collecting prompts: 40 hours. Generating responses: 10 hours. Annotation (with quality control): 200 hours. Quality checks: 20 hours. Total: 270 hours. Cost: $8,000-12,000 (depending on annotation rates).<\/p>\n
50,000 noisy pairs (crowdsourced):<\/p>\n
Everything is scaled 100x. Collecting prompts: 4,000 hours. Generating responses: 1,000 hours. Annotation: 20,000 hours. Quality checks: 2,000 hours. Total: 27,000 hours. Cost: $200,000-300,000.<\/p>\n
The small dataset is 25x cheaper and produces better results. This isn’t a trade-off. It’s a win-win.<\/p>\n
Recruit domain experts. Pay them well. Limit annotation batches (50-100 pairs per session). Use explicit rubrics. Measure inter-rater agreement. Remove outlier annotators. Iterate.<\/p>\n
It’s slower. It’s more expensive per pair. But you end up with something that actually trains good models.<\/p>\n
After you hit 500 high-quality pairs and see strong signal, then scale. Add more pairs while maintaining quality standards. But don’t sacrifice quality for volume.<\/p>\n
The scaling law isn’t linear. Your 501st pair contributes less than your 1st pair (diminishing returns). You need to be at least as rigorous.<\/p>\n
Teams love big numbers. “We built a 100k-pair dataset!” Sounds impressive. Doesn’t mean anything if 60% of it is garbage.<\/p>\n
The teams winning on model quality are building small, high-quality datasets. They’re not bragging about size. They’re obsessing over signal.<\/p>\n