The Quality-Quantity Tradeoff: 500 Good Pairs Beat 50,000 Bad Ones

There’s pressure to build big datasets. 100k pairs. 500k pairs. “More data is always better,” the thinking goes. It’s wrong.

Laeka’s research shows consistent pattern: 500 high-quality pairs outperform 50,000 noisy pairs. The difference isn’t marginal. It’s 2-3x better downstream task performance.

Why Quality Beats Quantity

Every noisy pair introduces contradiction into your training signal. If Pair 1 says “verbose is bad” and Pair 50000 (from a different annotator) says “verbose is good,” the model learns: maybe verbose is sometimes good? The model’s confidence degrades. It stops learning clear principles.

With 500 high-quality pairs, every pair reinforces the same principles. The model’s signal is clean. It learns with high confidence. This confidence transfers to novel prompts.

Quality is signal. Quantity without quality is noise.

The Math

Assume:

500 pairs, 90% annotator agreement = 450 signal pairs, 50 noisy pairs.

50,000 pairs, 60% annotator agreement = 30,000 signal pairs, 20,000 noisy pairs.

The noisy pairs don’t cancel out. They accumulate. With 20,000 contradictory signals, the model learns to ignore weak signals and memorize surface patterns.

With 50 contradictory signals, the model can afford to learn through them. They’re noise in signal.

Cost Analysis

500 high-quality pairs:

Collecting prompts: 40 hours. Generating responses: 10 hours. Annotation (with quality control): 200 hours. Quality checks: 20 hours. Total: 270 hours. Cost: $8,000-12,000 (depending on annotation rates).

50,000 noisy pairs (crowdsourced):

Everything is scaled 100x. Collecting prompts: 4,000 hours. Generating responses: 1,000 hours. Annotation: 20,000 hours. Quality checks: 2,000 hours. Total: 27,000 hours. Cost: $200,000-300,000.

The small dataset is 25x cheaper and produces better results. This isn’t a trade-off. It’s a win-win.

How to Get High Quality Pairs

Recruit domain experts. Pay them well. Limit annotation batches (50-100 pairs per session). Use explicit rubrics. Measure inter-rater agreement. Remove outlier annotators. Iterate.

It’s slower. It’s more expensive per pair. But you end up with something that actually trains good models.

When More Pairs Help

After you hit 500 high-quality pairs and see strong signal, then scale. Add more pairs while maintaining quality standards. But don’t sacrifice quality for volume.

The scaling law isn’t linear. Your 501st pair contributes less than your 1st pair (diminishing returns). You need to be at least as rigorous.

The Uncomfortable Truth

Teams love big numbers. “We built a 100k-pair dataset!” Sounds impressive. Doesn’t mean anything if 60% of it is garbage.

The teams winning on model quality are building small, high-quality datasets. They’re not bragging about size. They’re obsessing over signal.

Laeka Research — laeka.org