DPO is powerful. But most datasets shipped to train models are noisy, biased, and inconsistent. This ruins training. Understanding the failure modes is the first step to fixing them.<\/p>\n
Annotators disagree. One person marks Response A as better; another marks B. Without inter-rater agreement metrics, you’re training on contradiction.<\/p>\n
Fix: Enforce minimum agreement thresholds. Flag pairs where annotators disagree. Review those manually or remove them. A smaller, consistent dataset beats a large incoherent one.<\/p>\n
Humans prefer the first option shown. Or the last. Or whichever is longer. These biases leak into DPO datasets.<\/p>\n
Fix: Randomize presentation order. Don’t tell annotators which is “Option A.” Show responses without metadata. Audit your final dataset for position bias\u2014plot preference distribution across positions.<\/p>\n
After evaluating 200 responses, annotators get tired. Quality drops. They start marking responses “good enough” without real deliberation.<\/p>\n
Fix: Limit annotation batches. 50-100 pairs per annotator per session. Track agreement over time. If it degrades, pause and rotate annotators.<\/p>\n
“Is this response better?” is vague. Better for what? In what context? The annotator and the person who wrote the criterion interpret “good” differently.<\/p>\n
Fix: Write explicit rubrics. Define what “clear” means, what “complete” means, what “safe” means. Give examples. Then measure consistency against the rubric.<\/p>\n
You train on generic preference data but deploy in a specialized domain. The model never saw examples of what “good” looks like in your domain.<\/p>\n
Fix: Use domain-specific prompts and responses. Recruit annotators familiar with the domain. Their preference signals will be grounded in domain reality.<\/p>\n
Run these checks before training:<\/p>\n
Check 1: Inter-rater agreement. Measure Cohen’s kappa or Fleiss’ kappa across annotators. Target 0.7+.<\/p>\n
Check 2: Position bias. For each response position, count how often it was marked preferred. Should be uniform.<\/p>\n
Check 3: Label distribution. How many pairs are strongly clear vs borderline? Borderline pairs are noise sources.<\/p>\n
Check 4: Annotator composition. Are all pairs from the same person? Hire multiple annotators; their disagreements are where you learn.<\/p>\n
Check 5: Prompt coverage. Are all prompts from one domain? One genre? Real datasets are diverse.<\/p>\n
Bad data in, bad model out. But most teams skip quality assurance because it’s unglamorous. The teams that win are the ones that obsess over dataset quality before training.<\/p>\n