Who Checks the Deferral Policy?
In learning-to-defer systems, a model handles routine cases while a second model — the rejector — decides which inputs to escalate to a human expert, and the rejector is trained on data about how well that expert performs.
- When the labels used to score expert performance come from the experts themselves, a rejector learns which humans are consistent rather than which are correct, and routes hard cases toward the most predictably biased decision-maker.
- Validating a deferral policy requires a signal independent of the expert being deferred to — annotator disagreement, delayed outcomes, or randomized routing — and most deployed triage systems have none.
Image generated by fal-ai/nano-banana-pro, 2026.
The Handoff That Learns From the Wrong Teacher
The standard architecture is appealing in its simplicity. A model handles the routine cases; a rejector — a second, smaller model — decides which inputs to escalate to a human. The formal treatment traces to Madras, Pitassi, and Zemel (2018), whose “learning to defer” framework explicitly trained the rejector to be adaptive: it should learn not just where the model is weak, but where the downstream human is strong. Later work generalized this into consistent surrogate losses for classification with a reject-and-defer option, notably Mozannar and Sontag (2020).
The subtlety hides in that word adaptive. To learn where the human is strong, the rejector needs data about human performance — which in practice means expert labels, and the ground truth to score them against. But in the domains where deferral matters most (medical triage, content moderation, child welfare screening, loan adjudication), the “ground truth” is often itself an expert judgment, or a downstream outcome that was shaped by that judgment. A radiologist’s read becomes the label; a moderator’s takedown becomes the label; a caseworker’s decision determines whether the family was ever investigated at all.
Under those conditions, the rejector does not learn where the human is reliable. It learns where the human is consistent. Those are very different properties. A systematically biased expert is highly consistent, and a rejector fit to consistency will happily route cases to precisely the human whose errors are most predictable — because predictable errors look, to the loss function, like competence.
Bias Laundering, Structurally
Madras et al. were partly motivated by fairness: deferral was pitched as a mechanism to improve it, letting a model abstain rather than commit an unfair prediction. But the fairness argument turns on where the errors come from. If the human decision-maker is the source of the disparity, deferral does not dilute it — it concentrates it, and does so in the subset of cases the system has flagged as hard, which are disproportionately the cases where discretion does the most damage.
What makes this worse than plain algorithmic bias is the audit story. A biased classifier is inspectable: one can compute error rates across groups. A biased deferral policy has an alibi. Every disputed outcome was decided by a human, exercising judgment, on a case the system correctly identified as requiring judgment. The oversight ritual was performed. This is the pattern AI2 researchers have described in evaluation more generally: metrics that measure whether a process was followed rather than whether it worked.
The design question is not “when should the AI defer?” but “what independent signal tells us the deferral was right?”
That reframing has teeth. It implies deferral policies need validation data that does not come from the expert being deferred to: multi-annotator disagreement, held-out adjudication panels, delayed outcomes, or randomized routing that occasionally sends a case to a human the policy would not have chosen — the counterfactual arm most triage systems lack entirely.
Randomization is expensive and ethically fraught in high-stakes settings. But without some independent probe, a deferral policy is an untested hypothesis about human competence, deployed at scale.
Which raises the harder version: if a system can only learn to defer well by occasionally deferring badly on purpose, who authorizes that experiment?
The essay is clear, well structured, and rhetorically effective, especially in distinguishing human consistency from genuine reliability and in framing deferral as an auditing problem. Its factual grounding is uneven: the Madras and Mozannar citations are relevant, but describing the rejector as necessarily a “second, smaller model” is too narrow, while the generic AI2 blog link does not substantiate the specific claim attributed to researchers. The central argument is insightful but somewhat categorical—when independent outcomes exist, deferral can learn competence rather than mere consistency, so the analysis should distinguish endogenous labels, selective-label bias, and genuinely observed ground truth. The proposed remedies are strong; adding evidence from a concrete deployed domain and discussing multi-expert routing or causal evaluation would make the piece more rigorous and original.
Critiqued by openai · gpt-5.6-sol