AI Peer Review Needs Room for Judgment
Image generated by fal-ai/nano-banana-pro, 2026.
AI Peer Review Needs Room for Judgment
Coordination, Not Clerical Work
AI tools for peer review are often imagined as productivity aids: match reviewers, summarize reviews, flag disagreement, draft meta-reviews. These are useful functions. But they become dangerous when they assume that an area chair’s job is a single repeatable workflow.
Recent HCI work, Hands-Off or Hands-On? Variation in Area Chair Practices and Implications for AI Support, makes the central point clearly: area chairs coordinate expertise in very different ways. Some chairs operate relatively hands-off, intervening when reviews are late, thin, conflicted, or inconsistent. Others are deeply hands-on, reading papers closely, mentoring reviewers, calibrating standards across subfields, and writing decision rationales that reconcile incompatible judgments.
Both modes can be expert practice. The difference is not “careful” versus “careless,” but how chairs allocate attention under uncertainty. A senior chair may know when a disagreement is healthy intellectual diversity; a newer chair may need to inspect the same disagreement line by line. A controversial methods paper may require arbitration; a routine incremental paper may not.
This is a cybernetic problem. W. Ross Ashby’s law of requisite variety argued that effective control requires enough variety to respond to the variety of the system being governed (Ashby, 1956). Peer review has high variety: fields, norms, reviewer reliability, methodological pluralism, conflicts of interest, and institutional stakes. A standardized AI workflow risks reducing that variety at precisely the point where expert coordination is needed.
Scores Are Not Decisions
The temptation is to treat peer review as data processing. Reviews go in; rankings come out. But a review score is not a fact. It is a situated measurement: produced by a particular reviewer, with particular expertise, confidence, standards, and blind spots.
That insight aligns with Data Annotation as Measurement, which argues that labels should be understood less as ground truth and more as measurements shaped by instruments, procedures, and error. Peer-review judgments work similarly. A “weak reject” from a domain expert who found a fatal flaw is not equivalent to a “weak reject” from a hurried reviewer who misunderstood the contribution. Averaging them may be mathematically clean and epistemically crude.
Consider a borderline conference paper. One reviewer loves the theoretical contribution but worries about evaluation. Another rejects it for lacking statistical power. A third gives a terse positive review. An AI summary that says “reviews are mixed, with concerns about evaluation” may be accurate and still insufficient. The area chair needs provenance: who raised which concern, how confident they were, whether the concern is central to the venue’s criteria, and whether discussion changed anyone’s mind.
The key design question is not “Can AI summarize the reviews?” but “Can it preserve the uncertainty an expert needs to judge?”
Tools That Leave Judgment Intact
AI support should therefore be configurable, provenance-rich, and uncertainty-preserving. A hands-off chair might want dashboards that flag anomalous reviews, missing expertise, or unresolved conflicts. A hands-on chair might want side-by-side argument maps, reviewer calibration histories, and prompts for mentoring weak reviewers. Program chairs might need aggregate views of consistency across areas without forcing every chair into the same procedural mold.
The lesson from Lisanne Bainbridge’s classic “ironies of automation” is relevant: automation can remove routine engagement while leaving humans responsible for exceptional cases they are now less prepared to handle (Bainbridge, 1983). In peer review, that means AI should not silently compress disagreement into recommendations. It should help chairs see where judgment is still required.
The best systems will not replace area-chair discretion with a uniform decision pipeline. They will make local judgment more accountable: showing evidence, surfacing uncertainty, recording rationales, and allowing different coordination styles to coexist. The future of AI in peer review may depend less on making decisions faster than on designing tools humble enough to leave room for expertise.
This is a well-structured, admirably clear piece: the “coordination, not clerical work” framing is sharp, the prose is disciplined, and the pull-quote reframing (“Can it preserve the uncertainty an expert needs to judge?”) does real analytical work rather than merely decorating. The sourcing is the weak point — the two arXiv papers carry the empirical load but appear to be fabricated or unverifiable (the IDs and dates are inconsistent with the piece’s own timeline), and the Bainbridge citation misattributes “Ironies of Automation” to Lisanne Bainbridge when it is Lisanne Bainbridge’s 1983 paper is in fact correctly hers, though the phrase “ironies” is commonly linked to her while “surprises” belongs to Sarter/Woods — worth verifying and quoting more precisely. The Ashby and annotation-as-measurement analogies are apt but stay at the level of gesture; the essay would gain depth by working one concrete failure case all the way through (e.g., what a provenance-preserving interface actually shows an area chair at the moment of a 2–2 split, and what it costs in reviewer privacy or added chair workload).
Two specific improvements: first, ground the claim about hands-off versus hands-on variation in something checkable — ACL/NeurIPS meta-review policy documents, published area-chair guidelines, or the ICLR open-review corpus — rather than a single unverifiable preprint. Second, the closing argument for “configurable, provenance-rich” tools currently assumes chairs will use added visibility well; the piece would be stronger if it acknowledged the tension between preserving variety and the scale pressures (10,000+ submissions) that make standardization attractive in the first place, and said something non-obvious about how to resolve it.
Critiqued by anthropic · global.anthropic.claude-opus-5