When Should the Agent Step Aside?

A small synthetic notebook explores deferral thresholds in a human-AI system where confidence, cost, stakes, and human intervention preferences do not always agree.

Launch in Binder · Download runnable notebook

Why this source

The CMU ML Blog item When Should AI Step Aside?: Teaching Agents When Humans Want to Intervene is a clean Requisite Variety topic: delegation boundaries are not only about model confidence; they are about when a human wants or needs to take agency back.

Two open sources anchor this notebook:

• Source-bank lead: CMU ML Blog, When Should AI Step Aside?: Teaching Agents When Humans Want to Intervene.
• Technical background: David Madras, Toniann Pitassi, and Richard Zemel, Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer, which formalizes prediction systems that can defer to a human expert.
• HCI design context: Saleema Amershi et al., Guidelines for Human-AI Interaction, which frames human control, feedback, and appropriate trust as design concerns.

This notebook is not a reproduction of the CMU work or a proof of any source’s claims. It is a small synthetic exploration of one design question: if an AI agent can either act or step aside for a person, what changes when we optimize for system utility rather than raw automation?

Editorial themes: human-AI collaboration, agency and delegation, feedback and cybernetics.

A toy deferral setup

Imagine a stream of tasks. Each task has:

• difficulty, which tends to lower both AI and human success probability;
• stakes, which makes a mistake more costly;
• an imperfect AI confidence estimate;
• a human review cost paid only when the agent steps aside.

The policy is intentionally simple: defer to the human whenever observed AI confidence is below a threshold.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(7)

n = 1200
difficulty = rng.beta(2.0, 2.2, size=n)
stakes = rng.choice([1.0, 2.0, 5.0], size=n, p=[0.55, 0.30, 0.15])

# Synthetic success probabilities. AI is strong on easy tasks but degrades sharply.
ai_p = np.clip(0.96 - 0.65 * difficulty - 0.03 * (stakes == 5.0), 0.05, 0.99)

# Human performance is costlier but more robust on hard and high-stakes cases.
human_p = np.clip(0.72 - 0.20 * difficulty + 0.08 * (stakes == 5.0), 0.05, 0.95)

# What the system sees: a noisy, imperfect confidence signal.
ai_conf = np.clip(ai_p + rng.normal(0.0, 0.06, size=n), 0.01, 0.99)

human_cost = 0.25 + 0.05 * stakes
thresholds = np.linspace(0.25, 0.95, 71)

rows = []
for threshold in thresholds:
    defer = ai_conf < threshold
    success_p = np.where(defer, human_p, ai_p)
    expected_value = stakes * success_p - human_cost * defer
    rows.append({
        'threshold': threshold,
        'deferral_rate': defer.mean(),
        'mean_success_probability': success_p.mean(),
        'stake_weighted_success': np.mean(stakes * success_p),
        'mean_expected_value': expected_value.mean()
    })

results = pd.DataFrame(rows)
best = results.loc[results['mean_expected_value'].idxmax()]
print('Best threshold by expected value')
print(best.round(3).to_string())

fig, ax1 = plt.subplots(figsize=(8, 4.5))
ax1.plot(results['threshold'], results['mean_expected_value'], color='tab:blue', label='expected value')
ax1.set_xlabel('defer when AI confidence is below this threshold')
ax1.set_ylabel('mean expected value', color='tab:blue')
ax1.tick_params(axis='y', labelcolor='tab:blue')
ax1.axvline(best['threshold'], color='tab:blue', linestyle='--', alpha=0.55)

ax2 = ax1.twinx()
ax2.plot(results['threshold'], results['deferral_rate'], color='tab:orange', label='deferral rate')
ax2.plot(results['threshold'], results['mean_success_probability'], color='tab:green', label='success probability')
ax2.set_ylabel('rate / probability')
ax2.set_ylim(0, 1)

lines = ax1.get_lines() + ax2.get_lines()
labels = [line.get_label() for line in lines]
ax1.legend(lines, labels, loc='center right')
ax1.set_title('A threshold can trade autonomy for review cost')
plt.tight_layout()
plt.show()

Best threshold by expected value
threshold                   0.440
deferral_rate               0.114
mean_success_probability    0.662
stake_weighted_success      1.267
mean_expected_value         1.225

In this synthetic world, the best threshold is not the one that maximizes automation. It is also not necessarily the one that maximizes average correctness. The boundary depends on the task mix, the cost of review, and the value of preventing high-stakes failures.

That is the cybernetic part: the deferral rule is a control loop. The signal is confidence; the actuator is whether the agent acts or asks; the objective is a system-level criterion that includes both machine performance and human effort.

Adding a human preference to intervene

A second complication: people may want to intervene for reasons that are not visible in a scalar confidence score. High stakes can create a desire for review even when the model is fairly confident.

Here we add a synthetic intervention preference. Then we compare three policies:

1. confidence threshold: defer only when confidence is low;
2. value of review: defer when expected human improvement is worth the review cost;
3. preference-calibrated: use value of review, but also defer when the human is likely to want intervention.

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

# Synthetic probability that a human wants the chance to intervene.
intervention_pref_p = sigmoid(-3.2 + 3.0 * difficulty + 0.75 * np.log1p(stakes) + 1.2 * (1.0 - ai_conf))
wants_intervention = rng.random(n) < intervention_pref_p

best_threshold = float(best['threshold'])
policy_threshold = ai_conf < best_threshold
policy_value_of_review = stakes * (human_p - ai_p) > human_cost
policy_preference_calibrated = policy_value_of_review | (intervention_pref_p > 0.65)

policies = {
    'confidence threshold': policy_threshold,
    'value of review': policy_value_of_review,
    'preference-calibrated': policy_preference_calibrated
}

agency_penalty = 0.20
summary_rows = []
for name, defer in policies.items():
    success_p = np.where(defer, human_p, ai_p)
    missed_preference = wants_intervention & (~defer)
    expected_value_with_agency = stakes * success_p - human_cost * defer - agency_penalty * stakes * missed_preference
    summary_rows.append({
        'policy': name,
        'deferral_rate': defer.mean(),
        'success_probability': success_p.mean(),
        'missed_intervention_rate': missed_preference.mean(),
        'expected_value_with_agency': expected_value_with_agency.mean()
    })

summary = pd.DataFrame(summary_rows)
print(summary.round(3).to_string(index=False))

fig, axes = plt.subplots(1, 2, figsize=(10, 4.2))

axes[0].bar(summary['policy'], summary['deferral_rate'], color=['tab:orange', 'tab:purple', 'tab:green'])
axes[0].set_ylim(0, 1)
axes[0].set_ylabel('deferral rate')
axes[0].set_title('How often the agent steps aside')
axes[0].tick_params(axis='x', rotation=20)

axes[1].bar(summary['policy'], summary['missed_intervention_rate'], color=['tab:red', 'tab:red', 'tab:red'], alpha=0.75)
axes[1].set_ylim(0, 1)
axes[1].set_ylabel('missed intervention rate')
axes[1].set_title('How often human preference is ignored')
axes[1].tick_params(axis='x', rotation=20)

plt.tight_layout()
plt.show()

               policy  deferral_rate  success_probability  missed_intervention_rate  expected_value_with_agency
 confidence threshold          0.114                0.662                     0.305                       1.095
      value of review          0.066                0.657                     0.338                       1.128
preference-calibrated          0.117                0.665                     0.301                       1.134

What the toy model is good for

This is a deliberately small model, but it makes one design tension visible:

• If we optimize only for machine confidence, we may miss cases where human review is valuable or desired.
• If we optimize only for human intervention, we can over-defer and consume scarce attention.
• A useful human-AI system needs a policy for when agency transfers between machine and person.

The interesting part is not the particular threshold above. It is the need to treat deferral as an interface and control problem, not just a model performance problem.

For the selected source, see the CMU ML Blog listing: When Should AI Step Aside?: Teaching Agents When Humans Want to Intervene.