Guardrails, Escalation & Human Review

No decision system is right all the time. The question is what happens when it isn’t. This lesson covers the controls around the decision itself: which decisions run automatically, which go to people, how people’s corrections flow back, and what stops the system when something breaks.

Tiered autonomy

Not every decision deserves the same level of automation. Assign each decision type (and sometimes each case) to a tier based on stakes (how much an error costs, how reversible it is) and certainty (how sure the system is):

The tier isn’t fixed per decision type. A fraud system can auto-approve 95% of transactions, auto-block the obvious 1%, and send the remaining 4% to review, with the split determined by calibrated probabilities and the costs of each error. That’s the conformal set-size idea from Lesson 6 applied operationally: singletons automate, ambiguous sets escalate.

Capacity-aware escalation

Review capacity is a scarce resource, which makes “which cases go to review?” an optimization problem. The value of reviewing a case is roughly:

value_of_review = P(automated decision is wrong)
                  × cost of that error
                  × P(reviewer catches and fixes it)
                  − cost of the review itself

Rank cases by this and fill the available capacity:

import numpy as np

def select_for_review(p_wrong, error_cost, catch_rate, review_cost, capacity):
    """All arrays per case. catch_rate can vary by case type,
    estimated from seeded test cases (see below)."""
    value = p_wrong * error_cost * catch_rate - review_cost
    ranked = np.argsort(-value)
    chosen = [i for i in ranked[:capacity] if value[i] > 0]
    return np.array(chosen, dtype=int)

Two things fall out of this. Cases where the model is uncertain but the stakes are tiny shouldn’t go to people. And high-stakes cases can merit review even at modest uncertainty. When capacity changes (a reviewer is sick, a holiday spike), the system adapts automatically. And just as with budgets in Lesson 7, the marginal value of the last reviewed case tells you what an extra reviewer is worth.

Designing the review experience

How reviewers see a case changes what they decide.

• Evidence first, recommendation second. Showing the system’s recommendation first anchors reviewers. Show the evidence (extracted facts, key features, the LLM’s quoted spans) and let them form a view, then reveal the recommendation.
• Reason codes and top drivers. Show why the system leaned the way it did, in business terms.
• Make the safe action easy. If the reviewer is unsure, the default should be the conservative option or “request more information”, not “approve”.
• Measure time per case. A reviewer spending four seconds on a complex case isn’t reviewing it.

Automation bias and seeded cases

Humans tend to over-trust automated recommendations, especially when they’re usually right. A 99.6% agreement rate might mean the model is excellent, or that reviewers have stopped checking. You can’t tell from the agreement rate alone.

Measure reviewer effectiveness with seeded cases: occasionally insert cases where the right answer is known and the system’s recommendation is deliberately wrong. The share that reviewers catch is the catch_rate in the formula above. If it’s low, review isn’t providing the protection it’s assumed to, and either the process or the tooling needs to change. Keep seeded cases a small share of the queue and make sure they can never result in real actions.

Overrides are data

Every time a reviewer changes the system’s decision, record it in structured form:

from dataclasses import dataclass

@dataclass
class Override:
    decision_id: str
    system_action: str
    final_action: str
    reviewer_id: str
    reason_code: str     # e.g. MISSING_INFO, MODEL_IGNORED_RECENT_EVENT, POLICY_EXCEPTION
    note: str            # free text for context
    seconds_spent: float

Then use them:

• Aggregate by reason code each week. A cluster of “model ignored recent payment” overrides is a feature engineering task. A cluster of “policy exception” overrides is a rules conversation with the policy owner.
• Use as labels, carefully. Overrides are labels on a selected, non-random subset, so treat them as a signal for where to look rather than an unbiased training set.
• Track per reviewer. Very high or very low override rates compared with peers are worth a conversation, in both directions.

Guardrails after the decision

The last line of defence checks decisions after they’re made and before they take effect. Guardrails catch failures the components can’t see in themselves: bad data feeds, broken deployments, attacks.

• Volume and mix limits. Alert or halt if the share of any action moves far outside its normal range. If the 25% discount normally goes to 8% of customers and today it’s 40%, something is broken.
• Budget and exposure caps. Hard limits on total spend, total credit extended, or total items reserved per hour, independent of the optimizer.
• Consistency checks. Contradictory decisions for the same customer across channels, actions conflicting with recent decisions, decisions for units that don’t exist.
• Input sanity checks. Feature values outside training ranges, missing upstream data, a sudden shift in score distributions.
• Circuit breakers. When a guardrail trips, fall back automatically to a known-good policy: the previous model version, a rules-only path, or “no action”. Decide in advance what the fallback is for each decision; designing it during an incident is too late.
• Kill switch. A single, tested, documented way for the decision owner to stop automated decisions for a decision type, without a deployment.

flowchart LR
    D["Proposed decisions"] --> G{"Guardrails\nvolume, caps,\nconsistency, inputs"}
    G -->|"pass"| A["Execute"]
    G -->|"single case flagged"| H["Human review queue"]
    G -->|"systemic anomaly"| CB["Circuit breaker:\nknown-good fallback\n+ page owner"]

    style D fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style G fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style A fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style H fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style CB fill:#fef2f2,stroke:#ef4444,color:#0F172A

Test guardrails the way you test backups: deliberately trip them in staging and in a controlled production drill, and confirm the fallback works and the right people are alerted.

That completes the decision stack itself. Module 7 asks the question every stakeholder eventually asks: is it working?