Capstone: A Production Retention Offer Engine

This capstone brings every module together. You’ll design and build a retention offer engine for a subscription business, replacing the rule from Lesson 1 (“20% discount to everyone with churn probability above 0.6”) with a decision system that is calibrated, causal, constrained and accountable.

Work through it as a project. Each step names the lesson it draws on and the artifact you should produce. By the end you’ll have a design document, working code for each layer, an evaluation report and a rollout plan.

The brief

A subscription business has about 400,000 active subscribers on monthly and annual plans. Retention offers are currently sent by the rule above, around 40,000 a month. Finance believes the programme loses money; the retention team believes it’s essential. Nobody can settle the argument with data.

You have:

• 24 months of subscriber history: plan, tenure, usage, billing, support tickets (free text), cancel-page visits.
• Logs of past offers (who got what, when), but no propensities and no holdout.
• A budget of $120,000 per month for offer costs, and 600 account-manager calls per month.
• Five offers: none, 10% for 3 months, 25% for 3 months, free upgrade for a month, account-manager call (annual plans above $2,000 only).
• Policies: at most one discount per customer per rolling 12 months; no offers to accounts in collections.
• Decisions run in a monthly batch plus in real time when a subscriber visits the cancel page (latency budget 100 ms).

Architecture

flowchart TB
    subgraph Inputs
        D["Subscriber data\n(point-in-time)"]
        TK["Support tickets"] --> LLM["LLM signal extraction\n(Lesson 17)"]
    end
    D --> RU["Eligibility rules\ndecision table v1 (Lesson 16)"]
    LLM --> SC
    D --> SC["Calibrated uplift per offer\n(Lessons 4, 11, 12)"]
    RU --> VAL["Expected incremental margin\nper customer x eligible offer (Lesson 5)"]
    SC --> VAL
    VAL --> OPT["Monthly allocation\nbudget + call cap (Lesson 7)\npublishes lambda"]
    VAL --> RT["Real-time cancel-page path\nvalue - lambda x cost (Lesson 23)"]
    OPT --> RT
    OPT --> EX["Exploration + 5% holdout\n(Lessons 10, 13, 14)"]
    RT --> EX
    EX --> GR["Guardrails + escalation\n(Lesson 18)"]
    GR --> LOG["Decision log\n(Lesson 23)"]
    LOG --> EV["Scorecard, OPE, drift,\nfairness (Lessons 15, 19, 21, 22)"]

    style RU fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style GR fill:#fef2f2,stroke:#ef4444,color:#0F172A
    style LOG fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style OPT fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SC fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style VAL fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style EV fill:#EEF0F7,stroke:#6366F1,color:#0F172A

Step 1: Specify the decision (Lesson 2)

Write the decision specification: actions, objective (12-month net margin), hard and soft constraints, cadence, feedback signals and delay, and the current baseline. Get the retention lead, finance and CX to agree on the trade-off weights, notably the cost per unwanted contact.

Artifact: decision_spec.yaml, signed off by the decision owner.

Step 2: Fix the data foundation (Lessons 10, 23)

Before any modeling:

• Build point-in-time training data: features as they were on each historical decision date, outcomes from the following 90 days and 12 months.
• Reconstruct the old policy’s targeting from the logs and draw the DAG: what drove offers historically (churn score, cancel-page visits, account manager discretion)? Interview the retention team.
• Start the decision log and the 5% randomized holdout now, even before the new engine exists. Every month you wait is a month of unmeasurable decisions.

Artifacts: feature pipeline with time-travel joins; DAG diagram; decision log schema in production for the current rule.

Step 3: Eligibility rules (Lesson 16)

Implement the eligibility table with a UNIQUE hit policy, reason codes and a fail-closed evaluator. Write table-driven tests with the policy owner, plus a completeness check across the input domain.

Artifacts: eligibility_v1 table; tests; a report of how many subscribers fall under each rule.

Step 4: Signals from support tickets (Lesson 17)

Extract cancellation reason, competitor mentions, discount requests and frustration level from recent tickets with a schema-validated LLM component. Label 500 tickets with the retention team, evaluate per-field accuracy, and fit a calibrator for the reason classification. Run extraction in batch when tickets arrive and store signals as features; the real-time path reads the stored signals and never calls the LLM.

Artifacts: extraction prompt and schema; evaluation report; signal features in the feature store.

Step 5: Estimate uplift per offer (Lessons 4, 11, 12)

This is the hardest step, because historical offers weren’t randomized.

1. Run a randomized pilot if the business will allow it: for six to eight weeks, assign offers at random among eligible actions to a slice of subscribers. This gives clean data for the main model.
2. In parallel, estimate effects from historical logs with double machine learning / causal forests, adjusting for the confounders in your DAG. Check overlap, run refutation tests, and compare against the pilot where they overlap.
3. Train an X-learner (or causal forest) per offer against “none”, and calibrate the predicted retention probabilities under each action.
4. Evaluate with Qini curves and uplift by decile on held-out pilot data.

Artifacts: uplift models with Qini report; comparison of observational vs randomized estimates; calibration plots.

Step 6: Compute expected incremental value (Lesson 5)

For each subscriber and each eligible offer:

def incremental_value(c, offer, p_retain_none, p_retain_offer, horizon_margin):
    uplift = p_retain_offer - p_retain_none
    discount_paid = p_retain_offer * offer.discount_value(c)   # sure things pay too
    return (uplift * horizon_margin(c)
            - discount_paid
            - offer.contact_cost
            - ANNOYANCE_COST * offer.is_contact)

Apply shrinkage to uplift estimates for small segments (Lesson 8), and keep the conformal or causal-forest interval for each estimate to drive exploration in Step 8.

Artifact: a values[customer, offer] table for the month, with intervals.

Step 7: Allocate the budget (Lessons 7, 9)

Solve the monthly allocation: maximize total incremental value subject to the $120k budget, the 600-call cap and one action per subscriber. With 400k subscribers, use the Lagrangian threshold for the budget and a small MIP or greedy pass for the call cap. Record λ (the budget’s shadow price) and the call cap’s shadow price, and report both to finance: they answer “should the budget be bigger?” and “should we hire another account manager?”.

Stress the plan with scenarios: what happens if uplift is 30% lower than estimated? Is the plan still profitable?

Artifacts: monthly plan; shadow prices; scenario results.

Step 8: Exploration and holdout (Lessons 10, 13, 14)

• Keep the 5% holdout: chosen at random, decided by the baseline policy (no offer, or the old rule if the business insists), never optimized.
• For the remaining 95%, add bounded exploration: subscribers whose uplift interval straddles zero for an offer get that offer with a small probability; every eligible action keeps a floor probability of 2%.
• Compute and log the propensity of every chosen action.
• On the real-time cancel-page path, choose the eligible action maximizing value − λ × cost, with the same exploration rules.

Artifact: policy code producing actions and propensities; verification that propensities sum to 1 across eligible actions.

Step 9: Guardrails and escalation (Lesson 18)

Add post-decision guardrails: action mix limits (25% offers at most 15%), daily spend cap, consistency with recent decisions, input sanity checks. Define the fallback (rules-only, no discount) and the kill switch. Route account-manager-call candidates with low certainty to a review queue, ranked by expected value of review.

Artifacts: guardrail config; fallback policy; a drill showing a tripped guardrail falls back correctly.

Step 10: Bundle, log and replay (Lesson 23)

Package rules, models, calibrators, extraction prompt version, objective parameters, exploration settings and guardrails as policy bundle v1. Every decision record carries the bundle version, eligible actions, scores for all of them, the chosen action, propensity, holdout flag and reason codes. Prove that you can replay last week’s decisions deterministically.

Artifacts: bundle manifest; decision log populated; replay test.

Step 11: Evaluate before rollout (Lessons 15, 24)

• OPE: the old rule had no propensities, so treat its decisions as deterministic and rely on the pilot and on any period with randomization. Report DR and SNIPS estimates with confidence intervals, effective sample size and the share of decisions without support. Be explicit about what OPE couldn’t evaluate.
• Simulation: calibrate a simple subscriber simulator to the last 12 months, validate it on the pilot, then stress test the new policy against churn shocks, weaker offers, faster fatigue and cancel-page gaming.
• Fairness: check action rates and incremental value by region, age band and other sensitive groups available for audit (Lesson 22).

Artifact: evaluation report with a clear recommendation and the risks it depends on.

Step 12: Roll out (Lessons 20, 21)

1. Shadow mode for two weeks: compare action mix, spend pacing, latency and guardrail hits with the current rule.
2. Canary at 5% of subscribers.
3. Experiment: 50/50 between old rule and new engine with the budget split in proportion, CUPED on prior-period margin, 90-day retention and net margin as primary metric, complaints and unsubscribes as guardrails, fixed horizon.
4. Rollout with the 5% holdout permanent.
5. Monitoring: input PSI, score distributions, action mix, λ, spend pacing, override rates, leading indicators, then calibration and holdout value as outcomes arrive.

Artifacts: experiment design document; monitoring dashboard; runbook.

Step 13: Present it (Lesson 19)

Prepare a decision scorecard for the owner: incremental value against holdout with intervals, regret against the best known policy, cost per retained subscriber, automation and override rates, guardrail trips, fairness metrics, and the shadow prices. Show the assumptions the result depends on, and the monitoring that would catch it if they stopped holding.

That presentation settles the argument from the brief. Finance gets a measured incremental value. The retention team gets a system that targets persuadables and shows what more budget would buy. Both get a holdout that keeps measuring.

Checklist

• Decision spec agreed with owner, including trade-off weights
• Point-in-time features; decision log and holdout live
• Eligibility table with tests and reason codes
• LLM signals evaluated and calibrated; batch-only, with fallback
• Uplift per offer, calibrated, validated with Qini on randomized data
• Incremental values with shrinkage and intervals
• Allocation with shadow prices; scenario results
• Exploration floors, holdout, propensities logged
• Guardrails, fallback, kill switch, review queue
• Policy bundle versioned; replay works
• OPE, simulation stress tests and fairness review
• Shadow, canary, experiment with split budget; monitoring live
• Scorecard presented; quarterly decision review scheduled

Where to go next

• Multi-period policies: model offer fatigue explicitly and plan offers over a sequence of months, moving towards reinforcement learning with the simulator as a testing ground.
• Price and offer design: treat discount depth and duration as continuous decisions optimized jointly with targeting.
• Cross-programme coordination: retention, upsell and win-back programmes competing for the same customers, budget and contact limits, solved as one allocation.

The tools carry over unchanged to credit limits, fraud review, inventory, pricing, collections and claims triage. In every one of them, the model is one input. The decision is what you’re building.