Token Budgets, Cost Tracking & Chargeback

Lesson 1 opened this course with a concrete failure: “nobody can answer how much the marketing team spent on Claude last month without grepping logs across a dozen repos.” This lesson closes that loop for real: enforcing hard budgets so spend never silently blows past a cap, computing real per-request cost from actual usage, and rolling that up into the report that actually answers the question.

Enforcing a Hard Budget

Lesson 7 introduced max_budget when generating a virtual key. The enforcement side matters as much as the setting:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{
    "models": ["production-chat"],
    "max_budget": 50,
    "budget_duration": "30d",
    "metadata": {"team": "fraud-detection"}
  }'

def check_budget_before_call(key_id: str, estimated_cost: float) -> bool:
    spend_so_far = get_current_spend(key_id)   # tracked by the gateway per key
    budget = get_key_budget(key_id)
    # Reject before the provider call, not after: the whole point of a
    # hard budget is that spend never actually exceeds the cap, rather
    # than being reported as having exceeded it after the fact.
    if spend_so_far + estimated_cost > budget:
        raise PermissionError(f"key {key_id} would exceed its ${budget} budget")
    return True

This is why the budget check sits at lifecycle stage 3, before routing and before any provider is contacted (Lesson 4), rejecting an over-budget request there costs nothing; rejecting it after the provider has already answered means the money is already spent regardless of what the gateway does next.

Computing Real Cost

Cost can only be computed accurately after the provider responds with real token usage, which is why it belongs at egress, and why Lesson 17’s cached-token discount has to be accounted for explicitly rather than treating every input token as equally priced:

# Simplified pricing table: $ per million tokens. Real gateways (LiteLLM
# ships one) maintain this per exact model string, updated as providers
# change pricing.
PRICING = {
    "claude-sonnet-4-5": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
    "gpt-4o":             {"input": 2.50, "output": 10.00, "cached_input": 1.25},
    "hosted_vllm/meta-llama/Llama-3.1-8B-Instruct": {"input": 0.00, "output": 0.00, "cached_input": 0.00},
}

def compute_cost(model: str, prompt_tokens: int, completion_tokens: int, cached_tokens: int = 0) -> float:
    rates = PRICING[model]
    billable_prompt_tokens = prompt_tokens - cached_tokens
    cost = (
        billable_prompt_tokens / 1_000_000 * rates["input"]
        + cached_tokens / 1_000_000 * rates["cached_input"]
        + completion_tokens / 1_000_000 * rates["output"]
    )
    return round(cost, 6)

# Egress-stage call, after the real provider response is in hand
cost = compute_cost(
    model="claude-sonnet-4-5",
    prompt_tokens=4200,
    completion_tokens=180,
    cached_tokens=3800,   # from response.usage.cache_read_input_tokens, Lesson 17
)

The self-hosted vLLM deployment’s $0.00 rate above is deliberate: its real cost is the fixed GPU spend from Lesson 8, not a per-token provider bill, so it shows as free in this per-request accounting even though it isn’t actually free to the business, a nuance worth a comment in any real pricing table so nobody misreads it as “self-hosting is literally costless.”

Logging for Chargeback

Every request’s cost needs to land in a log tagged with the virtual key’s team metadata, exactly the metadata set at /key/generate time in Lesson 7:

def log_request_cost(key_metadata: dict, model: str, cost: float, timestamp: str):
    cost_log_entry = {
        "team": key_metadata["team"],
        "model": model,
        "cost_usd": cost,
        "timestamp": timestamp,
    }
    write_to_cost_log(cost_log_entry)  # append to a table/log store, e.g. Postgres, ClickHouse

The Chargeback Report

-- Monthly per-team spend: the concrete answer to Lesson 1's original problem
SELECT
    team,
    model,
    SUM(cost_usd) AS total_cost,
    COUNT(*) AS request_count
FROM gateway_cost_log
WHERE timestamp >= date_trunc('month', now())
GROUP BY team, model
ORDER BY total_cost DESC;

flowchart LR
    Req["Request via virtual key\n(team: fraud-detection)"] --> Egress["Egress: compute real cost\nfrom usage + pricing table"]
    Egress --> Log["Cost log entry\n{team, model, cost, timestamp}"]
    Log --> Report["Monthly chargeback report\nGROUP BY team"]

    style Req fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style Egress fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style Log fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style Report fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

This single query, grouping a cost log by the team metadata every virtual key already carries, is the entire mechanism behind the “which team spent what” answer that was architecturally impossible with direct SDK calls in Lesson 1. It costs nothing extra to maintain once budgets and cost logging are in place, it is a byproduct of enforcing budgets correctly. Module 7 builds the remaining governance layer on top of this same virtual-key foundation: role-based access, PII redaction, and audit logging.