Prompt Caching Across Providers

Lesson 16 cached whole responses. This lesson caches something narrower but often more valuable in practice: the static, repeated portion of a request’s input, a long system prompt, a large tool-definition schema, a big pasted document used across many turns, so the model doesn’t have to reprocess that unchanged content on every single call. This is provider-native functionality, not something a gateway builds itself, and the two major implementations, Anthropic’s and OpenAI’s, work in genuinely different ways.

Anthropic: Explicit Cache Breakpoints

Anthropic’s API requires you to mark exactly which content should be eligible for caching, using a cache_control field on a message or content block:

from anthropic import Anthropic

client = Anthropic(api_key="sk-ant-...")

LONG_SYSTEM_PROMPT = """You are a customer support agent for a logistics company.
[... several thousand tokens of policy documentation, tone guidelines,
and tool-use instructions that are identical on every single call ...]"""

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=500,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"},   # marks this block as a cache breakpoint
        }
    ],
    messages=[{"role": "user", "content": "Where is order #48213?"}],
)

# Verify caching actually engaged by reading the usage block
print(response.usage.cache_creation_input_tokens)  # non-zero on the first call: cache written
print(response.usage.cache_read_input_tokens)       # non-zero on later calls: cache reused, billed cheaper

The default cache TTL is 5 minutes from the last use; a longer-lived cache is available by setting "ttl": "1h" inside the cache_control object for content that stays static for longer, at a different pricing tier. Up to four cache breakpoints are allowed per request, meant to be placed at the end of static content blocks (a system prompt, a tool schema, a large static document), not mid-sentence inside content that actually varies between calls. Pricing reflects the trade being made: the first call that writes a cache entry pays roughly a 1.25x premium on those tokens versus normal input pricing, while every subsequent call within the TTL that reads from that entry pays roughly 0.1x, a 90% discount, which is why caching only pays off for content reused across enough calls to amortize that initial write premium.

OpenAI: Automatic, No Code Change Required

OpenAI’s prompt caching activates automatically for any request whose prompt exceeds an internal minimum length (1,024 tokens as of the current implementation), with no cache_control-style annotation needed:

from openai import OpenAI

client = OpenAI(api_key="sk-...")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": LONG_SYSTEM_PROMPT},  # same long, static prompt as above
        {"role": "user", "content": "Where is order #48213?"},
    ],
)

# OpenAI reports cached-token usage in a nested field rather than top-level
print(response.usage.prompt_tokens_details.cached_tokens)

The trade-off is the mirror image of Anthropic’s: no explicit annotation is required and there is nothing to misconfigure, but there is also no fine-grained control over exactly which portion of the prompt is treated as cacheable, OpenAI’s implementation matches on a prefix basis automatically.

How a Gateway Normalizes This

flowchart TD
    Req["Gateway request:\nlong static system prompt + short user turn"] --> Adapter{"Which provider?"}
    Adapter -->|Anthropic| A["Adapter inserts cache_control\non the static system block"]
    Adapter -->|OpenAI| O["No annotation needed;\nOpenAI caches automatically\nabove its length threshold"]
    A --> Metrics["Normalize cache_creation/cache_read\nand cached_tokens into one\ncanonical 'cached_input_tokens' field\nfor cost tracking (Lesson 18)"]
    O --> Metrics

    style Req fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style A fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style O fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style Metrics fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

This is exactly the adapter pattern from Lesson 6 doing real work: a gateway’s Anthropic adapter can automatically insert a cache_control breakpoint at the end of a long, static system prompt whenever one is detected in the canonical request, so callers get the cost benefit without needing to know Anthropic-specific syntax, while the OpenAI adapter passes the request through unmodified since no annotation is needed there. Both providers’ differently-shaped usage fields, cache_creation_input_tokens/cache_read_input_tokens versus prompt_tokens_details.cached_tokens, get normalized into one canonical field the gateway’s cost-tracking layer (Lesson 18) can reason about uniformly, regardless of which provider actually served the request. With both caching layers covered, response caching that skips a call entirely and prompt caching that discounts part of a real call, Lesson 18 turns to tracking and enforcing spend directly: token budgets, cost attribution, and chargeback across teams.