Retries, Timeouts & Circuit Breakers

Module 4 decided which deployment should serve a request. This lesson covers what happens when that deployment, having been correctly chosen, fails anyway, a transient network blip, a provider-side 500, an outright outage, and the three mechanisms that keep a single failed call from becoming a cascading production incident: retries, timeouts, and circuit breakers.

Retries: Exponential Backoff with Jitter

A naive retry, fail, wait one fixed second, try again, has a specific failure mode at scale: if a provider is struggling and returns errors to many concurrent callers at once, and every one of them retries after exactly the same fixed delay, they all hit the provider again in a synchronized burst, making the original problem worse instead of better. The standard fix is exponential backoff (each retry waits longer than the last) combined with jitter (a small random variation so concurrent callers don’t retry in lockstep):

import random
import time
from openai import APIError, APITimeoutError

def call_with_backoff(client, max_retries: int = 3, base_delay: float = 0.5, **kwargs):
    for attempt in range(max_retries + 1):
        try:
            return client.chat.completions.create(**kwargs)
        except (APIError, APITimeoutError) as e:
            if attempt == max_retries:
                raise  # exhausted retries, let the fallback chain (Lesson 14) take over
            # Exponential backoff: 0.5s, 1s, 2s... plus up to 25% random jitter
            # so many concurrent callers don't all retry at the exact same instant
            delay = base_delay * (2 ** attempt)
            delay += random.uniform(0, delay * 0.25)
            time.sleep(delay)

LiteLLM’s router already implements this internally via its num_retries setting (used since Lesson 7’s config), this implementation is shown to make the mechanism explicit, since it is exactly what a well-behaved retry loop needs to do underneath any gateway’s config surface.

Timeouts That Fit LLM Response Distributions

A typical REST API’s response time is fairly tight and independent of the request payload. An LLM call is not: token generation is roughly linear in output length, so a request capped at max_tokens: 50 and a request capped at max_tokens: 2000 from the identical deployment can differ in response time by an order of magnitude. A single global timeout constant is therefore wrong for at least one of them.

def timeout_for_request(max_tokens: int, tokens_per_second_estimate: float = 40) -> float:
    # A rough floor: expected generation time plus a fixed buffer for
    # network and provider queueing overhead. Tune tokens_per_second_estimate
    # per deployment from observed data (Module 8 covers measuring this for real).
    expected_generation_seconds = max_tokens / tokens_per_second_estimate
    return expected_generation_seconds + 5.0  # fixed buffer

client.chat.completions.create(
    model="production-chat",
    messages=[...],
    max_tokens=2000,
    timeout=timeout_for_request(max_tokens=2000),  # scales with the actual request, not a flat constant
)

Circuit Breakers: Stop Hitting Something That’s Already Down

Retries and timeouts handle one failing call. A circuit breaker handles the pattern across many calls: once a deployment has failed enough times in a row, further requests are short-circuited immediately rather than each one paying the full retry-and-timeout cost against something that is very likely still broken.

import time
from enum import Enum

class CircuitState(Enum):
    CLOSED = "closed"        # normal operation, requests flow through
    OPEN = "open"             # too many recent failures, block requests immediately
    HALF_OPEN = "half_open"   # cooldown elapsed, allow one test request through

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, cooldown_seconds: float = 30):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self.state = CircuitState.CLOSED
        self.consecutive_failures = 0
        self.opened_at: float | None = None

    def before_call(self):
        if self.state == CircuitState.OPEN:
            if time.time() - self.opened_at >= self.cooldown_seconds:
                self.state = CircuitState.HALF_OPEN  # let one request test recovery
            else:
                raise RuntimeError("circuit open: deployment assumed still down, skipping call")

    def on_success(self):
        self.consecutive_failures = 0
        self.state = CircuitState.CLOSED

    def on_failure(self):
        self.consecutive_failures += 1
        if self.consecutive_failures >= self.failure_threshold:
            self.state = CircuitState.OPEN
            self.opened_at = time.time()

stateDiagram-v2
    [*] --> CLOSED
    CLOSED --> OPEN: consecutive failures >= threshold
    OPEN --> HALF_OPEN: cooldown elapsed
    HALF_OPEN --> CLOSED: test request succeeds
    HALF_OPEN --> OPEN: test request fails

A circuit breaker and a fallback chain (Lesson 14) solve different problems and gateways need both: the circuit breaker decides whether to even try a specific deployment right now, sparing it (and the caller) the cost of a doomed attempt; the fallback chain decides what to do instead once a deployment is unavailable, whether by circuit breaker or by direct failure. In production, before_call() gates every attempt at a deployment, and a request that finds the circuit open moves immediately to the next deployment in the fallback chain rather than wasting a retry-and-timeout cycle on something already known to be down. With individual-call resilience covered, Lesson 14 goes one level up: designing the fallback chain itself, including what “graceful” degradation actually means when the fallback model answers differently than the primary would have.