Semantic & Task-Based Model Selection

Lessons 10 and 11 both assumed the pool of candidate deployments was already fixed, static routing picked among them by priority, dynamic routing picked among them by live performance. This lesson adds a decision that happens earlier: which pool is even the right one for this specific request, based on what the request actually asks for. Not every prompt needs a frontier model, and routing every prompt to one anyway is a real, measurable cost most teams are overpaying.

The Idea: Classify, Then Route

flowchart TD
    Prompt["Incoming prompt"] --> Classifier["Task classifier\n(small, fast model or heuristics)"]
    Classifier -->|"simple_faq / extraction"| CheapPool["Cheap pool:\nself-hosted Llama, Groq"]
    Classifier -->|"complex_reasoning / code"| CapablePool["Capable pool:\nclaude-sonnet, gpt-4o"]
    Classifier -->|"low confidence"| CapablePool

    style Prompt fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style Classifier fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style CheapPool fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style CapablePool fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

The classifier itself should be cheap and fast, its cost is added to every single request, so it needs to be far smaller than the decision it’s making room for. A common, pragmatic choice is a small open-weight model running through Groq (verified in Lesson 5, extremely low latency for short classification calls) or a self-hosted model via Ollama, not a frontier model.

Building the Classifier

import json
from openai import OpenAI

# A small, fast model dedicated to classification, not the task itself.
# Groq's inference speed keeps this call's added latency in the tens of
# milliseconds, small relative to the savings on the routed call.
classifier_client = OpenAI(
    api_key="$GROQ_API_KEY",
    base_url="https://api.groq.com/openai/v1",
)

CLASSIFY_PROMPT = """Classify the user's request into exactly one category:
- simple_faq: a factual lookup, definition, or short direct answer
- complex_reasoning: multi-step reasoning, analysis, or judgment call
- code_generation: writing or debugging code

Respond with strict JSON: {{"category": "...", "confidence": 0.0-1.0}}

Request: {request}"""

def classify_task(user_request: str) -> dict:
    response = classifier_client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": CLASSIFY_PROMPT.format(request=user_request)}],
        temperature=0,
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

classify_task("What's your return policy for items bought over 30 days ago?")
# -> {"category": "simple_faq", "confidence": 0.94}

classify_task("Given these three vendor contracts, which has the most exposure to currency risk?")
# -> {"category": "complex_reasoning", "confidence": 0.88}

Wiring the Classification into Gateway Routing

With a classification and confidence score in hand, the routing decision becomes a lookup, with a confidence floor that defaults anything uncertain to the safer, more capable pool:

CONFIDENCE_FLOOR = 0.75

MODEL_ALIAS_BY_CATEGORY = {
    "simple_faq": "cheap-pool",              # self-hosted Llama / Groq, configured in the gateway's model_list
    "complex_reasoning": "capable-pool",     # claude-sonnet, gpt-4o
    "code_generation": "capable-pool",
}

def route_request(user_request: str, gateway_client: OpenAI) -> str:
    classification = classify_task(user_request)
    category = classification["category"]
    confidence = classification["confidence"]

    # Low-confidence classifications default to the capable pool: the cost
    # of misrouting a hard request to a weak model outweighs the savings
    # from misrouting an easy request to a strong one.
    if confidence < CONFIDENCE_FLOOR:
        target_alias = "capable-pool"
    else:
        target_alias = MODEL_ALIAS_BY_CATEGORY.get(category, "capable-pool")

    response = gateway_client.chat.completions.create(
        model=target_alias,
        messages=[{"role": "user", "content": user_request}],
    )
    return response.choices[0].message.content

target_alias here is exactly the model_name alias pattern from Module 3, cheap-pool and capable-pool are each backed by their own model_list entries in the gateway’s config, and within each pool, the load-balancing and dynamic routing strategies from Lessons 9 and 11 still apply. Task-based routing decides which pool; the router’s routing_strategy decides which deployment within that pool. The two compose rather than compete.

The Real Risk: Misclassification

The confidence floor above exists because a classifier is never perfectly accurate, and the asymmetry of misrouting matters: sending an easy request to a strong model wastes some money; sending a genuinely hard request to a weak model can produce a wrong or unhelpful answer a user actually sees. Setting CONFIDENCE_FLOOR conservatively, and periodically reviewing cases the classifier scored just above and below it, is what keeps task-based routing’s savings from coming at the cost of a quality regression nobody notices until a customer complains. With Module 4’s three routing strategies covered, static, dynamic, and task-based, Module 5 turns to what happens when routing decisions aren’t enough on their own, when a chosen deployment fails outright and the gateway needs to recover reliably.