Query Fan-Out

Query fan-out is the technique of taking one query and deliberately turning it into many. Instead of embedding or keyword-matching the user’s words as written and retrieving once, a fan-out system generates a set of related sub-queries, runs them concurrently against one or more indexes or the open web, and then merges the many result lists into a single ranked set that feeds the final answer. The name became widely used in 2025 when Google described AI Mode as running a “query fan-out technique, issuing multiple related searches concurrently across subtopics and multiple data sources,” but the same idea had already been circulating in retrieval-augmented generation circles as multi-query retrieval and RAG-Fusion. It is best understood as the widest, most parallel member of the query rewriting family: where a single reformulation swaps one query for a better one, fan-out swaps one query for a small population of them and pays for the extra retrieval to cover more of the corpus.

Why One Query Is Not Enough

A single query is a single point in whatever space the retriever searches. In dense retrieval it is one vector; in sparse retrieval it is one bag of terms. Either way, it can only be close to documents that were written with compatible vocabulary and framing. Real information needs are rarely that tidy. A question like “how do we handle refunds for annual plans?” implicitly spans the refund policy document, the billing service’s proration logic, the finance team’s revenue-recognition notes, and the support team’s canned responses, and each of those was authored by different people using different words. Retrieving once with the user’s phrasing tends to surface whichever of those happens to share the most surface vocabulary and miss the rest.

Fan-out attacks this by covering the need from several directions at once:

• Sub-topic decomposition. The question is split into the distinct facets it actually contains, so each facet gets its own targeted retrieval rather than competing for slots in one result list.
• Implicit angles. A fan-out planner adds queries the user did not type but almost certainly wants, such as edge cases, prerequisites, and recent changes.
• Vocabulary spread. The same facet is phrased several ways (“refund”, “cancellation credit”, “money back”) so that documents indexed under any of those terms have a chance to surface.
• Comparative and temporal variants. Queries like “annual plan refund vs monthly” or “refund policy changes 2025” pull in framing the original never expressed.

None of this is free, and the cost is the mechanism, not overhead to trim: fan-out helps precisely because it retrieves many times instead of once.

How Query Fan-Out Works

graph TD
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    Q([Raw user query]):::data --> GATE{Complex enough<br/>to fan out?}:::process
    GATE -->|No| ONE[Single retrieval]:::process
    GATE -->|Yes| PLAN[LLM planner:<br/>expand into N sub-queries]:::process
    PLAN --> S1[Sub-query 1<br/>sub-topic]:::process
    PLAN --> S2[Sub-query 2<br/>implicit angle]:::process
    PLAN --> S3[Sub-query 3<br/>reworded]:::process
    PLAN --> S4[Sub-query N<br/>comparative / recent]:::process
    S1 --> R[(Retrieve<br/>concurrently)]:::data
    S2 --> R
    S3 --> R
    S4 --> R
    R --> DEDUP[Deduplicate<br/>near-identical hits]:::process
    DEDUP --> FUSE[Fuse ranked lists<br/>with RRF]:::process
    ONE --> FUSE
    FUSE --> GEN[Generation with<br/>grounded citations]:::output

Every fan-out system has the same four moving parts:

1. A gate. Not every query deserves fan-out. “What is the capital of Spain” needs one lookup. A gate, sometimes a cheap classifier and sometimes just a prompt, decides whether the added latency and spend are justified.
2. A planner. An LLM (or a small purpose-trained model) expands the query into N sub-queries. Good planners produce typed sub-queries, deliberately covering different sub-topics rather than paraphrasing the same one N times.
3. Parallel retrieval. Each sub-query is retrieved independently, ideally concurrently, so wall-clock latency stays close to a single retrieval even as the number of retrieval calls multiplies.
4. Fusion. The N ranked lists are deduplicated and merged into one. Reciprocal Rank Fusion (RRF) is the standard choice because it needs no score calibration across lists: a document’s fused score is the sum over lists of 1 / (k + rank), with k around 60.

# Scenario: the gate. A one-line complexity check keeps fan-out spend off
# the 60% of queries that are simple lookups.
def should_fan_out(query: str, llm) -> bool:
    verdict = llm.complete(
        "Answer YES only if this question has multiple distinct sub-parts or "
        "needs comparison, synthesis, or recent information. Otherwise NO.\n\n"
        f"Q: {query}"
    ).strip().upper()
    return verdict.startswith("YES")

# Scenario: a typed planner. Asking for labelled angles forces coverage
# instead of N paraphrases of the same sub-topic.
def plan_sub_queries(query: str, llm) -> list[str]:
    raw = llm.complete(
        "Generate search queries for the question below, one per line, "
        "covering these angles: [core], [prerequisite], [edge-case], "
        "[comparison], [recent-change]. Prefix each with its tag.\n\n"
        f"{query}"
    )
    return [line.split("]", 1)[-1].strip() for line in raw.splitlines() if "]" in line]

Google AI Mode’s Query Fan-Out

Google’s usage of the term is specific. When a query enters AI Mode or Deep Search, a custom version of Gemini (a custom Gemini 2.0 build when AI Mode launched in March 2025, upgraded to a custom Gemini 2.5 at Google I/O in May 2025 and to Gemini 3 in late 2025) plans a set of related searches, issues them concurrently “across subtopics and multiple data sources,” reads the returned results, and can issue further searches based on what it found. Google has said Deep Search can fire off “hundreds” of queries for a single complex question. The results are synthesized into one response with links, and Google reports its AI search experiences now reach roughly 1.5 billion users a month.

Two consequences follow for anyone doing Generative Engine Optimization:

• You are competing for sub-queries you never see. A page might never rank for the user’s literal question but still get cited because it answered one of the fan-out’s sub-queries well. The unit of optimization shifts from “the keyword” to “the set of latent sub-questions around the topic.”
• Passage-level coverage beats a single hero paragraph. Because each sub-query retrieves independently, a page that addresses several facets of a topic in distinct, self-contained passages has several chances to be pulled in, while a page optimized around one exact phrase has one.

Query Fan-Out in RAG: RAG-Fusion

The retrieval-augmented-generation version predates Google’s terminology. Rackauckas’s RAG-Fusion writeup describes exactly the same loop: prompt an LLM for multiple variations of the user query, retrieve for each, and combine with RRF before generation. It builds directly on the trained-rewriter idea from Ma et al.’s “rewrite-retrieve-read” work, and on the fusion primitive that also underlies hybrid search. Frameworks now ship it as a default: LangChain’s MultiQueryRetriever and LlamaIndex’s query-transform modules both implement multi-query fan-out with a few lines of configuration.

# Scenario: near-duplicate hits waste the generator's context window.
# Collapse passages that are almost the same before fusing.
def dedupe(passages, embed, threshold: float = 0.97):
    kept = []
    for p in passages:
        v = embed(p.text)
        if all(cosine(v, embed(k.text)) < threshold for k in kept):
            kept.append(p)
    return kept

# Scenario: reciprocal rank fusion, written out. No cross-list score
# calibration needed, only ranks.
def rrf(ranked_lists: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranked in ranked_lists:
        for rank, doc_id in enumerate(ranked):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

The Coverage-Versus-Cost Trade-Off

Fan-out’s central knob is N, the number of sub-queries. More sub-queries cover more of the corpus’s vocabulary and more sub-topics, but each one adds a retrieval call, adds candidates that the reranker and generator must process, and (past a point) mostly re-retrieves documents earlier sub-queries already found. A rough but useful model: if each sub-query independently has probability p of surfacing a given relevant document, then N fused sub-queries surface it with probability 1 - (1 - p)^N, which climbs fast and then flattens. The widget below lets you feel where the flattening starts.

Failure Modes

• Redundant fan-out. A weak planner emits N rewordings of one sub-topic. Coverage barely improves, cost still multiplies. Typed prompts and a diversity check on the generated set help.
• Drift. A sub-query wanders off the original intent (“annual plans” becomes “annual reports”) and injects confidently irrelevant passages into the context. Keeping the original query in the fused set and letting the reranker see it against the drifted results limits the damage.
• Context flooding. Fan-out produces more candidates than the generator can use well. A reranking pass after fusion, not just RRF, is usually needed once N is above 4 or 5.
• Latency from a bad gate. If the gate fans out simple lookups, median latency and cost balloon for no benefit. The gate is worth tuning before N is.

What’s New (2025-2026)

• “Query fan-out” became mainstream vocabulary. Google’s March 2025 AI Mode announcement and the May 2025 Google I/O update put the exact phrase into wide circulation, and the SEO industry rebuilt its playbook around optimizing for unseen sub-queries rather than head keywords.
• Fan-out as an agent action. Agentic RAG systems increasingly treat fan-out as one tool an agent chooses at run time, deciding per query whether to retrieve once, fan out, or decompose into sequential hops, rather than always running a fixed multi-query step.
• Iterative fan-out. Deep-research style systems run fan-out in rounds: fan out, read, then fan out again on the gaps the first round revealed, which is closer to Google’s description of AI Mode adjusting “based on discovered results” than the single-shot RAG-Fusion loop.
• Fan-out-aware analytics. Tools now try to reconstruct the likely sub-query set for a target keyword so publishers can audit whether their pages cover those facets, treating the fan-out itself as the thing to optimize against.

Tuning and Practical Guidance

Query fan-out is not a strict upgrade over single-query retrieval; it is a bet that a given question hides several sub-questions and that covering them in parallel is worth several times the retrieval cost. For head lookups that bet loses. For the messy, multi-part questions people increasingly bring to conversational search, it is now the default.

# Scenario: an internal docs assistant where one question ("how do we
# handle refunds for annual plans?") touches policy, billing code, and
# support macros, each indexed under different vocabulary
import asyncio

async def query_fan_out(question: str, llm, retriever, k: int = 60):
    # 1. Plan: expand the question into a handful of angled sub-queries
    plan = llm.complete(
        "Break this question into 4-6 standalone search queries covering "
        "distinct sub-topics, implicit angles, and likely follow-ups. "
        "One per line.\n\n" + question
    )
    sub_queries = [q.strip() for q in plan.splitlines() if q.strip()]

    # 2. Fan out: retrieve for every sub-query concurrently
    ranked_lists = await asyncio.gather(*[
        retriever.asearch(q, top_k=20) for q in sub_queries
    ])

    # 3. Fuse: reciprocal rank fusion across all lists
    scores: dict[str, float] = {}
    for ranked in ranked_lists:
        for rank, doc in enumerate(ranked):
            scores[doc.id] = scores.get(doc.id, 0.0) + 1.0 / (k + rank + 1)

    fused = sorted(scores, key=scores.get, reverse=True)
    return fused[:10]  # hand the top fused passages to the generator