Embed, retrieve, let the LLM decide works for about 100 tools. At 1,000 it needs a control plane: hybrid retrieval, hard filters, a learned reranker, progressive disclosure and per-stage evals.

This is an adapted edition of the original article. It preserves the section argument and evidence; use the source link for the published version and any interactive material.

Read the original article

01 / Baseline

What the pipeline gets right

The idea is that a reliable classifier is a pipeline, not a prompt. Embeddings find the neighborhood. Retrieval returns definitions. The LLM compares them. A validation step chooses between a category and an abstention. I would keep every stage.

The pipeline has a ceiling, and it deserves one extension. If the correct category is not retrieved, the LLM cannot pick it. True. It is equally true of every stage after retrieval, and that is where scale starts to hurt.

At around 100 items the plain pipeline is enough. A 100-tool catalog at roughly 400 tokens per schema is 40k tokens. That figure is the one I used in my Tool Attention paper, and real servers run higher: the public audits I cited put the full GitHub server at about 55,000 tokens for 93 tools, close to 590 each. Forty thousand tokens is wasteful but it fits, and a single top-10 embedding lookup rarely misses.

At 1,000 tools the same arithmetic gives 400k tokens. Retrieval stops being an optimization and becomes the only way in. That is the point where embed, retrieve, decide and validate is necessary and no longer sufficient. The rest of this post is about the parts I would add.

02 / Framing

Tools are not categories

It is tempting to treat tool selection as classification with a bigger label set. The analogy holds for the retrieval half and breaks for everything around it.

  • Wrong answer: Category classification: A wrong value in a row. Fixable later.; Tool selection: A side effect: a sent email, a deleted record, a charged card.
  • Ground truth: Category classification: A taxonomy that changes slowly.; Tool selection: The catalog, plus per-user permissions, plus live connection state. It changes per session.
  • Decisions: Category classification: One per input.; Tool selection: A sequence. Each observation changes which tools are relevant next.
  • Near-duplicates: Category classification: Adjacent categories.; Tool selection: Overlapping tools from different servers, often three different search tools.
  • Abstention: Category classification: One exit: insufficient evidence.; Tool selection: Three exits: ask the user, widen the search, or report the missing capability.
  • Trust: Category classification: The input text is untrusted.; Tool selection: The definitions are untrusted too. A tool description is text the model obeys.

Row one is the one that changes the architecture. Safety cannot live in the prompt when a wrong choice writes to a production system. It has to live in code that runs regardless of what the model says.

03 / Why retrieval at all

It is a context-quality problem, not only a knowledge problem

The baseline pipeline frames retrieval as supplying information the model lacks. That is half of it. The other half is that a model with all the information in a crowded context still does worse, even when the right answer is technically in there.

  • Position effects. Liu et al. (Lost in the Middle, TACL 2024) found that accuracy is lowest when the relevant item sits in the middle of a long context. A tool list is exactly that shape.
  • Tool overload, measured. RAG-MCP (Gan and Sun, 2025) reports tool selection accuracy of 43.13% with retrieval against 13.62% for the baseline that shows the model everything, and prompt tokens cut by more than half.
  • Vendor numbers agree in direction. Anthropic's tool search write-up reports Opus 4 going from 49% to 74% on their MCP evaluation and Opus 4.5 from 79.5% to 88.1%, with about 85% fewer tokens.
  • My own number is a simulation. On a calibrated 120-tool, six-server catalog, gating cut per-turn tool tokens from 47.3k to 2.4k (95.0%). No live agent ran. Treat it as an arithmetic result, not an accuracy result.

These are four different benchmarks, so I am not comparing them to each other. They agree on the direction, and that is enough for the design rule: retrieval is a context-control mechanism. Expose the smallest decision space that still contains the answer.

04 / Measurement

The recall chain

Once you add a reranker, the ceiling is no longer one number. Every stage can drop the right tool, and the drops compound. The precise form is a product of conditional recalls:

End-to-end recall

P(right tool used) =   R_retrieve                    # gold is in the candidate set\
                     * R_filter  | retrieved         # gold survives policy filters\
                     * R_menu    | filtered          # gold survives the reranker cut\
                     * R_select  | menu              # the LLM picks it from the menu

The word "conditional" matters. Multiplying raw retrieval recall by raw reranker recall double counts, because the reranker only ever sees what retrieval returned. Here is an illustration with numbers I made up to show the shape, not measurements:

  • Retrieve 40: Conditional recall: 0.95; Cumulative: 0.950
  • Hard filters: Conditional recall: 0.98; Cumulative: 0.931
  • Rerank to 6: Conditional recall: 0.92; Cumulative: 0.857
  • LLM selects: Conditional recall: 0.90; Cumulative: 0.771

Raising retrieval from 0.95 to 0.99 lifts the end result by about three points. Raising selection from 0.90 to 0.95 lifts it by about four. The weak stage is rarely the one people tune first, because retrieval is the stage everyone knows how to tune. The fix is to log four sets per request and compute the numbers per stage:

stage_recall.py

STAGES = ["retrieved", "filtered", "menu", "chosen"]   # each is a set of tool_ids on the trace\
\
def stage_recall(traces):\
    alive, out = traces, {}\
    for stage in STAGES:\
        kept = [t for t in alive if t.gold in getattr(t, stage)]\
        out[stage] = len(kept) / max(len(alive), 1)   # conditional on the previous stage\
        alive = kept\
    return out                                      # multiply the values for end-to-end

Even in the baseline pipeline, evaluating only the LLM is misleading, so you track retrieval and selection separately. At scale two numbers are not enough. You need four, and you need a labeled set where the gold tool is one the user could actually call.

One caveat: the chain above starts at retrieval, but in the per-subgoal design of section 06 there is a stage before it, splitting the request into subgoals. A bad split caps everything downstream, so give it its own term, R_decompose: the share of requests where the gold tool is covered by some extracted subgoal.

05 / Control part 1

The catalog is a product

The first thing that breaks at scale is not the model or the index. It is the catalog. In my simulation 29 of 500 tasks failed with tool gating. Fourteen (48%) came from a query matching several semantically similar tools. Seven (24%) came from cryptic legacy descriptions. That is 72% of failures where what the catalog says was the proximate cause. Five more (17%) were multi-hop cases, covered below, and the last three fit none of these buckets. It is a simulated workload, so read the proportions as a hypothesis worth testing on your own traces.

So keep the registry outside the prompt and give every tool a card written for retrieval, not for the implementer:

tool card (indexed fields in the top block, the rest loaded on demand)

{\
  "tool_id":      "github.search_code",\
  "domain":       "source_control",\
  "summary":      "Find where something is implemented across connected repositories",\
  "capabilities": ["code_search", "repository_read"],\
  "entities":     ["repository", "file", "commit"],\
  "input_concepts": ["query", "repository", "organization"],\
  "prefer_over":  ["web.search"],\
  "not_for":      "Reading one known file. Use github.read_file.",\
  "examples":     ["Find where login sessions are validated in the checkout-service repository"],\
\
  // not indexed: fetched only after the tool is selected\
  "requires":     { "connection": "github", "scopes": ["repo:read"] },\
  "risk":         "read_only",\
  "cost": "low", "p50_ms": 900,\
  "owner": "platform-team", "schema_version": 7,\
  "input_schema": { /* full JSON Schema */ }\
}

Three fields do more work than they look. summary is written in the voice of the user's intent. In my paper, regenerating summaries that way cut their average length by 63% and raised retrieval F1 by 8 points, on the same synthetic benchmark. not_for and prefer_over are the disambiguation fields. They give the reranker and the model a reason to choose between near-duplicates, and near-duplicates were the largest failure bucket. owner and schema_version exist because at 1,000 tools somebody has to be paged when a description rots.

One structural change belongs here too. A flat top-k over thousands of tools tends to fill the shortlist with ten variants of one family and crowd out the second family you needed. Group tools into domains and route in two hops, domain first and tool second. This is my rule from building these systems, not a measured result. Test it against a flat index on your own catalog before you trust it.

06 / Control part 2

Discovery per subgoal, three retrievers, one pre-filter

The baseline pipeline embeds the input once. A tool request is rarely one intent. "Find where sessions are validated in checkout-service and check whether the fix is deployed" is two subgoals with different tools. Embedding the whole sentence returns a blurred neighborhood that serves neither. I extract structured intent first and run discovery per subgoal:

intent extraction output

{\
  "subgoals": [\
    { "goal": "locate session validation code",\
      "operations": ["repository_search", "code_read"],\
      "entities": ["checkout-service repository"],\
      "required_access": ["GitHub"], "mutation": false },\
    { "goal": "check deployment status",\
      "operations": ["rollout_inspect"],\
      "entities": ["checkout-service"],\
      "required_access": ["Kubernetes"], "mutation": false }\
  ]\
}

Retrieval then runs three ways and fuses the ranks. Vectors catch paraphrase. BM25 catches the exact tokens embeddings blur: tool ids, product names, error codes. Capability matching uses the structured operations field. Reciprocal rank fusion (Cormack et al., SIGIR 2009, constant 60) merges them without needing comparable scores.

Two more signals feed the same stage: historical success on similar tasks, and whether the connector is available in this environment. I use history as a ranker feature and availability as a pre-filter. False positives are acceptable here and a miss is not. A missing tool costs the whole task, while an irrelevant one costs a few tokens that the reranker removes.

The detail people miss is the pre-filter. If you filter for permissions after retrieval, the top 40 can be full of tools this user cannot call, and your recall of usable tools collapses while the recall you measure looks fine. Push the static predicates into the index query.

discover.py (sketch)

def rrf(rankings, k=60):\
    score = defaultdict(float)\
    for ranking in rankings:\
        for rank, tool_id in enumerate(ranking, start=1):\
            score[tool_id] += 1.0 / (k + rank)\
    return sorted(score, key=score.get, reverse=True)\
\
def discover(sub, ctx, index, n=40, menu=6):\
    where = {"connection": ctx.connections, "env": ctx.env,\
             "scopes_within": ctx.granted_scopes}   # static per session, pushed into the index\
    fused = rrf([\
        index.vector(sub.text, n=n, where=where),\
        index.bm25(sub.text, n=n, where=where),\
        index.by_capability(sub.operations, n=n, where=where),\
    ])[:n]\
    allowed = [t for t in fused if request_ok(t, sub, ctx)]   # depends only on this subgoal\
    ranked = reranker.score(sub, allowed)                      # learned, relevance only\
    return prefer_cheap_within(ranked, margin=0.05)[:menu]\
\
def request_ok(tool, sub, ctx):\
    return (tool.risk != "read_only") == sub.mutation   # write tools for write subgoals, read tools for read

07 / Control part 3

Filter, then rank, and where I would change the score

Hard filters are deterministic policy, not LLM judgment. They remove a missing connection or scope, incompatible data types, a write tool for a read-only request (and a read-only tool for a write request), an action that policy disallows or that is riskier than the request needs, and any tool unavailable in this environment. They run before any model sees a candidate, and they are cheap to test.

Each stage optimizes something different, and it helps to say so out loud:

  • Retrieval optimizes recall. Missing the right tool is the expensive error.
  • Filters optimize correctness. They remove what cannot or should not run.
  • The reranker optimizes precision and order. It puts the strongest candidates first.
  • The LLM does a bounded comparison over a small menu.
  • Validation checks that the choice is supported and safe to run.

The reranker then cuts roughly 30 to 50 candidates to a menu of 5 to 8. I keep the cut generous. A reranker error deletes the right tool for good, so this is the stage where I trade a few extra tokens for recall. In my paper I recommended k of 8 to 12 with the threshold doing the precision work. The examples here use 6 because they show one-line menus; widen it if your measured R_menu is low.

A common way to specify the reranker is a weighted sum over the candidate tool t:

Reranking score

S(t) = w1*I + w2*C + w3*E + w4*P + w5*H - w6*R - w7*L - w8*K

  • I intent match, C capability match, E entity or data-source match.
  • P permission and availability, H historical success on similar tasks.
  • R operational risk, L expected latency, K execution cost.

The weights should move with the task. For a sensitive action, permission and risk count for more. For an interactive request, relevance and latency count for more. I like the inputs and I like that instinct. I would change how the terms combine:

  • Permission: Problem inside a weighted sum: A missing scope is not a small penalty. It is a no. It also double counts with the hard filter.; Where I put it instead: Filter only.
  • Risk: Problem inside a weighted sum: Subtracting risk lets a slightly more relevant destructive tool beat a safe adequate one, or the reverse, and nobody audits the weights.; Where I put it instead: A risk policy after selection: confirm, dry-run or deny.
  • Cost, latency: Problem inside a weighted sum: They should never outvote relevance.; Where I put it instead: A tiebreaker among tools within a small relevance margin.
  • History: Problem inside a weighted sum: Popular tools collect more history and crowd out better new ones.; Where I put it instead: A learned feature, with deliberate exploration of new tools.
  • Hand-set weights: Problem inside a weighted sum: They do not survive a catalog change.; Where I put it instead: Train the relevance ranker on (subgoal, tool used, success) traces.

The advice to weight permission and risk more heavily for sensitive actions points the same way. Past some weight they stop being weights and become gates.

The rule I use

Anything that must never happen belongs in a filter or a policy. Anything that is a preference belongs in the ranker. A weighted sum blurs the two, and the blur is where incidents come from.

08 / Control part 4

Progressive disclosure

Retrieval narrows which tools are visible. Progressive disclosure narrows how much the model sees about each one. The token figures below are rough, and they follow from the 400-token schema assumption above.

  • Discovery: The model sees: Name and one-line capability; ~Tokens per tool: 30 to 40; Trigger: Every subgoal
  • Selection: The model sees: Constraints, risk, cost, one example; ~Tokens per tool: 100 to 150; Trigger: Only when two menu items are close
  • Invocation: The model sees: Full input schema; ~Tokens per tool: 400+; Trigger: Only for the chosen tool
  • Recovery: The model sees: The error, alternatives, a wider menu; ~Tokens per tool: varies; Trigger: After a failure or a rejected call

A six-tool menu at discovery level is about 200 tokens. The same six with full schemas is about 2,400. The saving is real, and so is the risk: the model chooses from one-liners, so the one-liners must be good. That takes you straight back to the catalog. This is all the model sees at the discovery level:

discovery-level menu

1. github.search_code   Search code inside connected repositories\
2. github.read_file     Read a specific repository file\
3. jira.search_issues   Search issues and tickets\
4. web.search           Search public internet sources

Only after it picks github.search_code does the system reveal that tool's full input schema. Two operational details. First, keep the summary pool in the stable prefix of the prompt and put per-turn schemas right before the user message. In my simulation this gave an 84% prompt-cache hit rate over a 30-turn session against 22% for naive full-schema injection, because the tool list stopped invalidating the cache. This only works if the schemas travel as message content after the cached prefix. On the Claude API, tool definitions sit at the front of the prefix, so changing the tools parameter between turns invalidates everything behind it. Second, give the model an escape hatch: a search_tools meta-tool it can call when the menu is wrong. Anthropic's tool search is the same idea shipped as a platform feature.

Pair that with a gate: if the model calls a tool that is not in the active set, reject it with a structured error listing what is available. In my simulation the gate fired on 2.3% of turns. The model recovered on the next turn 78% of the time and asked the user a clarifying question the other 22%.

09 / Control parts 5 and 6

Validate, iterate, recover

Validate before executing. The model may pick one tool or a short sequence from the menu, and each call gets deterministic checks on required parameters, types, permissions, read versus write intent, confirmation requirements and target scope. For writes I add a dry-run where the tool supports one. Feed validator errors back to the model verbatim. They are the cheapest correction signal in the system.

Return observations, not payloads. The executor should cap and normalize the result, and keep a handle to the full output outside the context. A 200 KB response dumped into the prompt undoes everything the earlier stages saved.

Rediscover on every subgoal. A tool's output can reveal the next need: a config reference that points to a deployment system you had no tools for. In my simulation 5 of the 29 failures (17%) were multi-hop cases where the right tool only became relevant after an intermediate result. Re-embedding the query after each observation partly fixed them. The agent should not carry every plausibly useful tool for the whole session.

one request, three rounds of discovery (illustrative)

search the repository            # round 1: github.search_code\
  -> found a config reference\
config says it ships as a Kubernetes rollout # new subgoal\
retrieve Kubernetes tools        # round 2: k8s.get_rollout_status\
read the failing pod's logs                  # round 3: k8s.get_pod_logs

Name the failure class. "The agent used the wrong tool" is not a diagnosis. There are at least seven distinct failures, each with a different detector and a different fix:

  • No suitable tool exists: How you see it: Gold label is "none". Abstention rate on those cases.; Where the fix lives: Report the missing capability. Roadmap input.
  • Request mis-split: How you see it: Low R_decompose: the gold tool's need is missing from, or merged into, another subgoal; Where the fix lives: Intent-extraction prompt, examples and output schema.
  • Exists, not retrieved: How you see it: Low R_retrieve; Where the fix lives: Catalog text, hybrid retrieval, intent decomposition.
  • Retrieved, cut by reranker: How you see it: Low R_menu; Where the fix lives: Reranker training data, a larger menu, not_for fields.
  • Shown, not chosen: How you see it: Low R_select; Where the fix lives: Better one-liners, a selection-level disclosure step.
  • Chosen, invoked wrongly: How you see it: Validator rejection rate per tool; Where the fix lives: Examples in the schema, clearer parameter names.
  • Ran, returned too little: How you see it: Observation check fails; the subgoal is still open; Where the fix lives: Next tool in the candidate list, or a new discovery round.

Recovery paths follow from the table. Low discovery confidence: widen retrieval, reformulate the intent, or ask the user. A selected tool fails: return to the candidate list and take the next compatible one. Nothing viable: say which capability or connection is missing. Do not let the model invent an answer to fill the gap.

10 / Worked example

One request, end to end

Everything above is easier to judge on a single request. This is an illustration, not a benchmark: the catalog size, the counts and the tool names are mine, chosen to be realistic. The catalog has 1,200 tools across 30 servers, and the request is:

"The checkout-service deploy failed last night. Find what changed, check the rollout, and tell the payments channel on Slack."

  • Intent extraction: What happens: Three subgoals: find what changed (GitHub, read), check the rollout (Kubernetes, read), post a summary (Slack, write).; The model sees: The request only.
  • Retrieval, per subgoal: What happens: 40 candidates each, out of 1,200, after the connection and scope pre-filter (this user has no Argo CD connection, so those tools never enter the query). For the rollout subgoal, BM25 catches the literal word "rollout" in k8s.get_rollout_status, which the vector search had ranked 63rd, outside its top 40.; The model sees: Nothing yet.
  • Hard filters: What happens: Rollout subgoal: k8s.rollout_restart and nine other write tools removed because the subgoal is read-only. 30 remain.; The model sees: Nothing yet.
  • Rerank: What happens: 30 become a menu of 6. The not_for field on logs.search ("for Kubernetes pod logs use k8s.get_pod_logs") settles a near-duplicate.; The model sees: Nothing yet.
  • Disclosure and selection: What happens: The menu is shown at discovery level. The model picks k8s.get_rollout_status, and only then is its full schema revealed.; The model sees: Six one-liners (about 200 tokens), then one schema (about 400).
  • Validation: What happens: The required parameter namespace is missing. The validator returns an error, the model supplies payments, and the second attempt passes.; The model sees: The error string.
  • Execution: What happens: 200 KB of pod events is capped to a 1.2 KB observation, with a handle to the full output kept outside the context.; The model sees: 1.2 KB.
  • Write subgoal: What happens: Posting to Slack is a write. The risk policy shows the user the draft message and waits for a confirmation before slack.post_message runs.; The model sees: The draft message.

Count the tokens with the 400-token schema assumption from earlier. Showing the whole catalog would cost about 480k tokens on every turn, more than most context windows hold. This run used three menus of about 200 tokens and three schemas of about 400, roughly 1,800 tokens, a cut of more than 99%. Those are assumptions multiplied out, not a measurement.

The more useful observation is where the LLM acted. It made two decisions that choose tools: splitting the request, and choosing from a menu of six. It also supplied one missing parameter and drafted the Slack message, but neither of those picks a tool. Every other step was an index, a rule or a learned ranker. That is the point of the control plane. If this run had gone wrong, it would have gone wrong in one identifiable place. Had k8s.get_rollout_status never been retrieved, that is a retrieval miss. Had it been retrieved and cut, that is the reranker. Had it been chosen and called badly, the validator catches it before anything runs.

11 / Judgment

What to build at which scale

Here is the whole flow in one place, in three stages. It starts where the user does, with the request. The tag on each step says whether it is rules, a learned model, or the LLM. Two of the eight steps are the LLM, and every loop is decided by code.

Fig. 1. Discovery is a stage separate from execution, and the reasoning model never sees the full catalog. Tags: LLM (green), LEARNED (accent), RULES and INDEX (blue). Red arrows are failure paths, the accent arrow is the loop for a new subgoal.

You do not need all of it. The thresholds below are my judgment from building these systems, not measured boundaries. In my simulation the full-schema baseline crossed the 70% context-utilization line at about 50 tools, which is why the first row is short. That figure depends on the context window and on how much the system prompt and conversation already occupy, so rerun it with your own numbers.

  • Up to ~50: What breaks first: Nothing yet.; Add: Nothing. Send them all.
  • 50 to 200: What breaks first: Cost and mid-list misses.; Add: The baseline pipeline: one embedding, top-k, definitions, validation.
  • 200 to 1,000: What breaks first: Near-duplicates, multi-step tasks, unmeasurable failures.; Add: Hybrid retrieval, hard filters, progressive disclosure, per-stage recall.
  • 1,000+: What breaks first: Shortlists crowded by one family, catalog rot, stage latency.; Add: Domain routing, a trained reranker, per-subgoal discovery, catalog ownership and linting.

Where this breaks in production

Each stage adds latency, and I have not measured a full stack end to end, so budget it before you commit. Indexes go stale when a server ships a new schema, which is why schema_version exists. And tool descriptions are attacker-controlled text. In my simulation, gating cut the success rate of 50 poisoned descriptions from 38% to 6%. That was a side effect of never showing irrelevant tools, not a defense, and it needs a real one. The attacks to design for are tool poisoning (instructions hidden in a description), tool shadowing (a third-party tool describing itself to win selection over a trusted one) and a description that changes after it was approved. In the control plane that means hashing each description at approval and re-reviewing on change, treating third-party descriptions as untrusted input in catalog linting, and keeping write tools behind the risk policy whatever any description says. Log per-stage latency (p50 and p95) next to per-stage recall.

12 / Closer

The short version

For classification, the rule is to stop asking the LLM to remember the taxonomy and give it the taxonomy when it needs it. I would keep that and add its counterpart for tools.

Do not ask the LLM to be the only thing standing between a request and a side effect. Give it a small, filtered, well-described menu. Run the rules around it in code. Measure every stage, so that when the agent is wrong you know which part to fix.

My design rule fits in one line. The LLM reasons over the actions that are viable right now and does not carry the platform's whole capability surface in every prompt. Retrieve broadly, filter deterministically, rerank intelligently, and reveal full tool details only when the model needs them.

At 100 tools, the model does the work. At 1,000, the control plane does most of it, and the model makes the last call.

Read the original on Anuj Sadani | Tech