Tool definitions get resent on every turn because the model can't remember them. That makes bloat a recurring tax — and it's charged against both your bill and your accuracy.
If you've built an agent that connects to more than a handful of tools, you've paid for this whether you noticed it or not. Every tool you register — every MCP server you bolt on, every function you expose — has its full JSON schema shipped into the model's context on every single turn of the loop. Not once at setup. Every turn. The model has no memory of what tools exist between calls, so the schemas have to be there, in-context, each time it decides whether to reach for one.
That single mechanical fact is the whole story of tool-list bloat. It's why an unused tool is not free: it's a line item you re-pay on every turn, in tokens you're billed for and in attention the model spends reading past it to find the one tool that matters.
Why the model re-reads the toolbox every turn
The mental model that trips teams up is imagining the API as a session that "knows" its tools after you declare them once. It doesn't. LLM inference is stateless across requests. Each call to the messages endpoint is an independent forward pass over a freshly assembled input: system prompt, tool definitions, the entire conversation history, and the new user turn. Nothing persists on the server between calls. What looks like memory in a chat agent is just the client re-sending the full transcript every time.
There is a cache — but it's the wrong one for this intuition. Inside a single forward pass, the KV cache saves the model from recomputing attention over tokens it has already processed. That's an intra-request optimization. It does nothing to carry tool knowledge from one API call to the next. Across calls, the toolbox is reconstructed from scratch.
So the tool block is input, and it's input you send again and again. The tokens the model actually processes are the whole request — system prompt, tool schemas, retrieved context, and history combined — which is why usage should always be measured off the full assembled payload, not the visible user message. For an agent, tool definitions are a fixed cost you multiply by turn count.
The cost line, multiplied by turns
The per-request numbers are already larger than most people guess. A fairly ordinary five-server setup — GitHub, Slack, Sentry, Grafana, and Splunk — can expose around 58 tools that consume roughly 55,000 tokens before the conversation even starts. Add Jira and you're spending another ~17K on that server alone. Large internal systems routinely cross 100,000 tokens of tool definitions before anyone optimizes.
Now apply the multiplier the stateless model forces on you. A 55K-token tool block isn't a 55K cost — it's 55K per turn. An eight-turn task carries roughly 440K tokens of pure tool-schema overhead before a single line of real work is counted. Prompt caching claws some of that back (more on that below), but the arithmetic is the point: the thing you provisioned once is billed on repeat, and it scales with how chatty your agent is, not with how many tools it actually uses.
This is the part that's easy to model and easy to fix, which is exactly why teams over-index on it. The harder cost is the one that doesn't show up on the invoice.
The quality line, which people underweight
Bigger toolboxes don't just cost more — they make the model worse at picking the right tool. This is measurable, not vibes.
One instructive stress test, borrowed from the needle-in-a-haystack setup, presents the model with N tool schemas, exactly one of which is correct, and scales N up. Selection accuracy degrades as the pool grows. In that benchmark, naively presenting the full tool set landed at roughly 14% selection accuracy; retrieving only the relevant subset before invoking the model lifted it past 43% — more than triple — while cutting prompt tokens by over half. The failure isn't exotic. It's the model burning its decision budget disambiguating between schemas that overlap, and picking wrong more often as the distractors pile up.
Practitioner experience matches: token cost isn't the only issue — the most common failures are wrong tool selection and incorrect parameters, and they get worse when tools have near-identical names like notification-send-user versus notification-send-channel. Every extra schema in context is another near-neighbor the model can trip over, and another few hundred tokens diluting the signal from the tools that matter.
So bloat is a two-sided tax. The cost side is linear and visible. The quality side is the one that silently raises your error rate as your integration surface grows — the opposite of what "we added more capabilities" is supposed to buy you.
Prompt caching pays down the invoice, not the tax
The first objection is always: doesn't prompt caching solve this? Partly. If your tool block sits at a stable position at the front of the request, providers can cache that prefix and bill re-reads at a fraction of standard input pricing — on the order of a tenth of the base rate. That's real money saved on the cost line.
But caching pays down the invoice; it doesn't remove the underlying tax, for two reasons.
First, a cache read is cheaper, not weightless — and it does nothing for quality. The 55K tokens still occupy the context window and the model still attends over all of them when choosing a tool. Caching makes bloat cheaper to be wrong about. It does not make the model more accurate.
Second, caching and dynamism are in direct tension. A cache hit requires the cached prefix to be byte-stable. The moment you add, remove, reorder, or reword a tool, you invalidate the cache from that point forward and re-pay full price to warm it again. That puts you on the horns of a real trade-off: a static mega-list of every tool is cacheable but bloated and dilutive; a tool list tailored per request is lean but cache-hostile. Picking a lane here is an architecture decision, not a config flag — and it's the decision most teams make by accident.
What actually works
The durable fix isn't "prune your tools by hand" — that just moves the bloat problem into a spreadsheet someone has to maintain, and it caps the capability surface you can offer. The pattern that holds up is don't put tools in context until the model needs them.
On-demand tool discovery. Instead of loading every definition upfront, register the full library but defer most of it, exposing only a search tool plus your three-to-five highest-use tools. The model searches for what it needs, and only those definitions expand into context. The results are strong: upfront load can drop to a few hundred tokens for the search tool, total context consumption falls by around 85%, and — the part that matters — tool-selection accuracy goes up, because the model stops drowning in irrelevant schemas. Retrieval-augmented approaches implement the same idea: embed the tool catalog, pull the top-k relevant tools per query, and hand the model only those.
There's a design subtlety worth calling out, because it interacts with the caching trade-off above: if deferred tools are excluded from the initial prompt entirely, your stable system prompt and core tools stay cacheable while the on-demand tools load after the fact. Done right, you get the lean-context win without paying the cache-invalidation penalty on your hot path.
Keep intermediate results out of context, too. Tool definitions are one half of the bloat; tool results are the other. Chaining tools by passing every raw response back through the model inflates context and multiplies the chances to make a copying or reasoning error. Orchestrating those calls in code — where the model writes a script that fetches, filters, and aggregates, and only the final result returns to context — has been shown to cut token usage by roughly a third on complex research tasks, with accuracy improving alongside. The seed problem and this one share a root cause: the context window is a scarce, attention-bearing resource, and the discipline is to keep out of it anything the model doesn't need to reason over directly.
Know the failure modes before you adopt them. None of this is free. A search-before-invoke step adds latency, so it earns its keep when your tool block is large (a rough threshold: more than ~10K tokens of definitions, or 10+ tools, or when you're already seeing selection errors) and hurts on a small library where every tool is used every turn. Retrieval has its own ceiling: precision degrades once the registry scales into the thousands, which pushes you toward hierarchical or namespaced retrieval rather than one flat index. There is no configuration that makes the trade-offs disappear; there's only choosing which one fits your traffic.
Making it operational
If you run agents in production, treat tool-definition overhead as a first-class metric, not an afterthought:
- Measure the fixed cost. Log tool-schema tokens per request and multiply by your p50 and p95 turn counts. That product — not the per-call number — is your real tool-bloat bill, and it's the number that justifies the migration.
- Measure selection quality, separately. Build a small eval of representative tasks and track how often the agent picks the right tool and populates parameters correctly. Watch it as your tool count grows; a slow decline here is bloat showing up as accuracy loss long before anyone complains about cost.
- Watch your cache hit rate. If it's low, something is mutating your tool prefix per request — a reordered list, a templated description — and you're re-warming the cache constantly. That's a silent regression the token bill won't obviously explain.
- Migrate behind a flag, and keep the rollback. Moving from all-tools-loaded to deferred discovery changes which tools the model can see at any instant, which changes its behavior. Roll it out on a slice of traffic, compare selection accuracy and latency against the static baseline, and keep the old path one switch away until the eval confirms parity or better.
At Tilicho we've come to treat this as a middleware concern rather than something each agent solves on its own — a thin layer on the model wire whose job includes compressing what enters context, routing to the right tools and models, and measuring both cost and selection quality per turn. The specifics vary by workload, but the principle is constant: the context window is the budget, tool schemas spend it on every turn, and anything the model doesn't need to reason over right now should live outside it.
The one-line version to take away: the model re-reads its entire toolbox on every turn, so every tool you register is a recurring charge — in tokens and in accuracy — and the fix is to stop paying for tools the current task will never call.