Small models should decide what big models see

It is becoming common practice to let a powerful AI model orchestrate smaller ones. The large model plans the work, then delegates classification, extraction or predictable edits to faster, cheaper models.

This makes sense. But it only optimises who performs each task.

I think there is an even greater efficiency gain available: use small models not only under the large model, but around it.

They could prepare its context before it reasons, filter the output of its tools, and preserve useful information as the session grows.

The large model would still make the difficult decisions. But because it is smart, slow and expensive, every token it processes should earn its place.

Prepare context instead of accumulating it

Coding agents build a long working history: messages, plans, command outputs, files, errors and superseded versions of the same information. Each item may have been useful once. Few remain useful for every later decision.

Current systems often accumulate this material until they clear tool results, save memories or compact the conversation. Anthropic’s documentation treats irrelevant context as a threat to model focus, not just to context window capacity.

A small model could select context before each important reasoning step. Its output might look like this:

{
  "include_messages": [2, 7, 11],
  "include_files": [
    { "path": "src/auth.ts", "lines": "48-126" },
    { "path": "tests/auth.test.ts", "match": "refresh token" }
  ],
  "include_tool_results": ["call_184"],
  "durable_notes": ["The user rejected cookie-based sessions."],
  "omit_reason": "Unrelated build logs and superseded files"
}

The large model would receive the selected source passages, not a vague summary of the entire history. This is retrieval over the agent’s own evolving workspace.

On the left, a cluttered workspace of files, failed tests, old plans and scratch notes. A context selector picks four of them, shown on the right as a working context handed to the main model. — The workspace keeps everything. What reaches the model is only what the next decision needs.

Filter tool outputs before the main model reads them

An API or MCP server may return dozens of fields when the agent needs three. IDs, timestamps, pagination metadata and repeated descriptions then remain in the transcript, and may be processed again on later turns.

A small model could sit between the tool response and the main model. It would filter the raw output against the current objective, return a typed subset, and preserve a reference to the untouched result. This is most valuable when the agent developer cannot change the external tool.

The same layer could classify files, locate relevant sections and exclude generated content before the large model reads them.

A long raw tool response passes through a payload filter, which hands a short list of useful fields to the main model while the raw response is archived as retrievable evidence. — The filter cuts down what reaches the model, and the raw response stays retrievable.

Compress information while it is fresh

Compaction is usually reactive: once the context reaches a threshold, a large history becomes a small summary. By then, the main model may already have reread verbose content many times. One late summary must also preserve details from several stages whose future relevance is hard to predict.

An agent’s history has two readers with different needs. Humans need the complete messages and natural language so they can understand and audit what happened. AI models need the same facts in the fewest reliable tokens.

A small model could maintain both versions as information arrives: the full human-readable transcript, and a compact machine-readable representation loaded into active context. Each compressed item would link back to its original.

The machine version would not need polished prose. It could borrow from the recent “Caveman” trend in coding agents: telegraphic statements that remove articles, pleasantries, hedging and connective filler while preserving technical detail. The point is not to make humans talk this way. It is to stop paying large models to reread language written for humans.

A 2,000 token test log might become:

14 tests passed. refreshes expired token failed because expiresAt was undefined at auth.test.ts:88. Full output: artifact test-run-184.

The result, the failure, the relevant value, the location and the route back to the evidence all survive. The noise does not.

This is continuous housekeeping rather than emergency compaction.

Two timelines. In the first, raw messages pile up until a warning sign, then collapse into one compact summary. In the second, a memory keeper compresses each message as it arrives, producing a compact record throughout. — With late compaction you pay for the same tokens on every turn until the threshold hits. Compressing as you go means paying once.

The economics are plausible

The extra inference is worthwhile only if it saves more than it costs in money, latency and errors. A simple model shows that the financial threshold can be low.

Consider an illustrative 20 step coding session. The large model receives 1.45 million input tokens in total. A context model reads the same history, emits 10,000 tokens of selection instructions, and reduces the large model’s input to 560,000 tokens. Both versions produce 40,000 large model output tokens.

Two columns. On the left a tall stack of documents goes straight into the main model, which produces a large red cost tag. On the right a small filter thins the stack first for a fraction of the price, and the main model's cost tag is much smaller. — The small model is paid for twice over if what it removes costs more to read than it does to run.

Using standard API prices published on 4 July 2026, pairing Gemini 3.1 Flash-Lite ($0.25 input and $1.50 output per million tokens) with Claude Fable 5 ($10 input and $50 output) gives:

	Without filtering	With filtering
Large model input	1.45M tokens	0.56M tokens
Large model cost	$16.50	$7.60
Context model cost	none	$0.38
Total	$16.50	$7.98

Total cost of one 20-step session

Without filtering$16.50

With filtering$7.98

The last row of the table, drawn to scale. The saving is $8.52, or 51.7%.

Filtering pays for itself after removing 37,750 Fable input tokens, which is 2.6% of the raw input.

The filter can fail

Removing context can remove the clue that solves the task. A small model may discard an odd log line, distort a user constraint, or drop an identifier needed by the next tool call.

The architecture therefore needs a few hard rules:

keep raw inputs retrievable;
make filtering decisions structured and auditable;
never compress user instructions or security constraints;
include uncertain information;
let the main model request the original evidence;
measure task success, not only token savings.

Small models should manage access to evidence, not become an invisible source of truth.

The experiment matters more than the estimate

The idea should be tested against three alternatives: raw accumulated context, threshold-based compaction, and continuous semantic filtering.

Each system should run the same coding tasks. The evaluation should measure total token cost, latency, cache utilisation and task success. It should also record every case where filtering removes evidence later needed by the main model.

1One task setThe same coding work for each
2Three systemsRaw, threshold compaction, continuous filtering
3Four measuresCost, latency, cache hits, success
4One verdictCheaper only counts if it still works

Token savings alone would declare the most aggressive filter the winner, which is why completion quality is the deciding measure and not a footnote.

The architecture wins only if it reduces cost without reducing completion quality. Until such a test exists, the economics are a reason to experiment, not proof that the system works.

A new role for small models

Model routing asks: which model should perform this task?

This architecture asks: what deserves the best model’s attention?

Model routingWho does the work

Reads the task, picks a model
The cheap model does the easy jobs
Context is whatever accumulated

Small models around the big oneWhat the work is done on

The best model keeps the hard decisions
Small models choose what it reads
Context is selected before every step

The two are not alternatives. Routing decides who answers, this decides what the answer is based on.

Small models could become context librarians, payload editors and memory maintainers. They would prepare the workspace before the main model reasons, then preserve the useful results after it acts.

The idea matters most for coding agents because they generate so much intermediate material. It also applies to research, support and other long-running workflows that combine conversation, tools and external data.

The most capable model should not read every line simply because the system can send it.

Every token should be there for a reason.