---
title: "Small models should decide what big models see"
description: "Letting a big model orchestrate smaller ones is now common. I think the bigger win is putting small models around it instead: picking its context, filtering what its tools return, and compressing results while they're still fresh."
date: 2026-07-16
updated: 2026-09-15
language: en
canonical: https://gduv.club/articles/small-models-around-big-ones
source: gduv.club
---
It is becoming common practice to let a powerful AI model orchestrate smaller ones. The large model plans the work, then delegates classification, extraction or predictable edits to faster, cheaper models.

This makes sense. But it only optimises *who performs each task*.

I think there is an even greater efficiency gain available: use small models not only **under** the large model, but **around** it.

They could prepare its context before it reasons, filter the output of its tools, and preserve useful information as the session grows.

The large model would still make the difficult decisions. But because it is smart, slow and expensive, every token it processes should earn its place.

## Prepare context instead of accumulating it

Coding agents build a long working history: messages, plans, command outputs, files, errors and superseded versions of the same information. Each item may have been useful once. Few remain useful for every later decision.

Current systems often accumulate this material until they clear tool results, save memories or compact the conversation. Anthropic's documentation treats irrelevant context as a threat to model focus, not just to context window capacity.

A small model could select context before each important reasoning step. Its output might look like this:

```json
{
  "include_messages": [2, 7, 11],
  "include_files": [
    { "path": "src/auth.ts", "lines": "48-126" },
    { "path": "tests/auth.test.ts", "match": "refresh token" }
  ],
  "include_tool_results": ["call_184"],
  "durable_notes": ["The user rejected cookie-based sessions."],
  "omit_reason": "Unrelated build logs and superseded files"
}
```

The large model would receive the selected source passages, not a vague summary of the entire history. This is retrieval over the agent's own evolving workspace.

_The workspace keeps everything. What reaches the model is only what the next decision needs._

## Filter tool outputs before the main model reads them

An API or MCP server may return dozens of fields when the agent needs three. IDs, timestamps, pagination metadata and repeated descriptions then remain in the transcript, and may be processed again on later turns.

A small model could sit between the tool response and the main model. It would filter the raw output against the current objective, return a typed subset, and preserve a reference to the untouched result. This is most valuable when the agent developer cannot change the external tool.

The same layer could classify files, locate relevant sections and exclude generated content before the large model reads them.

_The filter cuts down what reaches the model, and the raw response stays retrievable._

## Compress information while it is fresh

Compaction is usually reactive: once the context reaches a threshold, a large history becomes a small summary. By then, the main model may already have reread verbose content many times. One late summary must also preserve details from several stages whose future relevance is hard to predict.

An agent's history has two readers with different needs. Humans need the complete messages and natural language so they can understand and audit what happened. AI models need the same facts in the fewest reliable tokens.

A small model could maintain both versions as information arrives: the full human-readable transcript, and a compact machine-readable representation loaded into active context. Each compressed item would link back to its original.

The machine version would not need polished prose. It could borrow from the recent "Caveman" trend in coding agents: telegraphic statements that remove articles, pleasantries, hedging and connective filler while preserving technical detail. The point is not to make humans talk this way. It is to stop paying large models to reread language written for humans.

A 2,000 token test log might become:

> 14 tests passed. `refreshes expired token` failed because `expiresAt` was `undefined` at `auth.test.ts:88`. Full output: artifact `test-run-184`.

The result, the failure, the relevant value, the location and the route back to the evidence all survive. The noise does not.

This is continuous housekeeping rather than emergency compaction.

_With late compaction you pay for the same tokens on every turn until the threshold hits. Compressing as you go means paying once._

## The economics are plausible

The extra inference is worthwhile only if it saves more than it costs in money, latency and errors. A simple model shows that the financial threshold can be low.

Consider an illustrative 20 step coding session. The large model receives 1.45 million input tokens in total. A context model reads the same history, emits 10,000 tokens of selection instructions, and reduces the large model's input to 560,000 tokens. Both versions produce 40,000 large model output tokens.

_The small model is paid for twice over if what it removes costs more to read than it does to run._

Using standard API prices published on 4 July 2026, pairing Gemini 3.1 Flash-Lite ($0.25 input and $1.50 output per million tokens) with Claude Fable 5 ($10 input and $50 output) gives:

| | Without filtering | With filtering |
| :--- | ---: | ---: |
| Large model input | 1.45M tokens | 0.56M tokens |
| Large model cost | $16.50 | $7.60 |
| Context model cost | none | $0.38 |
| **Total** | **$16.50** | **$7.98** |

**Total cost of one 20-step session**

|  | Value |
| :--- | ---: |
| Without filtering | $16.50 |
| With filtering | $7.98 |

_The last row of the table, drawn to scale. The saving is $8.52, or 51.7%._

Filtering pays for itself after removing 37,750 Fable input tokens, which is 2.6% of the raw input.

**These are assumptions, not measurements**

The model excludes orchestration cost and latency, assumes unchanged output and task quality, and compares providers with different tokenizers.

Prompt caching also narrows the opportunity. If every Fable input token received its $1 per million cache-hit price, the saving would fall to about $0.51. Semantic filtering is most valuable when context is large, dynamic and poorly cached. Stable prefixes are better served by caching.

## The filter can fail

Removing context can remove the clue that solves the task. A small model may discard an odd log line, distort a user constraint, or drop an identifier needed by the next tool call.

The architecture therefore needs a few hard rules:

- keep raw inputs retrievable;
- make filtering decisions structured and auditable;
- never compress user instructions or security constraints;
- include uncertain information;
- let the main model request the original evidence;
- measure task success, not only token savings.

Small models should manage access to evidence, not become an invisible source of truth.

## The experiment matters more than the estimate

The idea should be tested against three alternatives: raw accumulated context, threshold-based compaction, and continuous semantic filtering.

Each system should run the same coding tasks. The evaluation should measure total token cost, latency, cache utilisation and task success. It should also record every case where filtering removes evidence later needed by the main model.

1. **One task set**. The same coding work for each
2. **Three systems**. Raw, threshold compaction, continuous filtering
3. **Four measures**. Cost, latency, cache hits, success
4. **One verdict**. Cheaper only counts if it still works

_Token savings alone would declare the most aggressive filter the winner, which is why completion quality is the deciding measure and not a footnote._

The architecture wins only if it reduces cost without reducing completion quality. Until such a test exists, the economics are a reason to experiment, not proof that the system works.

## A new role for small models

Model routing asks: which model should perform this task?

This architecture asks: what deserves the best model's attention?

**Model routing** (Who does the work)

- Reads the task, picks a model
- The cheap model does the easy jobs
- Context is whatever accumulated

**Small models around the big one** (What the work is done on)

- The best model keeps the hard decisions
- Small models choose what it reads
- Context is selected before every step

_The two are not alternatives. Routing decides who answers, this decides what the answer is based on._

Small models could become context librarians, payload editors and memory maintainers. They would prepare the workspace before the main model reasons, then preserve the useful results after it acts.

The idea matters most for coding agents because they generate so much intermediate material. It also applies to research, support and other long-running workflows that combine conversation, tools and external data.

The most capable model should not read every line simply because the system can send it.

Every token should be there for a reason.

## Sources

- [Anthropic: context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing)
- [Anthropic: context engineering with memory, compaction and tool clearing](https://platform.claude.com/cookbook/tool-use-context-engineering-context-engineering-tools)
- [Caveman: token-efficient communication for coding agents](https://caveman.so/)
- [Google: Gemini 3.1 Flash-Lite](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-lite)
- [Anthropic: Claude API pricing](https://platform.claude.com/docs/en/about-claude/pricing)
- [Model Context Protocol: tools specification](https://modelcontextprotocol.io/specification/2025-06-18/server/tools)