Jev cut this AI agent's cost by 61%

In July I wrote that small models should decide what big models see: not routing a task to a cheaper model, but putting small models alongside the big one to choose which tokens it ever reads. I ended that piece by saying the economics were a reason to experiment and not proof, and that someone had to run the test.

Then Jev shipped on 15 September. It is TypeSafe’s first System One model: it writes no text at all, it returns calibrated probabilities over answers your code defines, it costs $0.042 per million input tokens, and it answers in a few hundred milliseconds. That combination makes two very concrete versions of the July idea cheap enough to actually measure.

So I tested those two.

Preloading the skills. An agent given a catalogue of internal documents spends a whole round trip deciding which to read and calling a tool to fetch them. Jev can make that decision before the first turn, for a fraction of a cent, and the documents arrive already in the prompt.

Filtering the tool payload. A real API response is mostly fields no agent will act on, and in an MCP setup all of it lands in the context window and gets re-sent on every following turn. Jev can score every field of a response before the agent sees it.

Both are classification problems, not writing problems. Here is what happened across 192 runs on 8 tasks, priced on Claude Opus 5:

Another possible decision for Jev is whether a chat title still fits the conversation. I describe that use case in my article on generating conversation titles. I haven’t measured it with Jev.

The best task came out 61% cheaper. The mean across all eight is 29%, and the worst is 13%. 99% of quality checks passed in every arm, including the plain one, so nothing was traded away to get any of it.

The whole experiment cost $1.17 to run.

What Jev is

It generates no text. You send it a state and a set of typed questions, and it returns a calibrated probability distribution over the answers your code defined. TypeSafe call it a System One model, trained with what they describe as reinforcement learning for calibrated decisions.

The property that makes this experiment work at all is batching. Every question goes in one HTTP call with the state sent once. Scoring 264 payload fields is one request, not 264.

Mechanism one: preloading the skills

Before the agent’s first turn, Jev reads the request and the one-line description of every skill. It never sees a skill’s body. Anything scoring at or above 0.55 is injected into the system prompt in full.

The read_skill tool stays available, so the agent can still fetch anything the router missed. That matters, because the router does miss things.

Mechanism two: filtering the tool payload

After a tool returns and before its output reaches the agent, every leaf field of the response is scored for relevance to the current request. Anything below 0.35 is stripped.

Worth noticing in those scores: the fields that get cut on this call sit at 0.31 to 0.34 and the ones that stay sit at 0.35 to 0.37. The boundary is genuinely a boundary, and a few fields cross it between runs. trial_end scored 0.34 in one run and 0.37 in another. Nothing the task needed was ever near it.

Where the safety actually comes from

That third step, the state, is where the whole thing is decided. The same threshold with a poorer state is dangerous. With a richer one it is both safer and more aggressive at the same time.

Measured in isolation on the field that decides the dunning task:

The two middle columns are the scores given to the field the task actually depends on. The last is how many of the 124 fields survive the 0.35 cut.

Adding the skills changes nothing. History helps a little. The trigger is the list of remaining actions. Until the scorer knows that a billing_set_account_state(account_id, …) call is waiting, an account id is just another string.

The general form of it, which is the part that carries to another setup:

A field is not useful in itself. It is useful for what the agent is about to do with it, and that information is not in the payload. It is in the schema of the tools still available.

Where the gains are, and where they are not

The two mechanisms track completely different things, and knowing which one applies to your workload matters more than the average.

The spread is large. On the task where six help-centre articles arrive and only updated_at matters, filtering removes 84% of the payload and 63% of the input tokens. On the incident task, where nearly every field is used, it removes 5%. Same agent, same threshold, a factor of 17 between them.

Cost on Opus 5, per task, plain agent versus both mechanisms:

An Opus 5 agent handling 10,000 tasks a month goes from $419 to $299, of which $9 goes to Jev.

The price of your main model decides whether filtering is worth it

This is the result that should change what you do.

Filtering costs Jev tokens on every tool call. Those tokens are a fixed price. The tokens it saves are priced at whatever your agent model costs. So the whole thing is a ratio, and it flips.

Same token counts throughout. Only the agent’s rate changes. On the cheap model Jev is 42% of the bill and filtering costs more than it saves. On Opus 5 it is 3% of the bill.

Solving for the crossover, holding the measured token counts and the usual rate structure, puts it at $0.55 per million input tokens with prompt caching on, and $0.37 without. Below that, do not filter. Above it, filter.

Preloading skills is profitable everywhere, including on the cheapest model, because it costs almost nothing: about 1,500 Jev tokens per run, $0.00006, against a saved turn.

There is a second-order effect worth knowing. Caching roughly halves the bill and reduces the relative value of filtering, because most of what filtering removes would have been cheap cache reads. Filtering is worth more to a deployment without caching than to one with it.

The failure mode

The 0.35 threshold is only safe because of the rich state. An earlier version without it produced the worst kind of failure.

The filter dropped account.id from a Stripe subscription, scoring it 0.18, and also dropped metadata.account_id, the second copy. The agent took the only identifier left and did this:

call:   billing_set_account_state(account_id="cus_PmT4k9WqLr2XbN", state="read_only")
                                              ↑ the Stripe customer id, not the account

report: "I set the associated account to read-only, in accordance with the day-10 policy."

The correct identifier was acct_borealis_4c77. The agent acted on the wrong object and filed a confident report about it.

Anything you deploy needs declared critical paths and a threshold calibrated against them. Two hundredths of a threshold separated “saves a turn” from “saves nothing” on the routing side, and a much bigger gap than that on the filtering side.

What I take from this

Working on AI efficiency is worth the time, and combining models of different sizes and different kinds is where a lot of it lives. What makes this interesting is that a model which cannot write text at all turns out to be a good judge of what a model that writes text should be allowed to see.

Two starting points were tested here. There are many more places the same pattern could apply: choosing which files an agent opens, deciding when a conversation should be compacted, scoring whether a retrieved document is worth its tokens, gating a write before it happens. None of that was measured, and I would not assume any of it works until it is.

The full record is below: the harness, every task, every threshold, all the tables, and the things this does not show.

The full details

Everything from here down is the complete experiment. It is long on purpose. If you only wanted the result, you already have it.

The setup

The agent

About 80 lines. No framework, no agent SDK, no orchestration library. A while loop around fetch:

1. POST to OpenRouter’s chat completions endpoint with the message list and the tool schemas in OpenAI function-calling format.
2. If the reply has tool_calls, execute each one locally, push the result as a role: "tool" message, and loop.
3. If the reply has no tool_calls, that text is the final answer and the loop ends.

On reasoning specifically: no reasoning parameter was ever passed, so the model ran at the provider default. A control call made afterwards returns reasoning_tokens: 0, so no separate reasoning budget was being spent and none of the token counts include hidden reasoning tokens. Re-running this at a higher reasoning effort would produce different absolute numbers.

Two safety rails, because a runaway agent loop burns money: 8 turns maximum per run, and 6 tool calls maximum per turn. Neither was hit in the 192 runs. A third rail sums every call and throws once the total crosses a cap, which was set at $2.60 for the campaign.

The world

A fictional B2B SaaS called Northwind Analytics, with the agent as its operations assistant.

17 tools whose payloads imitate the real shape of Stripe, Zendesk, HubSpot, Statuspage, PagerDuty, Slack, Linear and a help-centre CMS: same key names, same nesting, same noise. They are local functions returning fixtures, so nothing leaves the machine. Writes are recorded for grading and return a plausible acknowledgement.

11 skills, Markdown files with a one-line description in front matter and a body carrying the actual thresholds, numbers and formats: churn-save-offers, docs-review-cadence, dunning-playbook, enterprise-contract-terms, escalation-matrix, gdpr-data-requests, incident-comms, oncall-handover, refund-policy, release-notes-format, slack-style-guide. Several bear on no task at all. They are distractors for the router.

8 tasks, chosen to span two independent axes: length, from a single tool call to seven turns, and payload density, from “every field matters” to “76% of the bytes are text to throw away”.

The tasks, verbatim

Grading

Deterministic. 59 checks across the eight tasks, each one either a tool call with exact arguments, such as stripe_create_refund(charge="ch_3QRk9X…", amount=14900, reason="duplicate"), or a fact that appears only inside a skill document: the 25% SLA tier, the day-10 rule, the PRIV team, the 30-day deadline.

No LLM judge is involved anywhere. A run that never opens the skills fails the second kind of check mechanically, which is the point: it makes “did the agent actually have the policy” measurable rather than a matter of opinion.

Each task also declares its critical payload paths, the fields its answer depends on. These are never used to steer the filter. They exist so an over-eager filter is visible in the record even on runs that pass anyway.

The arms

8 tasks × 4 arms × 6 repetitions = 192 runs, repetitions differing only by seed.

Calibration

Both thresholds were swept offline before any agent tokens were spent.

0.55 for routing is the highest value that keeps full recall on the tasks it was calibrated against. At 0.60, slack-style-guide scores 0.58 on the refund task and the saved turn evaporates. Two hundredths separate a 15% turn reduction from nothing.

0.35 for filtering is only tenable because of the rich state described above. The calibration pass costs about $0.004 because it runs the router and scorer without running the agent at all.

One lesson that generalises: every framing of a question has its own calibration. A threshold does not transfer between two wordings of the same question. Distributions on completely different supports were measured for a task-specific framing and a task-independent one. Recalibrate whenever you reword.

Campaign totals

Overall, by arm

No technique degraded quality. 99% of checks pass in all four arms. Across 96 filtered calls in arm D, exactly one declared-critical path was dropped and that run still passed. The per-run “perfect” differences between 92% and 96% are within noise at n = 48. The robust effects are turns and tokens.

Mean tokens per run, split the way they are actually billed:

Arm D cuts fresh input tokens, the expensive kind, by 49%.

Input-token saving per technique, worst to best

The one negative cell in the whole matrix is A on the SLA task, where the router loads one skill too many without removing a turn. The extra skill goes into the system prompt and is then re-sent on every turn. The two techniques also do not always add up: on the SLA and postmortem tasks, D saves fewer tokens than B alone, for exactly that reason.

How the router actually scored

Mean probability per skill, per task. Loaded means at or above 0.55.

Recall: 132 of 144 needed skills loaded, 92%. One systematic miss and one systematic false positive, both worth stating plainly.

The miss is slack-style-guide on the postmortem task, scoring 0.47, below threshold, in 12 runs out of 12. The task says “tell the company where it stands” and the router does not read that as a formatting question. The agent fetched it itself when it needed it.

The false positive is refund-policy on the SLA task at 0.84 and the postmortem at 0.89. An SLA credit looks like a refund. It is exactly the confusion refund-policy exists to forbid, since the document says service unavailability is not a refund. The router is wrong about form and right about substance, and the agent was not harmed.

The lookup task loads nothing, correctly. No skill bears on it and the highest score is 0.07.

How the filter actually behaved

Arm D, by tool, across 96 filtered calls:

The read tools are where the money is, and the spread among them is enormous: from −89% on a document list to −25% on an incident where nearly everything is load-bearing.

The write acknowledgements go the wrong way. billing_apply_credit returns 147 characters, and the _trimmed marker appended to it costs more than the fields removed. Filtering a small payload is a net loss, and any production implementation should skip anything under a few hundred characters outright. It was left in so the effect is visible in the record.

Every task, every arm

Prices are the complete cost of one run, agent plus Jev, using the measured fresh and cached split. The Sonnet 5 and Opus 5 columns replay the measured traffic at those models’ prices. Jev’s cost is identical in all three columns.

T1-refund, medium · mixed payload

T2-sla, long · mixed payload

T3-dunning, short · mixed payload

T4-gdpr, medium · mixed payload

T5-incident, short · dense payload

T6-stale-docs, short · very sparse payload

T7-postmortem, very long · mixed payload

T8-lookup, one call · sparse payload

How the prices were computed

Measure once, then reprice. The token counts are the baseline, and every model’s bill is those same counts at that model’s rates.

The billing formula was reverse-engineered from the data rather than assumed. Across all 192 runs, the observed agent cost matches

fresh_input × $0.25/M  +  cached_input × $0.02/M  +  output × $1.20/M

to a median relative error of 0.04%. Two consequences. Prompt caching was on the whole time, automatically, with no cache_control markers, and 80% of input tokens were cache reads. And every fresh input token is billed at the cache write rate, which is 1.25× the nominal input price, so the nominal $0.20/M rate is essentially never what you pay in a multi-turn loop.

An earlier theoretical model of caching, where reads at turn t equal the prefix at turn t−1, under-estimated real cache reads by 19%. It was discarded in favour of the measured per-turn split.

The same runs with no caching at all, for reference:

One assumption is conservative rather than neutral. OpenAI caches automatically and bills every fresh input token at the write rate, which is what was measured. Anthropic caching is opt-in: only content inside a cache_control breakpoint is billed at the write rate and the rest at the plain input rate. Charging all fresh tokens at the write rate therefore over-states the Anthropic columns slightly. Recomputing with fresh tokens at the plain input rate moves the Opus 5 mean from $0.04191 → $0.02995 (−29%) to $0.03823 → $0.02808 (−27%), and no per-task figure moves by more than 3 points.

What this does not show

• One agent model. Everything is measured on openai/gpt-5.6-luna at default reasoning. The token counts are the deliverable and the price columns are arithmetic on top of them. A different model would produce different token counts, not just different prices.
• n = 6 per cell. Enough for turn counts and token counts, which are near-deterministic. Not enough to separate 92% from 96% quality.
• Fixtures, not live APIs. The payload shapes are faithful. Real endpoints have pagination, partial failures and rate limits that this harness does not exercise.
• The tasks were written by the same person who wrote the skills. The grading is deterministic, but the world is not adversarial.
• Jev’s own calibration was not audited. What was measured is what its scores do to an agent, not whether they are well calibrated in the statistical sense.
• One agent shape. A while loop with function calling. Sub-agents, parallel tool calls and long-lived sessions all change the arithmetic.

Sources

• TypeSafe: introducing System One models and Jev
• TypeSafe: System One concepts
• Jev 1.13 on OpenRouter
• My earlier article: small models should decide what big models see
• Model Context Protocol: tools specification