How to ship 10/10 MCPs

Most people ship an MCP server the way they used to ship a REST API. They write the Zod schemas, try two happy-path prompts in Claude Desktop, see a green response, and call it done.

Then real agents show up and quietly suffer through it.

A few weeks ago I wrote about giving agents a tool to send you feedback when they hit something confusing in production. That single tool started catching schema bugs I never would have heard about from a human user. Once I saw the quality of those reports, the next step was obvious: why wait for a customer’s agent to trip over my surface in production when I can send twenty fresh agents through every door before I release?

That’s the whole method I use now. I ran 94 end-to-end agent sessions across my product’s MCP server, REST API, CLI and SDK across multiple model tiers, fixed what they tripped on, and open-sourced the harness as agent-surface-tester.

Why agents give better UX feedback than humans

Getting honest product feedback from human developers is hard. When a developer hits an awkward endpoint, they guess a field name, get a 400, squint at your docs, figure out the quirk, hardcode a workaround, and get back to their own job. Or they close the tab and pick another tool. Either way, you hear nothing.

Even when the rare person does write in, it’s two days later. They paraphrase the error from memory, leave out the payload they sent, and soften the complaint because nobody wants to sound rude in a support ticket.

For the parts of your product that face an agent, agents are much easier to learn from.

An agent doesn’t care about your feelings. It read your tool description, formed a concrete expectation, sent JSON, and got surprised. Asking it where that gap was gives you a direct audit of your naming, your error envelopes, your pagination and your missing examples.

Two halves of the same loop

To get an MCP (and the API, CLI and SDK next to it) close to 10/10, I use two channels that feed each other.

The first half is passive. Every MCP server I build ships with send_feedback (covered in my earlier post). While I was rebuilding Lookio around a wider surface (48 MCP tools, 46 REST endpoints, 70 CLI commands and a TypeScript SDK), a real user’s ChatGPT session called send_feedback twelve times on its own initiative during normal work.

Look at three of the bugs that single live session handed me:

• An impossible JSON Schema. On sync_sitemap, the include_unchanged parameter had type: "boolean" alongside enum: ["true", "false"]. Strict validation rejected both true and "true". A human skimming prose docs never notices a broken schema annotation like that. An MCP client hits a wall on turn one.
• Contradictory pagination. list_resources returned truncated: false right next to a populated next_offset cursor. The agent stopped and asked a fair question: should I stop paging because truncated is false, or keep going because next_offset is there?
• Silent data loss on reindex. Calling update_resource to re-chunk a document quietly wiped its source URL unless the caller knew to send the old URL back in.

No user in that ChatGPT window would have emailed me about any of those. The agent worked around what it could, filed twelve clean reproduction notes in the background, and finished the user’s job.

Why your coding agent in Cursor can’t run the test

Once you see reports like that, you want them before you deploy. The obvious move is to open your repo in Claude Code or Cursor and type: “Test my MCP server and tell me what’s confusing.”

I tried that first. It misses half the bugs because your coding agent already knows too much.

Five minutes earlier, that same session read your route handlers, your Zod validators and your database schema. When llms.txt forgets to show whether the body field is called text or content, your coding agent doesn’t stumble. It remembers text from your source code. You get a clean test run, you feel good about your docs, and the very first external agent hits a 400.

A real user’s agent walks in cold. It has never seen your repository, and it only gets one door into your product:

• An MCP agent only sees your tools/list schemas and tool responses.
• A REST API agent only sees an HTTP client and your public llms.txt or OpenAPI spec.
• A CLI agent only sees your binary and --help.
• An SDK agent only sees your package README and exported methods.

Another technique: your coding agent driving a real browser into ChatGPT

Your coding agent cannot test your MCP directly in its own terminal without bias. But during active feature development, there is another technique I use all the time: letting the coding agent drive a real browser into ChatGPT.

When building an MCP that renders interactive cards or canvas widgets (like the site generator in Atelier201), I open a browser pane on the side logged into ChatGPT. That ChatGPT account has access to the local MCP server, but the ChatGPT conversation itself has zero repository context. It has never seen my codebase.

My coding agent in Claude Code acts as the test conductor:

1. It opens ChatGPT in the browser pane and starts a fresh conversation.
2. It types real user prompts into ChatGPT (such as asking to rewrite a headline or update a layout).
3. It watches ChatGPT pick tools from the MCP, observe the resulting UI card render in the browser, and concurrently reads the local dev server logs to see the incoming payloads, status codes and latencies in real time.
4. If ChatGPT misinterprets a parameter or if the card fails to update, Claude Code reads the failure directly in the logs, patches the server code, and tests again in the browser.

This interactive loop is brilliant while you are actively building a feature. You watch a real consumer model interact with your server and your UI in real time.

For automated testing before a release, though, you do not want to open a browser window and wait for typing animations. You want headless, repeatable sweeps across every door your product exposes. That is why I built agent-surface-tester.

What automated cold runs catch across surfaces

I built agent-surface-tester to run that headless setup locally. It spawns isolated models through OpenRouter with zero workspace context, hands each run a single door (mcp, api, cli or sdk), logs every tool call and latency to JSON, and asks the agent for structured UX friction at the end.

Here’s what a single API run looks like when a fresh model meets an under-documented endpoint:

Over eight test campaigns on Lookio V2, I ran 89 sessions across seven model families (Claude, GPT, Gemini, Grok, Qwen, MiniMax and Kimi). Running the same product tour across all four doors side by side turned up bugs that unit tests never touch:

The plan cap bug is my favourite example of agent honesty. Our test workspace had a three-widget limit on its plan, mentioned nowhere in the workspace metadata. When five different models tried to create a fourth widget during testing, the server replied feature_not_available. Every single model concluded that widgets were disabled on the account, rather than capped at three. A human user hits that, assumes widgets require an enterprise tier, and never upgrades. Five agents in a row pointed straight at the misleading error code.

What testing agent surfaces costs across model tiers

People assume autonomous agent testing requires a enterprise budget. On OpenRouter or compatible gateways, a comprehensive 20 to 50 step product tour typically costs between a fraction of a cent and a few pennies.

Different model tiers serve distinct roles in your testing loop:

• Fast models on flex tiers (like Gemini 3.8 Flash or GPT-6 Luna, at less than a cent a tour) are ideal for continuous regression sweeps. They quickly catch raw interface friction: broken endpoints, schema mismatches, and missing documentation examples. When we put Gemini 3.8 Flash through Lookio, it caught that our assistant search endpoint returned relevant passages without numeric similarity or distance scores, leaving an agent unable to evaluate match confidence before spending credits on generation.
• Frontier reasoning models (like GPT-6.1 Terra on flex, around one to two cents a tour) stress-test policy boundaries and logic. In our tests, Terra automatically used ephemeral store: false flags so test queries wouldn’t pollute logs, tested allergen cross-contact nuances, checked boundary conditions on ordering policies (10 items versus 11 for advance notice), and ran post-deletion verification requests to confirm cleanup.

Comparing the exact same full-surface tour on the same model across three doors also gives a clean measurement of protocol overhead:

Running a four-door sweep across fast and frontier models before a release costs less than a cup of coffee. You get an exhaustive audit of your entire developer surface before a single customer connects.

Two safety rules that have to live in code

If you unleash autonomous test agents against a real backend, two things go wrong fast unless you enforce them in code.

Prompts do not protect pre-existing data. In our very first run against a shared workspace, the system prompt said clearly: “ONLY modify or delete resources you created in this session.” A free model read that rule, listed existing widgets, picked one that belonged to us, renamed it, and paused it. Prompt rules are polite requests. So agent-surface-tester enforces an OwnershipGuard in the harness itself: it watches create responses, records the IDs born during the run, and blocks update or delete calls targeting any other ID before the request leaves your machine.

One credential, one agent at a time. When two agents run concurrently in the same workspace, one deletes a test resource while the other is paginating the list, or one burns the per-key rate limit and makes the second report a phantom API outage. The harness takes an OS file lock per credential hash, prefixes every test artifact (mcp_test_*), and sweeps leftover prefixes before and after each run. Locally, we seed one dedicated sandbox workspace per door so MCP, API, CLI and SDK suites can run in parallel without ever seeing each other’s state.

Listen closely, then filter ruthlessly

Listening to agents doesn’t mean letting them design your product. After reading 89 transcripts, three filters save a lot of wasted work:

1. Throw away the 1-to-10 score. Models are trained to be encouraging. Plenty of runs handed out an 8/10 or 9/10 overall rating right after hitting a 500 crash and three undocumented fields. Ignore the score completely; read the failed tool calls, the retries, and the top_3_fixes list.
2. Check claims against the trace. Models hallucinate in post-mortems too. One run claimed query_assistant didn’t return credit_cost, two minutes after another model logged the exact credit_cost number from the same endpoint. Because the harness saves every raw tool call next to the summary, checking a claim takes five seconds.
3. Keep architecture in human hands. Four separate CLI agents asked us to add a blocking --wait flag to async document ingestion commands. That felt reasonable until we checked it against our own design rule: every Lookio CLI command maps to one HTTP request, with no hidden polling loops that hang a shell for three minutes.

Run it against your own MCP

If you build an MCP server, an API, a CLI or an SDK, put both halves in place:

1. Add a send_feedback tool in production (here's the exact schema and prompt I use) so live agents can tell you what surprised them while they still hold the context.
2. Clone agent-surface-tester, point it at one door at a time with an OpenRouter key, and watch a cold model try to use your product without reading your code.

Your first five-cent sweep will find a broken schema, a misleading 400, or a missing example you swore was obvious. Fix those, run it again, and you’ll feel the difference the next time a real agent connects.