Open source
agent-surface-tester
Put fresh AI agents in front of your MCP, API, CLI or SDK and watch where they stumble.
A local harness that spawns isolated AI agents with zero repo context and gives each one a single door into your product: your MCP server, your REST API and llms.txt, your CLI, or your SDK. Every call is recorded, guarded in code so test runs cannot touch real resources, and folded into one triage report.
- Role
- Author
- Year
- 2026
- Stack
- Node with no framework, MCP SDK, OpenRouter
- Visit
- Source code
Why your coding agent misses the bugs
If you ask the agent in your editor to test your MCP server, it cheats without meaning to. It already read your source code, so when a parameter name is ambiguous or an example is missing from llms.txt, it remembers the answer from your TypeScript types. A real user's agent only sees what your surface exposes. This harness spawns fresh models that have never seen your repo.
One door per run
Each run connects to one surface at a time. An MCP agent only gets your MCP tools. An API agent only gets a guarded HTTP client and your public docs. A CLI agent only gets your binary. Running the same scenario across all four doors catches drift immediately: a clean 400 validation error on REST that turns into a 500 crash on MCP, or a CLI help output so large it gets truncated mid-turn.
Safety lives in code, not in the prompt
During our first test run, a free model ignored a prompt rule saying only modify resources you created and paused an existing widget. So the harness enforces ownership in code: it tracks the IDs created during the run and blocks updates or deletes on anything pre-existing before the request leaves your machine. Per-credential file locks keep parallel runs from colliding.
Fast regression sweeps and frontier stress-testing
A comprehensive tour costs between a fraction of a cent on fast models and a few pennies on frontier reasoning models. Fast models catch documentation gaps, schema typos and unhandled errors; frontier models probe policy boundaries, logic edge cases and clean-up verification.
4surfaces tested: MCP, API, CLI, SDK
94end-to-end test runs across 7 model families
< $0.05per comprehensive multi-step tour