Catch the wrong recipient. Block the forbidden action. Turn tool-call regressions into tests you can ship.
✦ LOCAL-FIRST. OPEN SOURCE.
No API keys. No live tools. Just evidence.
LOCAL CHECKS / NO FILE LOADED
Bring your own evidence.
Open a contract or a PlumbCheck report. Files stay in this tab. Reports are read-only and are not rerun.
FIXTURE REPLAY
Checks passed
Regressions caught
Tool calls blocked
0Live calls via replay
Execution evidenceEVERY CALL. EVERY CONTRACT.
Replay covers intercepted tool calls. Model reasoning and other I/O are outside this boundary.
01 / DEFINE THE BEHAVIOR
Your automation. Your contract.
Fixtures define the exact calls and responses you expect. Policies decide which tools are allowed. Change an argument below and run the checks to see a regression.
Approval context is a trusted test fixture. It models your app’s approval decision; it does not authenticate a human. Never accept it from agent-generated arguments.
02 / TEST YOUR MCP AGENT
Your agent chooses. PlumbCheck checks.
Save this contract and create a configuration for one test case. Add it to your MCP client in place of the live tool servers, then give your agent a task that matches the case. The fixture server checks the calls your agent actually makes.
Contract files contain raw fixture data. Review them before connecting an agent that sends tool responses to a model provider. This setup uses planned-call contracts as fixtures; agent-code contracts use the CLI runner.
Close the agent's MCP connection cleanly to finish the test. Then use Open JSON to inspect the generated report. The configuration uses the saved contract file; save again after editing.
02 / TAKE IT TO YOUR REPO
From one bug to a regression test.
How does it test different agents?
MCP agents: connect your client to PlumbCheck's fixture server. Your agent chooses the calls; the server checks their arguments, order, policies, and results. In the Windows app, open Contract to save a test case and generate the MCP configuration.
JavaScript agents: pass the replay client into your function, or use the CLI's --agent runner to check its final output too.
Other frameworks, Python, and workflow builders: use their MCP support when available, or write an adapter for their tool interface. Framework-specific adapters and automatic workflow imports are not included yet.
PlumbCheck checks intercepted calls. Disconnect your client's live tool servers for a fixture test; unrelated tools, model reasoning, and other agent I/O are outside this boundary.
01. Run it locally
The Windows app bundles its runtime. For the source-code version, use Node.js 22 or later:
npm test
npm start
# Intentional failure, exit code 1
npm run demo
# Corrected contract, exit code 0
npm run check
02. Gate a pull request
Commit a synthetic or reviewed fixture. Run a contract in CI and keep the evidence as a build artifact.
node bin/plumbcheck.js run \
examples/customer-support/safe.json \
--out reports/ci
See the runnable examples/customer-support/agent.js.
04. Know the boundary
Replay makes zero live tool calls. Recording through a real client executes real tools. Tool names and arguments match exactly; unknown tools and missing fixtures fail.
This alpha is a testing layer. It is not a sandbox, an approval service, or a guarantee of deterministic LLM behavior.
Save a scenario. Connect an agent. Keep the evidence.
Opening your workspace…
Start here / Build your first test
01 / Your projects
Check the local server, select an installed model, then save. This does not install a model or run a test.
02 / Add a model connection
Use a provider API or a local tool-capable model. A chat subscription is a separate connection route; pasting a chat login token is unsupported.
Ask your assistant.
Connect the running desktop app as an MCP testing service. The assistant can list saved projects, validate contracts, run offline tests, inspect evidence, compare runs, and cancel. It cannot execute arbitrary commands or spend model credits.
Local stdio configuration is for compatible desktop/CLI hosts. ChatGPT web/mobile needs a separately hosted authenticated HTTPS connection; this local setup does not claim that connection.Optional / Sync with your phone
Use your own HTTPS snapshot relay. Each upload creates an immutable version. Importing a version creates a local copy, preserving concurrent edits. Credentials and model settings stay on this device.
03 / Run a saved test
Choose or create a project
Tool responses are simulated. Coverage applies to the connected tools. The run-wide output budget caps subsequent generations. Usage is shown when the provider reports it. Input tokens and currency costs are separate.