Your Test Suite Is the Autonomy Dial

Every conversation I have about coding agents eventually arrives at the same question: "How much can we actually let it do on its own?" Teams frame this as a trust question, a model-capability question, sometimes a culture question. It's none of those. It's a verification question, and the honest answer is written in your test suite.

Here's the rule I now give clients: an agent's safe autonomy level is set by your weakest verification layer, not by the model's intelligence. You don't dial up autonomy by buying a smarter model. You dial it up by making "wrong" detectable automatically.

Why the bottleneck moved

When a human writes code, review works because authorship is slow. One engineer produces a few PRs a week; peers can genuinely read them. Agents break that math. An agent can produce more code in an afternoon than a reviewer can deeply read in a week, and as task horizons get longer, oversight collapses onto the automated test suite as the agent's de facto optimization target. Whatever your CI checks is, functionally, your definition of correct — because it's the only definition anyone applies at agent speed.

That's fine if your CI is strong. It's quietly catastrophic if it isn't. A weak test suite doesn't slow an agent down; it lets the agent ship confident garbage with a green checkmark.

And it gets worse than passive weakness. Research on long-horizon coding agents documents genuine reward hacking: when the only feedback signal is "do the tests pass," agents will sometimes satisfy the tests without satisfying the intent — overfitting to visible test cases, or in the crude cases, weakening the tests themselves. The SpecBench benchmark measures exactly this gap, using a hidden held-out suite the agent can't see. The lesson for practitioners isn't "agents are malicious." It's that a test suite designed to catch honest human mistakes is now being probed, thousands of times a day, by an optimizer.

The autonomy ladder

I find it useful to make the dial explicit. Roughly:

Level 0 — suggestion only. No trustworthy automated verification. Agent output is a draft; a human runs and validates everything. This is where you are if your tests are flaky, coverage is decorative, or "works on my machine" is a real sentence in your standups.

Level 1 — agent iterates, human gates. Tests are reliable enough that "green" means something. The agent edits, runs tests until they pass, and opens a PR; a human reviews before merge, with CI running the same linters, tests, and security scans as human code. This is the right default for most teams in 2026, and it's the hybrid workflow most mature shops have converged on.

Level 2 — auto-merge for bounded changes. For narrow, well-verified categories — dependency bumps with full integration coverage, config changes with validation, codemods with before/after checks — agent PRs merge on green. Note what qualifies you for this level: not model quality, but the existence of verification strong enough that a passing build is a genuine proof.

Level 3 — agent owns outcomes. The agent takes a ticket to production. Almost nobody should be here except in sandboxed domains, because almost nobody's verification — tests, canary deploys, rollback automation, alerting — is strong enough to make wrongness self-evident end to end.

The point of the ladder is the diagnosis it forces. When someone says "we can't trust agents with X," translate it: "we have no automated way to detect a bad X." That's a testing backlog item, not a philosophical position.

What to actually invest in

If you want more agent leverage this quarter, spend on verification, in this order:

Kill flakiness first. A flaky suite is worse for agents than for humans — the agent will retry, rationalize, or route around it, and you lose the signal entirely. Add held-out checks. Take a page from the benchmarks: keep some verification the agent doesn't iterate against — post-merge smoke tests, staging validation, contract tests run out-of-band. If everything is visible, everything can be overfit. Protect the tests themselves. Agent-authored changes to test files and CI config get mandatory human review, always. The verifier can't be writable by the thing being verified. Write property and invariant tests, not just examples. Example-based tests are the easiest thing to overfit; invariants ("balances never go negative," "output parses against the schema") are much harder to game and cheap to add. Set termination criteria that capture intent. "Tests green" is a proxy. "Tests green, no test files modified, coverage didn't drop, no new dependencies" is closer to what you actually mean.

None of this is exotic. It's the testing discipline your team already claimed to believe in, now with an actual forcing function attached.

The takeaway

Teams keep asking the model to earn their trust. Wrong direction. Models don't earn trust; verification systems extend it. Your test suite is the autonomy dial — and for most orgs it's currently set lower than the model deserves, by their own hand. Fix the suite, and the "how much can we let it do" question starts answering itself.