We asked an AI agent to plant bugs in real codebases. Our gate caught both in under a minute.

There is a number from recent research that should worry anyone merging AI-written code on the strength of green tests. Independent audits of SWE-bench found that a large share of “plausible” agent patches, patches that pass the available tests, are wrong anyway. One empirical study found roughly 30 percent of plausible fixes introduced regressions or were incorrect on rigorous retesting, and a detailed manual audit dropped one model's true solve rate from 12.47 percent to 3.97 percent.

Everyone in software has always known this. It just never mattered this much before, because code review by a human who wrote similar code was the backstop. Now the volume of machine-written diffs is outrunning the reviewers.

This is the exact problem Refactron exists for, so we ran the most honest experiment we could think of.

The setup

We took two mature, well-tested open-source codebases: Click, the CLI framework (1,917 tests), and attrs, the class-machinery library (1,382 tests). We then had a strong coding agent write five genuine, multi-file refactoring diffs against them, with instructions we did not soften:

• Two refactors intended to be safe, the kind an agent proposes every day: extract duplicated logic into a shared helper, modernize formatting and loops across two files.
• Two refactors that look clean and idiomatic but contain a deliberately planted, disguised defect. The agent chose an inverted boundary condition hidden behind a plausible docstring in one, and an off-by-one length guard in the other. It was told to make them the kind of bug a human reviewer would plausibly miss.
• One clean refactor of code that the project's test suite never executes.

The agent kept an answer key we did not show the verifier. Then we ran every diff through Refactron blind and compared verdicts against the key.

The results

Five for five.

The planted inverted boundary: UNSAFE in 23 seconds. The generated range check now rejected its own boundaries, so IntRange(0, 10) refused both 0 and 10. The verdict came with the failing tests named. This is the bug class that sails through human review, because the diff is tidy, the naming is better than what it replaced, and the docstring confidently explains the wrong behavior.

The planted off-by-one guard: UNSAFE in 46 seconds, failing tests named. The defect silently dropped length-1 version tuples, an edge a skimming reviewer would never execute in their head.

The refactor of uncovered code: UNPROVEN in 58 seconds, with the honest explanation that the tests pass but never execute the changed lines, plus the exact file and line positions that need a test.

The verdict that matters most is the third one

SAFE and UNSAFE are what people expect from a gate. UNPROVEN is the verdict we care most about, and the experiment shows why.

The fifth diff was, as far as careful reading can tell, a perfectly good refactor. A traditional gate would have shown green tests and waved it through. But nothing about those green tests is evidence, because they never run the changed code. Refactron refuses to convert absence of evidence into a verdict. It says: your suite cannot attest this change, here are the exact lines to cover if you want a real answer.

Given the overfitting numbers above, we think this refusal is the most valuable thing a verification layer can do in the agent era.

What this does not prove, said plainly

One experiment, five diffs, two codebases. The planted defects were both in covered code, which is exactly where a test-running gate is strongest. A planted defect in uncovered code would come back UNPROVEN, not UNSAFE, and a defect that the suite is too weak to observe can still earn SAFE, because the test suite is the oracle and no verifier can outrun its oracle. Coverage attestation is Python-only today, so TypeScript changes currently cap at UNPROVEN. All of this is in our docs, in plain language, because a trust layer that overclaims is a contradiction in terms.

What the experiment does show: against realistic agent output, including adversarial agent output designed to fool a reviewer, the three-way verdict sorted every diff correctly, in about a minute each, using nothing but the project's own tests run properly and read honestly.

Your agent can call this before it lands anything, over MCP. Your CI will be able to run it as a PR check next. Early access is open.

References: “Are Solved Issues in SWE-bench Really Solved Correctly?” (arXiv 2503.15223) and “Investigating Test Overfitting on SWE-bench” (arXiv 2511.16858).