Your tests passed. That is not the same as your change being safe.
Refactron 0.3.0 gives every diff one of three verdicts. The interesting one is the third, because a green check on code no test ran is a lie with a checkmark next to it.
Refactron 0.3.0 is out. You hand it a diff and it hands you back one of three words.
bashnpx refactron@0.3.0 verify-diff . --diff change.diff[SAFE] Tests pass and the changed code is covered.That is the whole product. The interesting part is the other two words, and why there are three of them instead of two.
The problem is not that AI writes bad code
The problem is that the check we all use to decide whether code is good has quietly stopped being sufficient.
“The tests pass” was always a proxy. It worked because a human wrote the change, a human read it, and the tests were a third opinion. Now a large share of diffs arrive already written, already plausible, already passing. The tests are no longer a third opinion. They are the only opinion, and they are being asked to carry weight they were never designed for.
Independent audits of AI-generated patches keep finding the same thing. One empirical study of SWE-bench found that roughly 30 percent of patches which pass the project's tests are still wrong. A detailed manual audit dropped one model's true solve rate from 12.47 percent to 3.97 percent. These are patches that went green.
So “did the tests pass?” is the wrong question. The right one is “did the tests actually run the code that changed?”
Three verdicts, and why the third one matters most
Refactron applies your diff in an isolated copy of your repo, runs your real test suite, and checks whether those tests executed the lines that changed.
| Verdict | What it means |
|---|---|
SAFE | Every gate passed and your tests exercise the changed code. |
UNSAFE | Something broke. You get the failing test output, not a summary of it. |
UNPROVEN | Your tests pass, but they never execute the lines you changed. Nobody has evidence either way, and Refactron says so instead of guessing. |
Most tools would call that third case green. Tests passed, ship it. We think that is the exact moment a verification tool earns its keep or becomes a liability.
UNPROVEN comes with the file and line numbers that no test touched, so the fix is mechanical: write that test, run it again, earn the SAFE.
What it looks like on real code
We did not test this on toy fixtures. Over four hardening rounds we ran it against Click, Rich, attrs, httpx, SQLAlchemy, Django, pydantic, Jinja2, and pytest itself.
One run worth describing. We had a coding agent write five genuine refactoring diffs against Click and attrs. Two were honest. Two contained deliberately planted defects, disguised the way real bugs are disguised: an inverted boundary condition sitting under a confident docstring, and an off by one in a length guard. One was a clean refactor of code the suite never runs.
Verified blind, five for five. The inverted boundary came back UNSAFE in 23 seconds with the failing tests named. It had made IntRange(0, 10) reject both 0 and 10. That is precisely the bug that survives human review, because the diff is tidy and the comment explains the wrong behavior with total confidence.
Five agent-authored refactors against Click and attrs: two clean, two with disguised planted defects, one touching untested code. The verdicts went five for five.
The clean refactor of untested code came back UNPROVEN. Correct answer. The tests really do pass, and they really do prove nothing about it.
Give it to your agent
The same gate runs as an MCP server, so a coding agent can check its own work before it touches your branch.
bashclaude mcp add refactron -- refactron-mcpThe agent calls verify_change with the diff it is about to apply and gets the verdict back as structured JSON. The loop that follows is the point: UNSAFE means it fixes its own change before you ever see it, UNPROVEN means it writes the missing test first, SAFE means it applies with evidence rather than optimism.
Everything runs locally. Nothing leaves your machine, and your working tree is never modified.
Install
bashnpm install -g refactron@0.3.0There is a Python wrapper if your tooling is pip shaped. It still needs Node 18 or newer, because it shells out to the same CLI.
bashpip install refactron==0.3.0What it does not do yet
Coverage attestation is Python only today, through coverage.py. A TypeScript or mixed language diff still runs all three gates, but it cannot reach SAFE: it returns UNPROVEN with the reason that coverage could not be determined. That is the honest answer rather than a flattering one, and closing it is the next thing we are building.
There is no CI gate yet. It is designed and it is next, and the number that made us confident is this: across six runs against a repo carrying a genuine one in three flaky test, we recorded zero false UNSAFE verdicts. A gate that blames your diff for a flake gets uninstalled within a week, so we measured that before building it rather than after.
And the deepest limit, which no verifier escapes: your test suite is the oracle. SAFE means your tests approve this change, not that the change is correct in some absolute sense. We found a clean example of the gap in Jinja2. Change a <= to a < in its truncate filter and the behavior genuinely changes at the exact boundary, yet all 911 of its tests still pass and the changed line is covered, so the verdict is SAFE. Correct about what it claims. Still not a proof of correctness.
Why we publish the limits
Because a verification tool that oversells itself is worse than no verification tool. If you cannot trust the UNPROVEN, you cannot trust the SAFE either.
We spent four rounds trying to make our own engine lie, and it did, three times. Each of those was a false SAFE: a Windows line ending bug that made an untested change look covered, a deletion silently dropped from a diff so the rest verified clean, and a coverage attribution bug where a single changed blank line let an executed neighbour vouch for untested code. Two of the three were introduced by the fix for another one. All three are dead, each with a regression test that was proven to fail before the fix landed.
How we adversarially test a tool whose only job is to be honest: 27 pre-registered cases on five real codebases, nine real bugs found and fixed, two of them proven false SAFEs.
That is the work. Not the demo.
The docs cover what each verdict promises and what it does not. Early access is open if you want the gate in front of your agent, or in your CI when that lands.