We spent two weeks trying to make our own verifier lie
How we adversarially test a tool whose only job is to be honest: 27 pre-registered cases on five real codebases, nine real bugs found and fixed, two of them proven false SAFEs.
Refactron has one job: you hand it a diff, it hands back a verdict. SAFE means your tests pass and they actually exercise the changed code. UNSAFE means something broke, and it shows you what. UNPROVEN means your tests pass but never touch the changed lines, so nobody has evidence either way.
A tool like that has exactly one failure mode that matters. If it ever says SAFE about a change that is not safe, it is worse than useless, because people trust it precisely at the moment they should not. So before we asked anyone to trust it, we spent two weeks trying to make it lie to us.
This is what that looked like, and what we found.
The method: pre-registered expectations
Every test case we ran was written down with its expected verdict before we ran it.
| The change | Must come back |
|---|---|
| Behavior-preserving rewrite of covered code | SAFE |
| Subtle break in covered code | UNSAFE, with the failing test named |
| Edit to code no test executes | UNPROVEN, with the exact uncovered lines listed |
We ran this against five real codebases, not fixtures we control: Click (1,917 tests), Rich (956 tests), attrs (1,382 tests), httpx (1,416 tests), and SQLAlchemy (24,257 tests). Real projects, real test suites, real coverage profiles, real weirdness.
What broke, honestly
The engine got the easy cases right on day one. The findings came from the seams, and some of them were exactly the kind of quiet lie we were hunting for.
A false SAFE from Windows line endings. On a CRLF checkout, an LF-authored edit made our changed-line detection think every line in the file had changed. The inflated set reached covered lines, so an edit to completely untested code came back SAFE. Any Windows repo verifying an LLM-authored diff would have hit this, because LLMs emit LF. We caught it the first time the full CI matrix ran on Windows, fixed it with line-ending normalization, and locked it with tests in both directions.
A false SAFE from silent deletion-skipping. Our diff parsing modeled content edits and quietly ignored operations it did not understand. A diff that deleted a module while also making one innocuous edit verified SAFE on the strength of the innocuous edit alone. Applying the same diff for real broke every import in the package. We proved it, then changed the rule: any operation the verifier cannot model (file deletions, renames, copies, binary changes) is now refused loudly with a named message and a non-zero exit. A partial verdict must never read as a verdict on the whole diff.
False UNSAFEs from being naive about real Python. Modern Python is full of imports inside if TYPE_CHECKING: blocks that never execute, and platform-conditional imports like msvcrt that cannot resolve on the “wrong” OS. Our imports check flagged both, blaming changes for imports they never introduced. On Click, any edit to a file with a typing-only import came back UNSAFE. The fix was to make the check delta-aware: only imports introduced or newly broken by the change can fail it. Pre-existing conditions belong to the repo, not to the diff.
A silently broken coverage pipeline. Rich's test suite executes dynamically compiled code with a phantom filename, which makes the coverage tooling refuse to emit its report. Our reporter swallowed that failure and returned zero covered lines, which silently downgraded every SAFE on such projects to UNPROVEN. Different direction than a false SAFE, still a lie. Fixed, with the failure mode reproduced in a regression fixture.
A probe fooled by a folder name. This one is my favorite. If the directory you run the CLI from contains a folder literally named coverage (which is what vitest and jest name their HTML output folder, so nearly every JS repo has one), Python happily imports it as an empty namespace package. Our “is coverage.py installed” probe passed, the real coverage run failed, and the verdict carried a misleading reason. The probe now tests whether the tool can actually execute, not whether something with the right name can be imported.
The scoreboard
After the fix rounds, we re-ran everything: 27 pre-registered adversarial cases across the five codebases, 27 correct outcomes. Nine real bugs found, fixed, and locked with regression tests, including two proven false SAFEs that would have quietly destroyed the product's entire reason to exist if a user had found them first.
Some cases we were happy to see pass without drama.
| Case | Verdict | Note |
|---|---|---|
| 74-line black reformat of a covered module | SAFE | the suite attests every reflowed line |
| isort import shuffle | SAFE | |
| Leftover merge conflict markers | UNSAFE | dead in one second at the syntax check |
| Kwarg rename applied across definition, callers, tests | SAFE | with a note that the diff touches test files |
| The same rename with one missed caller | UNSAFE | in 13 seconds |
Why we publish this
Every verification tool is a claim about honesty, and claims about honesty are cheap. What is not cheap is a written record of the exact ways your own tool lied, what you did about each one, and a test suite that keeps them dead. We would rather show you the bug list than a landing page full of adjectives.
We found them because we went hunting on real codebases with pre-registered expectations. That habit is now permanent. Every future release gets the battery.
Refactron is in early access. The docs cover what the verifier checks and what its verdicts mean, including its current limits (coverage attestation is Python-only today, so TypeScript diffs cap at UNPROVEN). If you want the gate on your repo or your agent, the early-access list is open.