What five real codebases taught us about verifying other people's diffs
Field notes from pointing a diff verifier at Click, Rich, attrs, httpx, and SQLAlchemy: what running other people's test suites actually takes.
For the past two weeks we have been doing one thing: taking Refactron, our verification layer for code changes, and running it against real open-source codebases with adversarial intent. Not demos. Click, Rich, attrs, httpx, SQLAlchemy, plus diffs authored by an actual coding agent with planted defects.
The verdict engine came out 27 for 27 on pre-registered expectations after the fix rounds. But the interesting part is everything we learned about real codebases along the way. If you are building anything that runs other people's test suites, this list is for you.
1. Modern Python will break your import analysis on day one
Type-checking-only imports are everywhere. Files import typing_extensions inside if TYPE_CHECKING: blocks, and that package frequently is not installed at runtime because it does not need to be. Platform-conditional modules like msvcrt exist only on Windows. If your analysis walks the whole AST and tries to resolve everything, every modern Python file is “broken” on some machine.
Our fix was borrowed from how good reviewers think: only blame the diff for what the diff changed. The imports check now compares the base state against the changed state and fails only on imports the change introduced or newly broke. Pre-existing quirks belong to the repo.
2. Coverage tooling fails in ways that look exactly like answers
Three separate discoveries here, each of which produced a wrong-looking verdict before we fixed our handling:
Rich's test suite executes dynamically compiled code with a phantom filename. Coverage records it, then refuses to emit a JSON report because the “file” has no source on disk, unless you explicitly ask it to ignore those entries. Swallow that error and you silently get zero coverage for everything.
A folder named coverage in the working directory, which is what vitest and jest name their HTML output, imports cleanly as an empty Python namespace package. Any probe that checks “can I import coverage” gets fooled. Probe whether the tool can execute instead.
Editable installs redirect imports back to the original checkout. If you verify a change in an isolated copy of a repo that was pip-installed with -e, the tests import the original unchanged code and happily attest the wrong tree. You have to control the import path deliberately.
None of these were bugs in coverage.py or in the projects. They are the reality of composing tools that each behave reasonably alone.
3. Formatting diffs are the easiest hard case
A whole-file black reformat looks terrifying in a diff view: 74 changed lines in our Click test case. It verified SAFE in 23 seconds, because every reflowed line sits on covered paths and the suite stays green. Same for isort. This matters because reformat diffs are the most common large automated diffs in existence, and a gate that panics on them trains people to ignore it.
The mirror case also behaved: merge conflict markers accidentally left in a file, which is a classic agent failure, died at the syntax check in one second.
4. Deletions are not edits, and pretending they are is a lie
A diff that deletes a file while also making one harmless edit is not “one harmless edit.” Our parser used to skip what it did not model, and the result was a SAFE verdict on a diff that, applied for real, broke every import in the package. Current behavior: deletions, renames, copies, and binary changes are refused loudly, with the operation and path named, until the day we can genuinely verify them. Refusing honestly beats verifying partially every time.
Removal-only diffs got their own honest answer too. Deleting code cannot be attested by tests the way added code can, so the report now says exactly that instead of implying a coverage miss.
5. Scale changes the problem completely
SQLAlchemy's full suite is 24,257 tests and took about 23 minutes on our hardware. Verify a change by running the suite before and after and you are past 45 minutes per diff, before coverage measurement. That is not a verdict pipeline, that is an overnight batch job.
Scoped to the affected subsystem, the same machinery verified a behavior-preserving rewrite in SQLAlchemy's expression internals as SAFE in 54 seconds, and caught a planted one-character precedence bug (a less-than that should have been less-than-or-equal) as UNSAFE in 22 seconds with the exact failing test named. Change-scoped test selection is the difference between a tool for libraries and a tool for monorepos, and it is the next big thing on our roadmap for exactly this reason.
6. The flakiness problem is bigger than the correctness problem
Google's testing research reports that 84 percent of pass-to-fail transitions in their CI were caused by flaky tests, not real regressions. For a verification gate this is existential: blame someone's diff for a flake twice and they will uninstall you. We are building failure-set comparison and retry-on-new-failure into the gate so that only failures the change actually introduced, reproducibly, can produce an UNSAFE. Flaky suspects get surfaced as a note instead of a verdict.
7. Know what your oracle can and cannot say
The deepest limit is not ours to fix: the test suite is the oracle. Research on agent-written patches shows a substantial fraction pass every available test and are still wrong, because the tests are weak. We say this in our docs in plain words, because the alternative is selling certainty we do not have.
Where this lands
Nine real bugs found in our own engine across these campaigns, every one fixed and locked with a regression test, including two that could have produced false SAFE verdicts. Five codebases, 27 adversarial cases, all correct after the fixes. An agent-authored experiment where both planted defects were caught in under a minute.
How we adversarially test a tool whose only job is to be honest: 27 pre-registered cases on five real codebases, nine real bugs found and fixed, two of them proven false SAFEs.
Five agent-authored refactors against Click and attrs: two clean, two with disguised planted defects, one touching untested code. The verdicts went five for five.
We will keep publishing these field notes as the hardening continues. If you want the gate on your repo, your agent, or eventually your CI, early access is open, and the docs describe exactly what each verdict does and does not promise.
References: Google flaky-test statistics via “Test Flakiness Causes, Detection, Impact and Responses: A Multivocal Review” (arXiv 2212.00908); SWE-bench patch-correctness audits (arXiv 2503.15223, arXiv 2511.16858).