) research

What a verdict proves, and what it doesn't

Refactron returns SAFE, UNSAFE or UNPROVEN by running your tests and measuring which changed statements they executed. This page is about what that does not establish: the limits of the mechanism, the eleven false SAFE verdicts we have shipped and fixed, and the fact that the central premise of this whole product category has never been published on by anyone, us included.

What it doesThreat modelLast updated August 28, 2026

What SAFE means, exactly

A diff is applied to an isolated copy of your repository. Three gates run there: syntax, imports, and your own test suite. Then changed-statement coverage is fused in. SAFE means every gate passed and every changed statement a test could reach was executed by a test that passed.

Four things SAFE does not mean

  • Not correctness. A verdict inherits exactly what your tests check, no more. If your suite does not assert the behaviour the diff changed, SAFE says nothing about that behaviour.
  • Executed, not asserted. Coverage proves a changed statement ran. It does not prove any test would fail if that statement behaved differently. A test that calls the changed function and asserts nothing produces the same coverage as one that checks every case. This is the sharpest limitation of the mechanism and it is tracked as issue #116.
  • Statement-level, not branch-level. A changed if whose true branch never executed still counts as covered.
  • Narrowing detection is a check, not a guarantee. We read the test command, the environment and your pytest configuration, and know the common flags of pytest, unittest, vitest and jest. A command using a flag we do not recognise reports unknown, and unknown does not cap the verdict.

There is a fifth, and it is the largest in practice: coverage attestation is Python only. A TypeScript or mixed diff runs every gate and caps at UNPROVEN permanently. We do not currently publish what share of real runs that affects, because we have not measured it.

What the report does not carry

A verdict report carries reportVersion, the verdict, all three gate results, changedFiles, testFilesChanged, changed-statement coverage, the reason, testScope, and engineVersion. It carries no hash of the base tree, no hash of the diff, and no signature.

So the report is not an attestation in the cryptographic sense and is not tamper-evident. You cannot hand it to a third party as proof that a particular tree produced a particular verdict. What it does give you is engineVersion, which names the rules that produced the verdict, so a stored SAFE from a year ago still tells you what standard it was held to. That matters because two releases have changed what SAFE means without changing the report's shape.

Eleven false SAFE verdicts, itemised

A false SAFE is the one defect this product cannot have. We have shipped eleven, counted the way each release counted them, and found every one ourselves.

  • 0.3.12 false SAFEs

    Coverage measured a different program than the tests gate ran, then reported the changed lines as covered. A leading PYTHONPATH= on the test command silently disabled measurement, and a console entry point was measured under an interpreter the shell would never have chosen.

  • 0.4.15 false SAFEs

    A narrowed test command, a --collect-only run that executed nothing, an ambient PYTEST_ADDOPTS filter, a unittest invocation naming one module, and a file where 1 of 40 changed statements ran. All five returned SAFE with a passing suite and measured coverage.

  • 0.4.24 false SAFEs

    Found by an adversarial review of the 0.4.1 narrowing check itself. The scanner stopped at the first flag it did not recognise and discarded any filter after it, so one stock pytest flag disabled the whole check.

What Refactron is not

Not an AI reviewer. No language model reads your code. We emit no findings and no suggestions, so we cannot be scored on the metric that category publishes. Tools like CodeRabbit and Greptile answer a different question, and neither executes your test suite.

Not a replacement for a patch-coverage gate. On Python, diff-cover and Codecov patch coverage compute a very similar number, and Codecov does it in one line of configuration for free. We have not measured whether our fusion adds signal over theirs, and the honest prior is that on the coverage number alone it does not. What differs is the rest: an isolated tree that never touches your working copy, syntax and imports gated before a long suite runs, a diff that fails to apply reported as such, and a verdict an agent can call over MCP before a pull request exists.

Not “safe” in the sense regression test selection means it. That field has used the word for two decades to mean a selection that never omits a test which could be affected by the change. That is strictly stronger than anything we claim. Our narrowing detection is a partial, undocumented cousin of test selection and it is not safe in that sense. We are using a taken word and this paragraph is the correction.

The open question

The premise underneath this product, and underneath every patch-coverage gate, is that coverage of the changed lines specifically tells you something useful about whether a change is safe to land. We are not aware of any published study establishing that, for us or for anyone.

What the literature does contain is a well-known challenge to the broader claim, a framework naming the exact gap our mechanism leaves, a 2026 replication finding that coverage stops being a reliable indicator in precisely the setting where the code under test may already be wrong, and a registered report that will ask the causal question and has not yet run. None of that settles the narrower question about changed lines. It is simply open.

Our mechanism competitors have published no evaluation of their own at all. That is not a point in our favour; it means the whole category rests on an unmeasured premise, and we are part of the category.

What we are measuring next

Two studies, both cheap, both capable of producing a result we would rather not publish. Stated here before the numbers exist, with what each would mean if it goes against us.

StudyQuestionIf it goes badly for us
Assertion strippingTake diffs that returned SAFE. Delete every assertion from the tests that covered the changed statements. Re-run. What share is still SAFE?We expect close to all of them, and that is the point: it puts a number on issue #116 instead of a caveat. The figure and the per-repository spread get published whatever they are.
Metamorphic conformanceRelations that must hold regardless of any label. A diff that only deletes tests must never be SAFE. A whitespace-only change must not alter the verdict. Verifying two diffs together must never be stronger than verifying the weaker one alone.Every violation is an unambiguous bug, not a matter of judgement, and each one goes into the list above.

References

Each of these was fetched and read before it was cited, and each entry says only what the source supports. A research pass proposed roughly a dozen further citations that are absent here because they could not be verified in time, several of them carrying numbers that pointed conveniently in our direction.

  1. [1]L. Inozemtseva and R. Holmes, “Coverage Is Not Strongly Correlated With Test Suite Effectiveness”, ICSE 2014

    The canonical challenge to coverage as a quality signal. We cite the claim its title makes and nothing further: the ACM page would not serve to us, so the figures inside it are not quoted here.

  2. [2]K. Jain, G. T. Kalburgi, C. Le Goues and A. Groce, “Mind the Gap: The Difference Between Coverage and Mutation Score Can Guide Testing Efforts”, arXiv:2309.02395

    Names the oracle gap: the difference between coverage of an element and its mutation score. That gap is exactly the distance between what Refactron measures and what a reader hopes SAFE means.

  3. [3]J. Zhao, S. Zhou and E. Cohen, “Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)”, ISSTA 2026

    Finds coverage carries signal where the code under test can be assumed sound, and is not a reliable indicator where the goal is to expose a bug the code already contains. Our tests gate lives in the first case; our coverage attestation lives nearer the second.

  4. [4]L. Schulte, G. Fraser and S. Herbold, “Causal Inference for the Effect of Code Coverage on Bug Introduction”, MSR 2026 registered report, arXiv:2602.03585

    A pre-registered protocol on JavaScript and TypeScript projects asking whether coverage causally affects bug introduction. Accepted as a registered report and not yet run, so it has no findings. We cite it as evidence the question is open, not as an answer.