Research: evidence-guided candidate ranking for Own.NET analyzer fixes (LoopRepair prior art) #440

Classification

• Type: bounded research / analyzer-fix evaluation spike
• Priority: P2 / backlog; design/pilot only, no automatic agent integration or production code changes by default
• Area: Roslyn extractor → OwnIR → ownership analysis → diagnostic evidence / regression testing
• Prior art: Fino2020/LoopRepair, Well Begun is Half Done: Location-Aware and Trace-Guided Iterative Automated Vulnerability Repair (ICSE 2026)
• Related Own.NET: Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 (semantic mutation harness), Research: refactoring-aware metamorphic regression testing for ownership analysis (RefactoringMiner prior art) #432 (refactoring-aware metamorphic tests), P-022 diagnostics: normalized AnalysisTrace and first-divergence minimizer for Python↔Rust shadow mode #269 (first-divergence trace), Research: evaluate Bifrost for C# code intelligence, agent navigation, and evidence contracts #426 (evidence/completeness), Research: SootUp-inspired OwnIR/ICFG semantic-fidelity and incomplete-call audit (bounded spike) #430 (source-to-OwnIR/ICFG fidelity)
• Default: learn from candidate search/ranking, do not port LoopRepair, add a patch agent, or equate empirical test success with soundness.

Problem / hypothesis

Own.NET analyzer changes can address an observed false negative yet regress adjacent alias, branch, interprocedural, reporting or false-positive behavior. An iterative coding agent can generate multiple plausible edits, but simply keeping the most recent compiling patch, or the patch that makes one failing fixture green, is a weak selection strategy.

Hypothesis: for a few independently validated analyzer defects, evaluating multiple failed or partially successful candidate fixes against structured independent evidence may produce a more reliable next candidate than sequential last-attempt-only repair, without claiming any new proof of analyzer soundness.

LoopRepair's transferable architectural ideas:

1. Location-aware change proposals: observed failure site and actual repair site can differ (for Own.NET, the discrepancy may begin in Roslyn extraction, OwnIR, a transfer rule, a method summary, or diagnostic reporting).
2. Trace-guided evaluation: retain and inspect failed candidates; measure specific effects on the failing behavior rather than discarding every unsuccessful patch.
3. Iterative selection: rank a small pool using independently collected evidence before selecting one for a further iteration.

The upstream implementation targets C/C++ vulnerabilities and uses CrashRepair/KLEE plus LLM-generated source patches. Those dependencies, metrics, and C/C++ assumptions are not suitable for direct adoption. Its repository implementation should not be treated as a reference oracle: e.g. scenario.py currently computes an impact_lines value from len(readlines) of taint.log, which deserves independent scrutiny rather than copying.

Bounded pilot (reuse #405, not a new harness)

0. Preregister the oracle and baseline

• Pin Own.NET revision, runtime/toolchain, exact fixtures, expected diagnostic/evidence behavior, and commands.
• Reuse Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405's reviewed, compiling semantic mutants and negative controls where applicable. Add at most 3–5 independently validated defect/regression scenarios spanning at least two failure families, e.g. alias/use-after-release, guarded disposal and a call/summary boundary.
• Record each expected result from C# ownership semantics before viewing candidate outcomes. Unsupported/ambiguous cases are not scored.
• Include existing real-world or regression cases if available; synthetic mutant wins must not be advertised as real-world utility.

1. Establish a boring baseline first

For each defect compare:

• Baseline A: straightforward single-candidate fix, evaluated on the target reproducer and the existing suite.
• Baseline B: sequential agent/manual fix attempts with identical budget; keep last passing candidate.
• Experimental: generate or collect the same bounded candidate set and evaluate/rank before choosing a follow-up.

If multiple candidates cannot be generated fairly or budget equivalence is impossible, record the limitation rather than reporting a win. LLM use is optional; hand-authored small diffs suffice to test the ranking idea first.

2. Evaluate every candidate as a reversible, isolated patch

At minimum record:

• candidate ID, parent/base revision, patch diff, modified subsystem and hypothesis;
• applies / builds / tests / times out / infrastructure failure (separate states);
• target diagnostic change including diagnostic category, location, evidence/path and affected resource, not merely count or exit code;
• previously detected unsafe cases retained or lost (regression);
• newly detected pre-registered mutants from Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 (positive cases);
• spurious diagnostics on clean/equivalent negative controls (precision regression);
• first known divergence along source → Roslyn facts → OwnIR/CFG/ICFG → analyzer state → evidence/report, where available (P-022 diagnostics: normalized AnalysisTrace and first-divergence minimizer for Python↔Rust shadow mode #269/Research: SootUp-inspired OwnIR/ICFG semantic-fidelity and incomplete-call audit (bounded spike) #430);
• cost: runtime, number of analyzer invocations, optional model tokens.

Use fresh worktrees or equivalent transactional restoration; verify repository cleanliness after each attempt. Preserve all failures and their reproducers. No mutation score, coverage percentage, or passing tests may be promoted to a soundness proof.

3. Rank transparently, with hard gates

First reject invalid patches (cannot apply/build/run, corrupted harness, oracle-invalid scenarios). Prefer Pareto or lexicographic comparison over an arbitrary magic weighted sum:

1. No independently established safety-detection regression on held-out positive cases.
2. No unexplained FP/evidence regression on negative controls.
3. Correct repair of target reproduced defect, including witness/evidence.
4. Additional independent mutation families addressed (not duplicate examples).
5. Lower complexity/cost only as a tie-breaker.

Keep unresolved, known limitation, infrastructure error, FP, FN, regression and valid improvement distinct. If all candidates fail hard gates, emit STOP / no safe candidate rather than choosing the least bad patch. A non-dominated set can be useful even if no total order exists.

4. Run a tiny matched comparison

Run on 3–5 pre-registered scenarios, with identical candidate/time/test budgets, and report:

• number of independently validated defect fixes;
• preserved vs lost previously detected unsafe cases;
• added false positives / evidence regressions;
• how often ranking genuinely selected a better candidate than baselines;
• cost relative to benefit; cases where no approach worked;
• whether the ranking was driven by the oracle, by test-suite leakage, or by one repeated mutation pattern.

If enough independent cases do not exist, stop at a design/protocol report; don't fabricate statistical conclusions.

Deliverables / acceptance criteria

• Small, documented evaluation protocol reusing Research: semantic mutation testing harness for Own.NET analyzer coverage (bounded pilot) #405 and existing Own.NET entry points; no parallel IR or test runner.
• Pinned fixture list, independent oracle, candidate budget and baselines recorded before results.
• Reproducible per-candidate JSON/Markdown evidence including all rejected and unresolved states.
• One compact comparison table for 3–5 scenarios or a documented reason the pilot cannot be evaluated.
• Explicit soundness caveat and regression accounting; no claims of proved safety, fixed CVE count, or universal accuracy.
• Recommendation PROMOTE / NARROW / STOP based on observed incremental value and maintenance/runtime cost.

Stop / promotion gate

STOP if #405's corpus/oracle isn't mature enough, candidates cannot be fairly compared, ranking only rediscoveries existing tests, false-positive/safety regressions go undetected, or the implementation would require a generic LLM repair service / KLEE port.

PROMOTE only with a reproducible, independently validated improvement over matched baselines on at least one previously uncovered analyzer gap without unacceptable regressions, followed by a separate implementation proposal. Even then, call it a promising engineering method, not evidence of soundness.

Out of scope: importing LoopRepair, automatic code rewriting in production, online PR bot, unconditional CI gate, speculative new OWN diagnostics, global repair search, concolic executor implementation, broad fuzzing/LLM benchmarking.