Paper of the Week — Try Again, Don't Look Back: Blind Resampling Outperforms Self-Repair in Small Code Models

Yuvraj Verma. Published 2025. arXiv:2607.26117

One sentence summary

When you control for the number of attempts, self-repair — feeding failed test output back to the model — offers little to no advantage over blind resampling in smaller code models, and the feedback tokens are often wasted compute.

Why this paper

Self-repair is essentially table stakes in every code agent pipeline right now. If that assumption is wrong, you’re paying for context overhead and added latency that isn’t buying you anything.

What they did

Self-repair evaluations almost always compare “repair with feedback” against “no retry at all,” which confounds two things: the value of the feedback signal versus the value of simply getting another attempt. This paper isolates the variable by running a controlled comparison — self-repair versus blind resampling at the same attempt budget — across multiple small code models on standard benchmarks. The question is specifically whether the test-output feedback is doing any meaningful work.

Key findings

Blind resampling matches or outperforms self-repair on pass@k for small models across tested benchmarks when attempt counts are held equal
The feedback context (error message + failed code) consumes significant tokens without a corresponding accuracy lift for smaller models
Self-repair’s apparent advantage in prior literature largely evaporates once the retry-count baseline is properly controlled
The result holds across different model sizes in the small-to-mid range; larger models show more consistent benefit from feedback, suggesting a capability threshold
Latency and cost per solved problem favor blind resampling in the regimes where the gap is negligible

Why it matters for practitioners

Most code agent scaffolds today default to feeding test failures back into the model loop without questioning whether that’s better than a clean retry — and that feedback round adds latency, burns context window, and costs tokens. If you’re routing to a smaller or mid-tier model (think GPT-5.4 Nano, Haiku 4.5, or a self-hosted Qwen3.6 variant) for cost reasons, blind resampling with a higher sample budget may be both cheaper and faster for the same solve rate. The implication for agent design is concrete: decouple your retry logic from your feedback logic, and measure whether repair is earning its overhead on your specific model tier.

What you can use today

Add a blind-resample baseline to your code agent eval harness before assuming self-repair is worth the context cost — compare pass@k at equal attempt budgets, not repair vs. no-retry
If you’re on a smaller model tier, try increasing your sample budget (or temperature) instead of implementing error-feedback loops; you may get equivalent solve rates at lower latency
When you do use self-repair, gate it on model capability: reserve feedback loops for your most capable frontier calls and fall back to resampling for cheaper routing tiers

Try Again, Don’t Look Back: Blind Resampling Outperforms Self-Repair in Small Code Models