Reflexion-SWE · mechanism ablation
Verbal reflection without weight updates
Actor proposes a patch; the evaluator generates up to six AST-filtered tests and runs them in a sandbox; failures become a verbal lesson in a three-slot episodic memory. Training-free GRPO distills group-level lessons into a weighted prior bank.
| Mode | Pass@1 | Mean attempts | What changed |
|---|---|---|---|
| base | 0.000 | 1.00 | No executable feedback; self-approval |
| -TestGen | 0.000 | 1.00 | No generated tests; self-approval |
| -Reflection | 0.000 | 3.00 | Tests run; no verbal lesson; repeats bug |
| full | 1.000 | 2.00 | Test failure → lesson → fixed patch |
| +TF-GRPO | 1.000 | 1.00 | Group priors select the passing shape first |
Example trace
task: toy-add (Return the sum of two integers.)
attempt 1: evaluator 0/2 passed — "assert add(2, 3) == 5" failed
reflection: "The previous patch failed because 1 test(s) failed. The next attempt must fix the failing behavior, keep the entry point signature, and add no new dependencies."
attempt 2: evaluator 2/2 passed — final hidden tests 2/2 passed
patch: def add(a, b): return a + b
These are deterministic fixture results, not HumanEval Rust or SWE-bench model results. The model-dependent run is explicitly pending.
arXiv:2303.11366arXiv:2510.08191arXiv:2310.06770 Source: Reflexion-SWE