Kit version: 0.15.2 (e2db8cc). The relevant code is unchanged on main (0.15.8).
What happened
The writing pipeline (theory/computational study, AC_AUTHOR=1) stopped at step 10/15. check_reproduction.py re-ran 15 experiments from runs/REPLAY_MANIFEST.json:
- 13 passed, with several thousand
exactnumbers matching and 0 mismatches. - 2 crashed with
FileNotFoundErrorbefore computing anything. Each is a follow-up analysis that reads another manifest entry's output, e.g. script B opensruns/results/A.jsonto compare against or combine with A's results.
The gate runs each entry alone in a fresh copy with results/ removed (SKIP in check_reproduction.py), so B can't find A's output. That behaviour is deliberate. The problem is that the agent is never told about it.
Gaps
- Step 5's prompt doesn't state the rule.
pipeline/run-pipeline.sh(theREPLAY_MANIFEST.jsonpart of step 5) only says the gate "re-runs each script from a clean copy". It doesn't say thatresults/is removed, that entries run independently with no order, or that a script must not read another experiment's results. Splitting a slow computation from the analyses built on it is a natural design, so the agent did exactly that and only found out at step 10, after the whole sweep had run (about 7 hours here). - The stop message is misleading. Step 10 prints
research: the numbers did not reproduce. Fix the experiment, not the paper.for any gate failure, including a script that crashed. The note instate/ASK_HUMAN.mdshows the last ~60 lines of output, which here were alltimingdrifts (per_round[0].t_s, recorded 0.11 / re-run 0.133). Timings are never compared, but they look like the cause. The two crashed entries and theirstderr_tail(the real cause) weren't in the visible part. - The gate isn't retried and the agent isn't told why. This is by design, but the owner has to read
runs/REPRO_GATE.jsonthemselves to find which step to go back to.
Suggestions
- Add one sentence to step 5's prompt: "Each manifest entry is re-run alone, in a copy without runs/results/; a script must compute everything it reports and may not read another entry's results file."
- Or support dependencies: an optional
"after": ["scripts/A.py"]that runs A first in the same copy. - Make the step-10 message depend on what failed (crash vs number mismatch vs unclassified number), and put the FAIL entries'
script/why/stderr_tailat the top of theASK_HUMAN.mdnote instead of the output's last lines.