← 论坛

Repro gate (step 10) fails when one experiment reads another's results; step 5 never states that rule

综合

Yuming Zhou

Kit version: 0.15.2 (e2db8cc). The relevant code is unchanged on main (0.15.8).

What happened

The writing pipeline (theory/computational study, AC_AUTHOR=1) stopped at step 10/15. check_reproduction.py re-ran 15 experiments from runs/REPLAY_MANIFEST.json:

  • 13 passed, with several thousand exact numbers matching and 0 mismatches.
  • 2 crashed with FileNotFoundError before computing anything. Each is a follow-up analysis that reads another manifest entry's output, e.g. script B opens runs/results/A.json to compare against or combine with A's results.

The gate runs each entry alone in a fresh copy with results/ removed (SKIP in check_reproduction.py), so B can't find A's output. That behaviour is deliberate. The problem is that the agent is never told about it.

Gaps

  1. Step 5's prompt doesn't state the rule. pipeline/run-pipeline.sh (the REPLAY_MANIFEST.json part of step 5) only says the gate "re-runs each script from a clean copy". It doesn't say that results/ is removed, that entries run independently with no order, or that a script must not read another experiment's results. Splitting a slow computation from the analyses built on it is a natural design, so the agent did exactly that and only found out at step 10, after the whole sweep had run (about 7 hours here).
  2. The stop message is misleading. Step 10 prints research: the numbers did not reproduce. Fix the experiment, not the paper. for any gate failure, including a script that crashed. The note in state/ASK_HUMAN.md shows the last ~60 lines of output, which here were all timing drifts (per_round[0].t_s, recorded 0.11 / re-run 0.133). Timings are never compared, but they look like the cause. The two crashed entries and their stderr_tail (the real cause) weren't in the visible part.
  3. The gate isn't retried and the agent isn't told why. This is by design, but the owner has to read runs/REPRO_GATE.json themselves to find which step to go back to.

Suggestions

  • Add one sentence to step 5's prompt: "Each manifest entry is re-run alone, in a copy without runs/results/; a script must compute everything it reports and may not read another entry's results file."
  • Or support dependencies: an optional "after": ["scripts/A.py"] that runs A first in the same copy.
  • Make the step-10 message depend on what failed (crash vs number mismatch vs unclassified number), and put the FAIL entries' script/why/stderr_tail at the top of the ASK_HUMAN.md note instead of the output's last lines.
回复

3 条回复

AutoConference官方#1

Thanks for the precise report, it's fixed in kit 0.15.9: step 5 now tells the agent that each experiment is re-run alone without runs/results/, and an analysis can declare "after": ["scripts/A.py"] so it re-runs after the entry it reads. Step 10 now names the failing script and its fix instead of timing drifts, and lets the agent repair such crashes itself (at most twice), re-running only what failed. To continue, run ./ac update and delete work//PIPELINE_STOPPED.

AutoConference官方#3
回复 匿名读者 #2Still encounter this issue, even with kit 0.16.0

Thanks for following up, and sorry for the confusion. We checked what your agents reported: on kit 0.18.1, step 10 no longer crashes on a missing file. It stops because, when the gate re-ran the experiments, some numbers declared "exact" came out different (in one paper after the agent had repaired a missing module). The gate treats that as a finding and sends the paper back to step 5 rather than letting numbers through that do not reproduce. Could you share the failing entries from runs/REPRO_GATE.json (the field, its recorded and re-run values, and what that script computes)? You can send it to our official email. Then we can tell whether these results really vary from run to run (seeds, GPU kernels, threads), which step 5 should fix or declare "tolerant", or whether it comes from how the gate re-runs them. Separately, while checking we fixed a gap in "after" in kit 0.19.0: an analysis now also gets the other files its experiment writes, not only its declared output.

发表回复

登录后可以回复。