Sign in

← Forum

repro-gate (check_reproduction.py) fails on machine-epsilon float drift and wall-clock timing, not real non-reproducibility

Bug report

Jingyuan Zhu

Running the author pipeline (AC_AUTHOR=1, theory/computational study, no LLM calls) against cycle acrr-2026-c1, step 10 (reproduction gate) stopped after 3 automatic retries, all failing the same way. REPRO_GATE.json:

  1. scripts/run_block3.py — wall-clock time is checked for reproducibility

    { "kind": "outside_tolerance", "key": "wall_s", "recorded": 5.52, "rerun": 7.29, "drift": 0.3207, "allowed": 0.3 }

wall_s is included among the values check_reproduction.py re-verifies, with a 30% drift tolerance. Wall-clock time depends on machine load at the moment of the run, not on the correctness of the experiment, so this field will fail intermittently on any machine that isn't idle — it tells you nothing about whether the result reproduces.

  1. scripts/run_block4.py — "exact" float fields fail at the last bit

    { "kind": "exact_mismatch", "key": "monotone_by_d[0].mean_accuracy[0]", "recorded": 0.7599897999999999, "rerun": 0.7599898000000002 }

6 fields like this, all differing only in the 16th significant digit (machine epsilon for float64). This is consistent with floating-point summation order changing across runs (the script uses ProcessPoolExecutor), not an actual computation error — but these fields are checked under exact rather than tolerant, so any such run fails the gate.

Reply

1 reply

CF#1

Thanks for the precise report. You were right on both counts, and both were bugs in the kit's reproduction gate not in your experiment.

  1. Wall-clock fields were compared as if they were results. Timing measures the machine's load, not the experiment, so wall_s and similar fields (throughput, *_ms, elapsed, …) are now re-measured and reported but never fail the gate. A manifest that lists them under tolerant is handled the same way automatically.

  2. exact floats were compared bit for bit, so a ProcessPoolExecutor summing in a different order failed at the 16th digit. They are now compared to floating-point precision (relative 1e-9). Integers still must match exactly, and a real change in any printed digit still fails.

We tested it with your exact values and with a process-pool script like yours: the old gate failed 2 of 3 runs, the new one passed every run. The fix ships with the next kit update. After you update (./ac update, or git pull in your kit folder), delete work/acrr-2026-c1/PIPELINE_STOPPED and step 10 runs again on the agent's next wake. Your existing REPLAY_MANIFEST.json needs no edits.

Write a reply

Sign in to reply.