I encountered a reproducible out-of-memory failure in Step 14 when running AutoConference under WSL2.
Step 14 invokes: check_reproduction.py --claims-only
However, result_entries() recursively scans every .json and .jsonl file under runs/. It parses every numeric leaf and stores tuples containing the value, metric-name set, and full field path in a single in-memory list.
In my computational study, the results directory contained:
- 52 JSONL files totaling approximately 834 MB
- A 417 MB predictions.jsonl
- Approximately 1.46 million prediction records
- Per-record labels, predictions, seeds, folds, and logits
Although the raw files were under 1 GB, expanding millions of values into Python strings, sets, tuples, and numbers exceeded the 7.6 GiB WSL memory limit. WSL then terminated with:
Catastrophic failure
Error code: Wsl/Service/E_UNEXPECTED
I locally changed the claims-only index and evidence mirror to read compact .json result artifacts only, excluding prediction-level .jsonl files. This reduced the eligible input from approximately 834 MB to 8.8 MB, and Step 14 subsequently completed in about 21 seconds and passed.
Possible fixes:
One possible approach would be to keep --claims-only focused on compact, manifest-declared result artifacts rather than raw prediction files. Prediction-level JSONL could be skipped by default and processed as a bounded stream only when explicitly cited. It may also be helpful to add a size guard before scanning unusually large result trees. Longer term, exact source-and-field provenance could make the check both more efficient and less likely to associate a printed value with an unrelated occurrence elsewhere in the results.
The last point may also prevent incorrect attribution: a broad numeric search can match a printed value to an unrelated seed value, standard deviation, or configuration merely because the rounded number happens to occur elsewhere in the results tree.