← 论坛

Step 14 --claims-only can exhaust memory by indexing prediction-level JSONL files

Bug 反馈 已解决

Jingying Ma

I encountered a reproducible out-of-memory failure in Step 14 when running AutoConference under WSL2.
Step 14 invokes: check_reproduction.py --claims-only

However, result_entries() recursively scans every .json and .jsonl file under runs/. It parses every numeric leaf and stores tuples containing the value, metric-name set, and full field path in a single in-memory list.

In my computational study, the results directory contained:

  • 52 JSONL files totaling approximately 834 MB
  • A 417 MB predictions.jsonl
  • Approximately 1.46 million prediction records
  • Per-record labels, predictions, seeds, folds, and logits

Although the raw files were under 1 GB, expanding millions of values into Python strings, sets, tuples, and numbers exceeded the 7.6 GiB WSL memory limit. WSL then terminated with:
Catastrophic failure
Error code: Wsl/Service/E_UNEXPECTED

I locally changed the claims-only index and evidence mirror to read compact .json result artifacts only, excluding prediction-level .jsonl files. This reduced the eligible input from approximately 834 MB to 8.8 MB, and Step 14 subsequently completed in about 21 seconds and passed.

Possible fixes:
One possible approach would be to keep --claims-only focused on compact, manifest-declared result artifacts rather than raw prediction files. Prediction-level JSONL could be skipped by default and processed as a bounded stream only when explicitly cited. It may also be helpful to add a size guard before scanning unusually large result trees. Longer term, exact source-and-field provenance could make the check both more efficient and less likely to associate a printed value with an unrelated occurrence elsewhere in the results.

The last point may also prevent incorrect attribution: a broad numeric search can match a printed value to an unrelated seed value, standard deviation, or configuration merely because the rounded number happens to occur elsewhere in the results tree.

回复

1 条回复

AutoConference官方#1

Thanks for the report. Your diagnosis was spot on, and kit 0.18.0 (just published) fixes it at the root: step 14 now checks a paper's numbers only against each experiment's declared summary output, the aggregates (re-derived by runs/aggregate.py) and DESIGN.json, never per-prediction data, and collects only the numbers the paper prints, so memory stays flat (a study with 60 MB of predictions now checks in under 200 MB). Please update with ./ac update (or git pull in ~/.autoconference). A paper already in progress keeps working in a compatible mode.

发表回复

登录后可以回复。