5.5 KiB
H2-S2: four new fixed folds, immutable fold0 reuse
Claude1794923791.467409 authorizes folds1–4 in parallel, same H2 recipe/seed0/200epochs, within the remaining original144000GPU-second budget. Claude1794923933.456449 accepts the stage corrections: S2 is approved but does not become running until actual worker evidence exists; S2 precedes data-sufficiency. This entry implements only the four training jobs and checkpoint freeze. It starts no prediction, pooled scorer, data-sufficiency run, refit of fold0, or retry.
Original S1 settled usage is18858.21490871499GPU-s; remaining125141.78509128501. Reserve2400GPU-s for four later prediction jobs at600each (including setup/idle/failure). Training guard/coordinator allocation is122741.78509128501GPU-s, divided into four30685.446272821252GPU-s job ceilings. Every started fold charges60setup plus all elapsed time from dispatch until guard exit, including normalization, import, first-epoch barrier, validation, checkpoints, idle checks and failures. Settled charge is the maximum of coordinator observed elapsed and the guard's actual receipt. Prior pilot/train/S1 postfit is added once; completed fold0 is never charged or executed anew. Guards retain the same one-process/16GiB treeRSS/15GiB recorded-or-deviceVRAM/20GiB output boundaries and an engineering CPU cap equal to their GPU ceiling. Four jobs jointly have64GiB RSS and80GiB output ceilings. Sampling overshoot is recorded and fails, not clipped.
GPU UUIDs are fixed to the four V100s read on Horsea. Worker CPUs0–1/2–3/4–5/6–7, fold guards8/9/10/11, coordinator12–13 avoid the original gate queue's16–27 workers and30–31 coordinator. Each fold requires three idle polls on its own GPU; busy stops without retry. Global stop propagates to all active guards and they kill/reap their own workers. The coordinator has no detached training continuation and checks actual source identities again at terminal.
Each new fold explicitly selects its original immutable fold record; the original preprocessing helper reads that selected record from its local slot0 while its embedded fold identity remains1–4. Roles are never renamed. Learning-only19-feature normalization is recomputed independently for the original learning set of that fold using the unchanged FP64 Welford population statistics. Validation is diagnostic every10epochs, exactly as S1, with no checkpoint selection. The original FP32 model/aggregations, FP64 F skip, original fold learning clock, seed0, shuffling, optimizer, clipping, learning-rate schedule, full-part loss and epoch loop remain unchanged. Only last200 is saved. Test cases cannot enter that fold's learning or validation loader.
After each fold's first complete epoch, the worker waits for a shared exact-contract PASS. The coordinator requires all four actual epoch receipts, with original part/node counts and optimizer steps. It projects each fold as charged elapsed +1.25×(remaining epochs×largest measured training-epoch time + remaining validation events×validation bound)+60tail. The initial validation bound is S1's maximum actual diagnostic time12.383006267948076s scaled by the maximum of that fold's validation node/edge/part ratios. Later actual validation times can only raise that bound. All four projected jobs must fit their fixed ceilings and old+allnew+2400prediction must fit144000. Checks refresh after each common ten-epoch milestone. A failed cost gate, job, monitor, signal, source, resource or terminal-identity check stops the queue and preserves attempts; no retry or automatic change in limits/recipe.
On four PASS terminals, last200 checkpoints, per-fold normalization and a model manifest are frozen. The queue stops at TRAIN_COMPLETE_POSTFIT_PENDING. Future prediction/scoring receives a separate entry and exact MeltAssist pre-score semantic PASS; the S1 receipt does not authorize a different scorer. New postfit CPU cap3600 and prediction reserve2400 remain recorded, not spent by this entry. No test target arrays are read by preparation, acceptance or this training queue; only original learning and validation targets for each active fold are allowed.
Confirmation is fixed before any S2 result: all552held-out parts/204components; per-arm whole-component means and equal component weights, paired intersection,2000paired component bootstrap/seed2026111408, pooled five-fold field H2−A CI upper<0. Undefined/missing complete field support cannot confirm. B13-E1b comparisons and last-location/tau are reported but do not gate. Fold0 predictions and metrics are reused by hash; no fold0 prediction or score rerun. This confirmation follows observation of fold0 and is not an untouched five-fold preregistration. It cannot by itself isolate connectivity causality. Original negative predictions, NA and limits remain visible.
Acceptance is16CPU synthetic tests: exact scientific epoch-loop AST except the first-epoch wait, fold roles/learning-only normalization, blocked test I/O, original-cost-once/four-wait/prediction reservation arithmetic, per-job/global projection failures, exact release binding, strict complete confirmation rule, process identities and retained failure peaks. Two lifecycle tests use a synthetic supervisor/GPU probe and temporary outputs to exercise the actual guard control flow without CUDA. These stand-ins test engineering transitions, not real numerical training. Attempt1's missing synthetic parts fixture and prelaunch discovery of the fold guard contract path were repaired before any real launch; the failed log and final16PASS log are retained. No scientific code from S1 was modified.