4.2 KiB
Seed1 projected-budget stop — 2026-11-18
Seed1 stopped at14:46:49 Asia/Shanghai under the frozen every-epoch cost gate. The sole service exited1 at14:46:52; all seed1 queue/guard/worker processes are absent. Earlier first-epoch PASS and the14:46 launch report are historical snapshots, not current running status. No restart, retry, cap change or new scientific workload occurred in this audit.
Trigger and interpretation
Fold1's measured training epoch rose from84.753674508s to104.409218447s at epoch2. Its fixed validation bound stayed11.502408962s. The reviewed projection uses the maximum completed-epoch duration with1.25 safety factor, remaining validation visits, actual elapsed/setup and terminal allowance. Its updated projection26515.733643303s exceeded the22320s per-fold cap. Total120793.704505946s exceeded118800s; it includes pending folds2/3/4 at22320s each and7200s postfit reserve. Fold0's epoch2 was78.598072116s, projection20117.970862643s, within its cap.
This is a projected resource stop, not actual budget exhaustion or a scientific performance failure. The underlying reason for the second-epoch timing increase is not established by retained samples. Reconstructing the budget arithmetic from original epoch records and wall-clock assignment/projection timestamps matches within0.1s. This audit did not recompute scientific metrics.
Terminal evidence and accounting
| Fold | Complete epochs | Partial epoch3 | Charged GPU seconds | Wrapper / worker exit |
|---|---|---|---|---|
| 0 | 2 | 135/320 parts;775 steps total | 269.536112742 | 1 /−9 |
| 1 | 2 | 1/374 parts;749 steps total | 269.535944258 | 1 /−9 |
Both wrappers record GLOBAL_STOP and coordinator SIGTERM, followed by incomplete worker/result. The reviewed guard's stop_group issues SIGKILL, consistent with worker exit−9; it is not relabelled as numerical failure. Wrapper monitor errors are empty and both underlying monitor_errors are exact integer0. Recorded loss/gradient values are finite; no worker numerical-failure receipt or completed checkpoint exists. Recorded reserved VRAM maxima1388314624/1390411776B; worker peak RSS1437351936/1457737728B. Poll gaps1.361595/0.998286s exceed the0.2s target but are preserved; no monitor error or memory-limit stop was recorded. Resource enforcement is sampled.
Actual seed1 charge539.072057000 GPU-s (0.149742h), including both setup/elapsed receipts. Prior phase3 item1 charge72421.315999387s is counted once, giving actual cumulative72960.388056387s. The193215.020505333s value is the stopped projection, not actual consumption. Worker CPU sum393.409430s; guard CPU charges including allowances560.598215s are a separate enclosing measure and must not be added to worker CPU. The stopped coordinator does not serialize its own CPU total; this reconstruction limitation is retained rather than inventing a complete CPU ledger.
Folds2/3/4 were never attempted. No models-terminal, predictions, score or postfit exists;7200s postfit reserve unused. Original BLOCKED, repaired PASS, all initial cost passes, assignments, raw histories, learning access logs, guard receipts and source pins are unchanged.
Evidence and next decision
Root /home/frank14f/MeltMind-runs/phase3-seed1-2026-11-18. budget-stop-audit.json SHA c1457c69c7b58ddc82470da000170a489f7d9acd8318d8e109282e03ce103a55; immutable local budget-stop evidence commit a556b58340410dbe3faeb2dc2143b3708a7e724b. Seed1 and original Track A source checks pass. Track A's existing coordinator1262364 and workers1262432/1262422 remain active, observed epochs147/151 during the audit; no changes were made to that queue or completed item1.
A later GPU1 occupant, pilot.py PID1369194, was observed starting at14:47:23, after seed1's stop. It is not seed1 residue and is not evidence for a cause of the earlier slowdown; it was not touched. Current GPU occupancy does not replace historical measurements.
The frozen budget-stop and complete evidence are submitted to Claude under the existing stage-review process. Any continuation, altered projection/cap, or new attempt needs that concrete ruling. None is performed or implied here. Seed1 has no completed comparison result, so future1b/c gates depending on it are not satisfied by this stopped run.