31 lines
4.2 KiB
Markdown
31 lines
4.2 KiB
Markdown
# Seed1 projected-budget stop — 2026-11-18
|
||
|
||
Seed1 stopped at14:46:49 Asia/Shanghai under the frozen every-epoch cost gate. The sole service exited1 at14:46:52; all seed1 queue/guard/worker processes are absent. Earlier first-epoch PASS and the14:46 launch report are historical snapshots, not current running status. No restart, retry, cap change or new scientific workload occurred in this audit.
|
||
|
||
## Trigger and interpretation
|
||
|
||
Fold1's measured training epoch rose from84.753674508s to104.409218447s at epoch2. Its fixed validation bound stayed11.502408962s. The reviewed projection uses the maximum completed-epoch duration with1.25 safety factor, remaining validation visits, actual elapsed/setup and terminal allowance. Its updated projection26515.733643303s exceeded the22320s per-fold cap. Total120793.704505946s exceeded118800s; it includes pending folds2/3/4 at22320s each and7200s postfit reserve. Fold0's epoch2 was78.598072116s, projection20117.970862643s, within its cap.
|
||
|
||
This is a **projected resource stop**, not actual budget exhaustion or a scientific performance failure. The underlying reason for the second-epoch timing increase is not established by retained samples. Reconstructing the budget arithmetic from original epoch records and wall-clock assignment/projection timestamps matches within0.1s. This audit did not recompute scientific metrics.
|
||
|
||
## Terminal evidence and accounting
|
||
|
||
| Fold | Complete epochs | Partial epoch3 | Charged GPU seconds | Wrapper / worker exit |
|
||
|---|---:|---|---:|---|
|
||
| 0 | 2 | 135/320 parts;775 steps total |269.536112742|1 /−9|
|
||
| 1 | 2 | 1/374 parts;749 steps total |269.535944258|1 /−9|
|
||
|
||
Both wrappers record GLOBAL_STOP and coordinator SIGTERM, followed by incomplete worker/result. The reviewed guard's stop_group issues SIGKILL, consistent with worker exit−9; it is not relabelled as numerical failure. Wrapper monitor errors are empty and both underlying monitor_errors are exact integer0. Recorded loss/gradient values are finite; no worker numerical-failure receipt or completed checkpoint exists. Recorded reserved VRAM maxima1388314624/1390411776B; worker peak RSS1437351936/1457737728B. Poll gaps1.361595/0.998286s exceed the0.2s target but are preserved; no monitor error or memory-limit stop was recorded. Resource enforcement is sampled.
|
||
|
||
Actual seed1 charge539.072057000 GPU-s (0.149742h), including both setup/elapsed receipts. Prior phase3 item1 charge72421.315999387s is counted once, giving actual cumulative72960.388056387s. The193215.020505333s value is the stopped **projection**, not actual consumption. Worker CPU sum393.409430s; guard CPU charges including allowances560.598215s are a separate enclosing measure and must not be added to worker CPU. The stopped coordinator does not serialize its own CPU total; this reconstruction limitation is retained rather than inventing a complete CPU ledger.
|
||
|
||
Folds2/3/4 were never attempted. No models-terminal, predictions, score or postfit exists;7200s postfit reserve unused. Original BLOCKED, repaired PASS, all initial cost passes, assignments, raw histories, learning access logs, guard receipts and source pins are unchanged.
|
||
|
||
## Evidence and next decision
|
||
|
||
Root `/home/frank14f/MeltMind-runs/phase3-seed1-2026-11-18`. `budget-stop-audit.json` SHA c1457c69c7b58ddc82470da000170a489f7d9acd8318d8e109282e03ce103a55; immutable local budget-stop evidence commit a556b58340410dbe3faeb2dc2143b3708a7e724b. Seed1 and original Track A source checks pass. Track A's existing coordinator1262364 and workers1262432/1262422 remain active, observed epochs147/151 during the audit; no changes were made to that queue or completed item1.
|
||
|
||
A later GPU1 occupant, pilot.py PID1369194, was observed starting at14:47:23, after seed1's stop. It is not seed1 residue and is not evidence for a cause of the earlier slowdown; it was not touched. Current GPU occupancy does not replace historical measurements.
|
||
|
||
The frozen budget-stop and complete evidence are submitted to Claude under the existing stage-review process. Any continuation, altered projection/cap, or new attempt needs that concrete ruling. None is performed or implied here. Seed1 has no completed comparison result, so future1b/c gates depending on it are not satisfied by this stopped run.
|