Preserve seed1 projected-budget stop with complete raw terminal evidence

This commit is contained in:
MeltAstra committed 2026-11-18 14:55:40 +08:00
1 parent 8108e52cd5
commit a45e375d0e
45 files changed
+11925

No files matched your search

@@ -642,3 +642,8 @@ Claude1794983369已明确撤回初次g方向误读并订正设计记录3c05a46
## Phase3后续顺序:seed1之后1b(2026-11-18)
已实读Claude1794983598.749809:接受Mold55ffc6f9作为未来item1b,seed1结果后才开展。原H2/100%池/seed0/末轮,100epoch完整日程(5warmup+95cosine到原最小LR),不是200epoch中途checkpoint;更新预算与schedule共同变化。先fold1 validation场差≤−.001且点效应绝对值大于同域seed1−seed0,再补四fold;test确认场CI上界<−.001且超过pooled种子差,末点/τ/负值另报。3.3/16.4GPUh为成本估计,仍需具体合同与首epoch投影;本会话未独立读取规格全文或实现运行。新顺序seed1→1b screen→c→Track B→b→ensemble;seed1结束前无1b计算,Track D触发仍未满足。现有seed1待审源及Track A训练不改。
## Seed1每epoch投影保护终态(2026-11-18)
原seed1两首门PASS后,fold1第二轮104.409218s触发固定最大时长×1.25投影,26515.733643>单fold22320;总120793.704506>118800,14:46:49保护停。两折真实只完成2epoch,其余三折未尝试,无seed1性能结果;实际GPU539.072057而非耗尽预算。guard/worker监控均零,源不变,全部进程退出,证据a556b583和budget-stop报告已提交Claude。耗时上升原因尚无归因证据;不称科学数值失败,不能据首门PASS自动恢复/调整帽/重试。后续依赖seed1结果的1b/c门目前未得到结果,Track A继续其原授权工作。
+6
View File
@@ -1,3 +1,9 @@
## 最新 seed1已STOPPED,禁止按首门快照续称运行(2026-11-18 14:46:49)
原1365333队列/1365362/1365363/1365444/1365428均退出,service exit1。fold1 epoch2训练104.409218s(首轮84.753675),投影26515.733643>22320;总120793.704506>118800,触发PROJECTED_SEED1_BUDGET。两折完成2epoch、epoch3中途被保护停止;2/3/4 never_attempted,无checkpoint/postfit/预测/评分。底层monitor_errors各integer0、wrapper[];无数值failure或内存越界,worker−9符合guard停止路径。实际GPU539.072057,phase3旧72421.315999一次,实际累计72960.388056,不是投影193215.020505实际消耗。coordinator停止路径CPU未序列化,保留账务限制。
budget-stop-audit SHA c1457c69,证据freeze a556b583;报告docs/phase3-seed1-budget-stop-2026-11-18.md。原ac45e2de/95d23536、BLOCKED及新PASS与首门历史均不改。已真@Claude交裁量,不恢复、不改帽、不重试。Track A原1262364仍正常(核时147/151),item1终态不动。下方“继续”皆历史快照。
## 最新 seed1首两折成本门PASS(2026-11-18 14:45)
原唯一1365333队列/1365444与1365428 worker已进入epoch2;首epoch门均PASS,fold0投影19207.933567s、fold1为21651.438886s,合计投影115019.372453/118800GPU-s,含pending2/3/4各22320整帽和7200postfit预留,不是五折全实测。phase3原72421.315999一次,总投影187440.688452/612000。first-two-epoch-audit记录原始history/门/进程;ac45e2de/95d23536源码与Track A原23pins均PASS。无重启/重试/后处理,原动态队列继续,后续若成本门停止须保留原因,不自动放宽帽。
@@ -0,0 +1,30 @@
# Seed1 projected-budget stop — 2026-11-18
Seed1 stopped at14:46:49 Asia/Shanghai under the frozen every-epoch cost gate. The sole service exited1 at14:46:52; all seed1 queue/guard/worker processes are absent. Earlier first-epoch PASS and the14:46 launch report are historical snapshots, not current running status. No restart, retry, cap change or new scientific workload occurred in this audit.
## Trigger and interpretation
Fold1's measured training epoch rose from84.753674508s to104.409218447s at epoch2. Its fixed validation bound stayed11.502408962s. The reviewed projection uses the maximum completed-epoch duration with1.25 safety factor, remaining validation visits, actual elapsed/setup and terminal allowance. Its updated projection26515.733643303s exceeded the22320s per-fold cap. Total120793.704505946s exceeded118800s; it includes pending folds2/3/4 at22320s each and7200s postfit reserve. Fold0's epoch2 was78.598072116s, projection20117.970862643s, within its cap.
This is a **projected resource stop**, not actual budget exhaustion or a scientific performance failure. The underlying reason for the second-epoch timing increase is not established by retained samples. Reconstructing the budget arithmetic from original epoch records and wall-clock assignment/projection timestamps matches within0.1s. This audit did not recompute scientific metrics.
## Terminal evidence and accounting
| Fold | Complete epochs | Partial epoch3 | Charged GPU seconds | Wrapper / worker exit |
|---|---:|---|---:|---|
| 0 | 2 | 135/320 parts;775 steps total |269.536112742|1 /−9|
| 1 | 2 | 1/374 parts;749 steps total |269.535944258|1 /−9|
Both wrappers record GLOBAL_STOP and coordinator SIGTERM, followed by incomplete worker/result. The reviewed guard's stop_group issues SIGKILL, consistent with worker exit−9; it is not relabelled as numerical failure. Wrapper monitor errors are empty and both underlying monitor_errors are exact integer0. Recorded loss/gradient values are finite; no worker numerical-failure receipt or completed checkpoint exists. Recorded reserved VRAM maxima1388314624/1390411776B; worker peak RSS1437351936/1457737728B. Poll gaps1.361595/0.998286s exceed the0.2s target but are preserved; no monitor error or memory-limit stop was recorded. Resource enforcement is sampled.
Actual seed1 charge539.072057000 GPU-s (0.149742h), including both setup/elapsed receipts. Prior phase3 item1 charge72421.315999387s is counted once, giving actual cumulative72960.388056387s. The193215.020505333s value is the stopped **projection**, not actual consumption. Worker CPU sum393.409430s; guard CPU charges including allowances560.598215s are a separate enclosing measure and must not be added to worker CPU. The stopped coordinator does not serialize its own CPU total; this reconstruction limitation is retained rather than inventing a complete CPU ledger.
Folds2/3/4 were never attempted. No models-terminal, predictions, score or postfit exists;7200s postfit reserve unused. Original BLOCKED, repaired PASS, all initial cost passes, assignments, raw histories, learning access logs, guard receipts and source pins are unchanged.
## Evidence and next decision
Root `/home/frank14f/MeltMind-runs/phase3-seed1-2026-11-18`. `budget-stop-audit.json` SHA c1457c69c7b58ddc82470da000170a489f7d9acd8318d8e109282e03ce103a55; immutable local budget-stop evidence commit a556b58340410dbe3faeb2dc2143b3708a7e724b. Seed1 and original Track A source checks pass. Track A's existing coordinator1262364 and workers1262432/1262422 remain active, observed epochs147/151 during the audit; no changes were made to that queue or completed item1.
A later GPU1 occupant, pilot.py PID1369194, was observed starting at14:47:23, after seed1's stop. It is not seed1 residue and is not evidence for a cause of the earlier slowdown; it was not touched. Current GPU occupancy does not replace historical measurements.
The frozen budget-stop and complete evidence are submitted to Claude under the existing stage-review process. Any continuation, altered projection/cap, or new attempt needs that concrete ruling. None is performed or implied here. Seed1 has no completed comparison result, so future1b/c gates depending on it are not satisfied by this stopped run.