Prepare one labelled item1 postfit recovery with bounded idle wait

This commit is contained in:
MeltAstra committed 2026-11-18 14:04:43 +08:00
1 parent 4f646232f4
commit 581208098f
24 files changed
+83789

No files matched your search

+8
View File
@@ -1,3 +1,11 @@
## 最新 item1 后处理最小恢复已固定待窄审(2026-11-18)
已实读Claude0643/9347及1566:原postfit STOP保留,predict0的686文件逐字hash复用,新根phase3-item1-postfit-recovery-2026-11-18只predict1–6;GPU1only、每job20s等待后原三idle polls,全部在900s及总账内,无换卡/重试。实现8a3962b4/contract7e991ea1/proofa8e671ff,comparison332f2515不变,54pins含全部原47不变。15CPU合成PASS,首测试断言失败/源已保留;尚无review/started/新预测。旧GPU261.150194630/CPU270.833960341/训练71192.071785253各计一次。详见docs/phase3-item1-postfit-recovery-2026-11-18.md;须Assist窄delta后执行,旧PASS不作新审阅。Track A原队列不改、2/3保留。
## 最新 Track A 首两个epoch成本门PASS(2026-11-18)
原唯一1262364继续,job0/1 worker1262432/1262422已进入epoch2;完整首epoch61.848708608s/59.255394293s,逐job门均PASS。最新总投影130716.145896GPU-s/162000,含未启动2/3/4整帽和7200s postfit预留,不是全5测量。source6701e4d6/contractbe4ac1f6不变,无重启/改源码。审阅收讫+实际启动+两门证据已远端固定4f646232,报告和原始first-two-epoch-audit.json保留。队列只checkpoint后停,test/predict/score需后续原pre-score审阅。
## 最新 Track A H2 唯一训练已实际启动(2026-11-18 12:28)
Assist1794975952原件1fa923f8已逐字接收/固定b7879eec并validate PASS。唯一service meltmind-track-a-h2-20261118.service/coord1262364,root /home/frank14f/MeltMind-runs/track-a-h2-2026-11-18;job0 guard1262373/worker1262432 GPU2,job1 guard1262374/worker1262422 GPU3,首epoch真实51/311和58/304快照,三idle polls各PASS。source6701e4d6/contractbe4ac1f6不变,23pins PASS。job2/3/4 pending,162000GPU-s含7200postfit预留,各job首epoch成本门自动执行,尚未成本PASS;禁止重复启动/改运行源。item1旧worker1135393仍GPU1(56503/64000),GPU0保留其postfit。队列只freeze模型后结束,test/predict/score仍须原独立pre-score审阅,训练PASS不得替代。实际证据actual-start-audit.json。下方未启动/待审仅历史。
@@ -0,0 +1,23 @@
# Item1 postfit: one labelled infrastructure continuation
Prepared and locally frozen; narrow delta review pending, no new prediction or scoring started. Claude1794979347/1794980643 authorize reuse of predict0 and exactly predict1–6; Claude1794981566 fixes GPU1 only and the bounded20s idle wait. The original STOP is infrastructure-labelled with cause undiagnosed, not a scientific failure or a proven utilization-sampling issue.
Original root `/home/frank14f/MeltMind-runs/phase3-item1-postfit-2026-11-18` remains untouched. Its complete SHA8062b900b8267d9b5a691f943beb11d533570f8cae76e3ba94e59717665204e8 records predict0 PASS, predict1 stopped before any worker (`worker=null`), GPU261.1501946300268s and CPU270.833960341s. Training71192.071785253GPU-s remains a separate once-only amount. Original started, resource-progress, six-receipt audit and raw guards are retained.
New root `/home/frank14f/MeltMind-runs/phase3-item1-postfit-recovery-2026-11-18`. Implementation8a3962b4c1267a7f021e392098098fb7ff20f562; contract7e991ea1b5477f92d4bdaa99eb0629ca23d50c068a1b11a1827c189bb8c70622; proofa8e671ffa4b43998eee3ad302bb9f2a40f7ade83b3209274245c75f6fddb4d1f. Original comparison332f2515911c99aadf2486086ddff82bd195d889faa8391296443e7aeacdfee9 stays byte-identical. expected-review-binding.json includes54 code pins; all original47 pins remain identical. This new delta does not reuse old PASS75bd9ff1 as its own verdict.
New files are `phase3_item1_resume.py`, `phase3_item1_resume_common.py`, `phase3_item1_resume_guard.py`, `prepare_phase3_item1_resume.py`, and `test_phase3_item1_resume.py`. The original prediction worker, feature loader, score, statistics and comparison functions are imported/run unchanged. The independent coordinator uses the new root and original seven endpoint metadata, but dispatches only indices1..6. No training, model-selection or field-decoding step is added.
Reuse: all686 predict0 files (206411454bytes) have their original manifest hashes checked as opaque bytes; no arrays or model values were decoded during preparation. The old manifest has complete epoch25 and c25F coverage over343parts. Model adoption is independently matched to the old adopted-model manifest and predict0 environment checkpoint hash. At execution, source bytes are checked again, copied into the new root with identical hashes, and recorded with reuse_original_path/reused_without_prediction. Source files/manifest/STOP are not changed. The original manifest validator then sees the canonical new-root paths; predict0 is marked REUSED_BY_HASH_NO_WORKER rather than a fabricated new worker PASS.
Training prerequisite: all six original train receipts are rechecked against TRON's hashes; monitor_errors must be integer0, statusPASS/returncode0, and each embedded wrapper receipt must equal the original receipt. Original training terminal/model freezes and process-exit check are retained through terminal_inputs.
GPU selection: UUID GPU-f1c9fd5a-e917-56e0-21fb-24035ba44a36 (GPU1) only, never Track A's2/3. One root lock, one GPU lock, existing-process check and started refusal prevent a duplicate continuation. Before each prediction worker, wait at most20s for selected GPU idle, then require the original three idle polls. The wait occurs inside that guard's clock; it and all setup/poll time count toward900s per job and cumulative7200GPU-s postfit/120000GPU-s item cap. Timeout or failure stops once, no GPU switching, extra recovery or job retry. Original sampled monitor/CPU/RSS/disk/failed-peak checks remain.
Accounting: new postfit ledger = exact old postfit amount + new amount; training is added once only. Pending slots retain original900GPU-s/300CPU-s reservations. New CPU starts with180s setup allowance (covering preparation/import/synthetic overhead); child CPU and coordinator CPU use the original max-reconciliation method. The900CPU-s score and120CPU-s terminal reserve remain, inside total3600postfit CPU-s. Both the old failure and fresh usage remain auditable; no budget reset.
Barriers: freeze all adopted models; predict1–6; validate all4582 canonical entries including predict0 aliases and121original S1 test predictions; freeze combined prediction manifest/source; validate the new actual review and both barriers; then execute the unchanged one-read552-part scorer. It retains original320learning/111validation/121test targets, PPS/multiplicity/clocks/component bootstrap and original score policy. No Track A postfit or seed1 launch is attached.
Validation:15 narrow CPU synthetic checks PASS, test-body2.229610314CPU-s/2.240639487wall-s (imports excluded), plus preparation metadata/opaque-hash2.165440337CPU-s. Tests cover recovery scope, GPU1-only, preserved budgets, zero-worker stop identity, once-only accounting, six training receipts, byte-identical copy/no-overwrite, bad-hash rejection before writing,20s timeout with zero worker launches, wait+three polls charged, and monitor-error rejection. First attempt had one incorrect test expectation: seven reservations plus the selected synthetic CPU still fit3570.833960341<3600. Its failure record and source are preserved; the test fixture was corrected without changing production limits. No CUDA, scientific arrays, new predictions or score were executed by these checks.
Next: actual MeltAssist narrow delta review of resume/reuse/GPU/wait; adopt and freeze its exact receipt, then one authorized launch. Original failure evidence and training-only Track A remain untouched.