Freeze single labelled seed1 continuation with old cost inside final ceilings
This commit is contained in:
1 parent
a45e375d0e
commit
9535201ac7
16 files changed
+6961
No files matched your search
@@ -0,0 +1,11 @@
|
||||
test_common_projection_idle_and_release_functions_unchanged (test_phase3_seed1_continuation.ContinuationTests) ... ok
|
||||
test_duplicate_detection_includes_both_generations (test_phase3_seed1_continuation.ContinuationTests) ... ok
|
||||
test_final_fold_or_total_overrun_stops (test_phase3_seed1_continuation.ContinuationTests) ... ok
|
||||
test_full_pending_caps_include_old_cost_once (test_phase3_seed1_continuation.ContinuationTests) ... ok
|
||||
test_settled_new_cost_does_not_double_charge_old (test_phase3_seed1_continuation.ContinuationTests) ... ok
|
||||
test_worker_guard_queue_only_entry_names_change (test_phase3_seed1_continuation.ContinuationTests) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 6 tests in 0.025s
|
||||
|
||||
OK
|
||||
@@ -647,3 +647,8 @@ Claude1794983369已明确撤回初次g方向误读并订正设计记录3c05a46
|
||||
## Seed1每epoch投影保护终态(2026-11-18)
|
||||
|
||||
原seed1两首门PASS后,fold1第二轮104.409218s触发固定最大时长×1.25投影,26515.733643>单fold22320;总120793.704506>118800,14:46:49保护停。两折真实只完成2epoch,其余三折未尝试,无seed1性能结果;实际GPU539.072057而非耗尽预算。guard/worker监控均零,源不变,全部进程退出,证据a556b583和budget-stop报告已提交Claude。耗时上升原因尚无归因证据;不称科学数值失败,不能据首门PASS自动恢复/调整帽/重试。后续依赖seed1结果的1b/c门目前未得到结果,Track A继续其原授权工作。
|
||||
|
||||
|
||||
## Seed1一次接续裁量与旧账(2026-11-18)
|
||||
|
||||
Claude1794984996及1794985731已实读:一次独立root接续,同seed1/200epoch/原科学配方;fold0/1新鲜重启,2/3/4首次尝试。累计每fold32000、seed1总167200含7200postfit,旧539.072057从原fold和新总帽扣除,phase3_prior72960.388056含旧一次;612000不变。投影max×1.25、真实idle、每fold首门和Track A GPU2优先保持。新75eff37b/e90bc1b5已固定及6CPU检查,需Assist本次窄审实际PASS后唯一启动;不是重复科学主审。当前未启动。另一次超帽即停止报告,无自动扩帽/重试;原STOP和a45e375d证据永久保留。后续1b仍待seed1结果。
|
||||
@@ -1,3 +1,7 @@
|
||||
## 最新 seed1单次接续固定待窄审(2026-11-18)
|
||||
|
||||
Claude1794984996授权一次labelled continuation;1794985731实读确认旧539.072057 GPU-s在167200总帽内计一次。新root `/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18`,implementation75eff37b/contracte90bc1b5/proof9188a5fd/comparisoncba51291/44pins,6CPU检查PASS、config PASS,尚无新review/started/训练。Assist窄审已经TRON请求;旧PASS不替代。新train159460.927943、gpu166660.927943含7200postfit;fold0/1新余额31730.463887/31730.464056,余三折32000;phase3_prior72960.388056已含旧seed1一次,不再重复加。fold0/1从头同配方,2/3/4首次尝试;再超帽停止不自动延长/重试。原root/STOP/a45e375d/108冻结文件保持,旧科学/投影/idle/首门/GPU2优先不改,独立入口仅名称路由及合同核验。说明docs/phase3-seed1-continuation-2026-11-18.md。Track A1262364仍在,启动前重核主机负载和idle,不根据当前外来进程解释旧慢轮。
|
||||
|
||||
## 最新 seed1已STOPPED,禁止按首门快照续称运行(2026-11-18 14:46:49)
|
||||
|
||||
原1365333队列/1365362/1365363/1365444/1365428均退出,service exit1。fold1 epoch2训练104.409218s(首轮84.753675),投影26515.733643>22320;总120793.704506>118800,触发PROJECTED_SEED1_BUDGET。两折完成2epoch、epoch3中途被保护停止;2/3/4 never_attempted,无checkpoint/postfit/预测/评分。底层monitor_errors各integer0、wrapper[];无数值failure或内存越界,worker−9符合guard停止路径。实际GPU539.072057,phase3旧72421.315999一次,实际累计72960.388056,不是投影193215.020505实际消耗。coordinator停止路径CPU未序列化,保留账务限制。
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
# Seed1 one labelled continuation
|
||||
|
||||
Claude 1794984996.938059 authorizes one fresh restart of folds 0/1 and first attempts of 2/3/4, with unchanged seed1/200 epochs, science recipe, max-epoch ×1.25 projection, per-fold first-epoch gate and actual idle dispatch. GPU2 remains reserved for Track A postfit. A further cap exceed stops and reports; no automatic extension or retry.
|
||||
|
||||
Claude 1794985731.591689 explicitly confirms the old 539.0720570000121 GPU-s is **inside** the final cumulative seed1 ceiling 167200, not an extra allowance:
|
||||
|
||||
- Cumulative per-fold cap: 32000 GPU-s. New fold0 cap is 32000−269.53611274203286; fold1 is 32000−269.5359442579793; folds2/3/4 remain32000.
|
||||
- New-run training cap: 160000−539.0720570000121 =159460.927943 GPU-s. New-run total cap: 167200−539.0720570000121 =166660.927943, including unchanged7200 postfit reserve.
|
||||
- Runtime receipts and projections describe **new-run cost**. Add old539.072057 once to report cumulative seed1. Phase3 prior already equals original item1 72421.31599938683 + old seed1 =72960.38805638684; add only new-run cost/projection to it. Never add old seed1 again. Phase3 ceiling612000 remains unchanged.
|
||||
- Original STOP has no serialized coordinator CPU total; this limitation remains. It is not filled with estimated CPU.
|
||||
|
||||
Original root `/home/frank14f/MeltMind-runs/phase3-seed1-2026-11-18`, implementation ac45e2de, contract95d23536, budget-stop freeze a556b583 and published a45e375d0e8797e4d63258e446647191ef253585 are immutable. The old STOP, partial two epochs, original PASS/BLOCKED reviews and all failures remain. Both old attempted folds start fresh, without partial-checkpoint adoption. Scientific comparison SHA remains cba51291811c224966ccab82ff64a9b0690c83eaf2482316853ec9c656322493.
|
||||
|
||||
New root is exclusively `/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18`. Isolated worker/guard/queue copies differ only by their import/child-command names, preserving old pinned source. Common functions for scientific roles, projection, dispatch, real idle checks, GPU2 release priority and assignment validation are byte-identical. Common contract validation checks the new remaining constants against original STOP receipts and includes both entry generations in duplicate detection. GPU locks retain their original shared names. No unrelated counters, monitor logic or scientific fixes are included.
|
||||
|
||||
Six narrow CPU checks passed on first attempt: unchanged worker/guard/queue except entry names; unchanged projection/idle/release helpers; duplicate detection across both generations; full pending caps plus old charge exactly167200; settled new cost and phase3 old accounting once; fixed fold/total exceed STOP. Log: `artifacts/phase3-seed1-continuation-acceptance-2026-11-18/attempt-01.log`. No datasets, model weights or CUDA workloads were read/run by this preparation.
|
||||
|
||||
Status at preparation: no new run, review or launch. Require MeltAssist's actual new PASS bound to this implementation/contract/proof/comparison/full code map, then launch once with current host-load, RAM/disk, real idle and duplicate checks. The old PASS cannot authorize this binding. Foreign GPU jobs are avoided and never stopped; a current host-load snapshot does not establish the cause of the old slow epoch. Track A remains unchanged. No seed1 prediction/scoring or later item1b scope is implied.
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"reviewed_commit": "75eff37b9a2ba656adc3617e11a56f51384ff666",
|
||||
"contract_sha256": "e90bc1b5e3781b22d2aed4b7a9b4f1f6b64ca0eebdcb1c6fa7eb42696d9752b0",
|
||||
"implementation_proof_sha256": "9188a5fd7c2b26e45018b1901429a7660f8edc46628365937ac234a9c065b8b8",
|
||||
"comparison_contract_sha256": "cba51291811c224966ccab82ff64a9b0690c83eaf2482316853ec9c656322493",
|
||||
"code_sha256": {
|
||||
"/home/frank14f/optane/MeltMind/tools/freeze_g1_components.py": "d0cee60d95243ac2c59ef3f2244ba818315718c4e3d1895f2f9f064f1f7f5ecd",
|
||||
"/home/frank14f/optane/MeltMind/tools/gre24_models.py": "53b6fee8b700b9b982b06d29908ce2ce23e0c57512c46dbf74b2aade79e3ac3a",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_gate_outcome_review.py": "915617aeb8292d03c654e637bc34a014c254fd178a773f135d1806c4b41372a8",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h1_worker_v2.py": "1543df85b615bb69b79bacd572074a7c77ae19dab110197601573a7e5805ed20",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_data.py": "7e03dd7a94e7bb43593277f183c222473d7cb7aaf2f60e53b45ab8d9499fcb09",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_model.py": "fab2470dd0654ca74c3b7d893f35ad1f3ae6c5c08ffe334ef6312cf6ec87892a",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_pilot_worker.py": "14651ca402e6d02466d8afdb77bc9e62fb2e752f0c0c3d20d98593157d5ebe81",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_common.py": "38a29a34663a5d6310571eeff53b5d18a0355e51b9fa58a08033f64fa379a9df",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_home_freeze_io.py": "bf09204b85ed2a7038acdd202257f618474d1110a9d6d772f105a3ffb334d3ff",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_local_freeze_io.py": "60adf3382577613c5234f5cf30779336b9951b66e90c075cb7cd0adab9cc42dd",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_residual_math.py": "ffbd9624dec79bdaabf95244fe87209055908f31a977e73ea4377045d6f8fe62",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_common.py": "a7b2b4c77b8646ab0b6616f1065390f91d2cc7ec386d52dd119cda2822d57c95",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_common.py": "2eec8859f97c47e3b5e6424f042384079f45303fba4411064b442c67e0c2c253",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_guard.py": "783ad2ce470a5453a238ca9c6699b30eec51cc92ef8c5ae6829304d1ff211d9c",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_queue.py": "5912e88b38c18cc9c895be51e6dfc932c7598f97ff58563b61bb482edb3893d3",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_worker.py": "5a07472a2d99c03daa6915fbc233f35f0251caac6667adc248d177bb52bf722e",
|
||||
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1_continuation.py": "3c0289a06567a90077f8bb715ecf5e1e9f325338cc8b555c0a087488ff87e758",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_gpu_idle_gate.py": "e2b1613e7e6ea398a8d555220a9365a1ad998ab4ef4b2bf8ec7cc4e31f8b4b9f",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_main_memory_guard.py": "e094f5443e11518efe9018215c11718a8e18aa462a977bf43e974b2d0b2aebb1",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py": "e9cceead51a5cc092aad23409ad98909c84a62e7e10571e750d1e9be1ad3b948",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_precision_continuation_guard.py": "01235d897c7c03e5ffa109c58b14f5d24d128e8f1ba873e1f76cfb14e02f7a7f",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_recovery_gpu_supervisor.py": "689c0f3179e4fcd362e546d90214829a9e4f8739abdbdf74b560763e5a392f15",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_recovery_monitoring.py": "442c2a9c75e8c00b30a8f2e91cb52d87c466b19a80aea9128b10c021e61b922e",
|
||||
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1_continuation.py": "740da12c8bd36d812b281735cd2bde752d2c53dd2ce41f33cef7a7ae8f8df861",
|
||||
"/home/frank14f/optane/MeltMind/tools/track_a_h2_common.py": "79918e7de887d1f0a2de5523156b27d00dc2d85ac930a07efc34d817d904fe65",
|
||||
"/home/frank14f/optane/MeltMind/tools/track_a_h2_prepare.py": "f657be036dc48fc5936f56c4b8510a4b295d4787af6877cc5ccaad663107be17",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_fixed_io.py": "ea17fdfc7508f3f980e3cdf69534b1eb8ceb8a8e469c32451b2d5e16d0ada575",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_common.py": "308486507ae3589a0e6cc40565f6e60e6d2f63ae74c4d828bdacf33b3b8b8062",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_worker.py": "37dfd75fff50b3d65f4ef37f4255536616582fe5f3d01155a0e037325ab5c7cd",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_guard.py": "c394b7b8b12ba316910ac00abeba7ba0632a1d240d8e3fd03ddbf0cbc3b66ba1",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_queue.py": "ee858a52878032b49601e56e98e224640c62f86523495d85ceba644ce0c86172",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_worker.py": "03f33f3c0d30a3b51602a46422ecc8a0392bdc9435f335a90bcc1c1084a3f381",
|
||||
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1.py": "eed47b0cfba0861700f5e1731051bae92065e9bedec727a4f15e6f5c77ce367c",
|
||||
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1.py": "f6415a4f7d4778ae79c5886f0ff1d6080b4ef32f493080e4bcb626ed7c7fd3e5",
|
||||
"/home/frank14f/optane/MeltMind/tools/test_phase2_h2_s2.py": "4235890fd194b7778e937f0be9245b964512df2c75d962ef703e3a988c5b8805",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_models/network.py": "5137e845d81d1223afc13bf87e03ccba1132a3baa75d5189a8f320b2752be071",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_guard.py": "6bc7624b0cfe850591f77153cd2034eefdd62f7e63b93211ccdb87eb8aba346c",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_worker.py": "ecee04b183ef2b93b7b39a447d48c40a5843d3190e48d2d7eb4939811a5db9b0",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_prepare.py": "964fae13ac95e4de4dc50b3c0a80791d86349ce3a9f3c5a6e7dc3cf8ca041fad",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_prepare.py": "3a31af0766f628a80f38e0b7286beb11ca3109188eabc4adfdb2ecb0eef3dc27",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_queue.py": "d82dfb365956e08250758bb4c651a19fbecce92fe209072f5f8ed35970450735",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_launch.py": "480fac25dea5a2006bb84130a4b0a0b253b3040b2150a56c38eada62b2c259e6",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_guard.py": "ae94711adc17a7630f398e8d4120fa3cc8f60979a00122c4cd2dc364c382ced5",
|
||||
"/home/frank14f/optane/MeltMind/docs/phase2-h2-s2-execution-contract-2026-11-17.md": "1e8e5ef31650519737cd6cd247abc0bd04744dd93ffb013eff88866d42f03073"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"contract_sha256": "e90bc1b5e3781b22d2aed4b7a9b4f1f6b64ca0eebdcb1c6fa7eb42696d9752b0",
|
||||
"proof_sha256": "9188a5fd7c2b26e45018b1901429a7660f8edc46628365937ac234a9c065b8b8"
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
{
|
||||
"commit": "75eff37b9a2ba656adc3617e11a56f51384ff666",
|
||||
"files": {
|
||||
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/contract.json": "e90bc1b5e3781b22d2aed4b7a9b4f1f6b64ca0eebdcb1c6fa7eb42696d9752b0",
|
||||
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/comparison-contract.json": "cba51291811c224966ccab82ff64a9b0690c83eaf2482316853ec9c656322493",
|
||||
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/preparation-audit.json": "5579b033d1d9659aa33c055d870d2b0db922d64298da95f0277c8a74c258465f",
|
||||
"/home/frank14f/optane/MeltMind/tools/freeze_g1_components.py": "d0cee60d95243ac2c59ef3f2244ba818315718c4e3d1895f2f9f064f1f7f5ecd",
|
||||
"/home/frank14f/optane/MeltMind/tools/gre24_models.py": "53b6fee8b700b9b982b06d29908ce2ce23e0c57512c46dbf74b2aade79e3ac3a",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_gate_outcome_review.py": "915617aeb8292d03c654e637bc34a014c254fd178a773f135d1806c4b41372a8",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h1_worker_v2.py": "1543df85b615bb69b79bacd572074a7c77ae19dab110197601573a7e5805ed20",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_data.py": "7e03dd7a94e7bb43593277f183c222473d7cb7aaf2f60e53b45ab8d9499fcb09",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_model.py": "fab2470dd0654ca74c3b7d893f35ad1f3ae6c5c08ffe334ef6312cf6ec87892a",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_pilot_worker.py": "14651ca402e6d02466d8afdb77bc9e62fb2e752f0c0c3d20d98593157d5ebe81",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_common.py": "38a29a34663a5d6310571eeff53b5d18a0355e51b9fa58a08033f64fa379a9df",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_home_freeze_io.py": "bf09204b85ed2a7038acdd202257f618474d1110a9d6d772f105a3ffb334d3ff",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_local_freeze_io.py": "60adf3382577613c5234f5cf30779336b9951b66e90c075cb7cd0adab9cc42dd",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_residual_math.py": "ffbd9624dec79bdaabf95244fe87209055908f31a977e73ea4377045d6f8fe62",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_common.py": "a7b2b4c77b8646ab0b6616f1065390f91d2cc7ec386d52dd119cda2822d57c95",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_common.py": "2eec8859f97c47e3b5e6424f042384079f45303fba4411064b442c67e0c2c253",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_guard.py": "783ad2ce470a5453a238ca9c6699b30eec51cc92ef8c5ae6829304d1ff211d9c",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_queue.py": "5912e88b38c18cc9c895be51e6dfc932c7598f97ff58563b61bb482edb3893d3",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_worker.py": "5a07472a2d99c03daa6915fbc233f35f0251caac6667adc248d177bb52bf722e",
|
||||
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1_continuation.py": "3c0289a06567a90077f8bb715ecf5e1e9f325338cc8b555c0a087488ff87e758",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_gpu_idle_gate.py": "e2b1613e7e6ea398a8d555220a9365a1ad998ab4ef4b2bf8ec7cc4e31f8b4b9f",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_main_memory_guard.py": "e094f5443e11518efe9018215c11718a8e18aa462a977bf43e974b2d0b2aebb1",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py": "e9cceead51a5cc092aad23409ad98909c84a62e7e10571e750d1e9be1ad3b948",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_precision_continuation_guard.py": "01235d897c7c03e5ffa109c58b14f5d24d128e8f1ba873e1f76cfb14e02f7a7f",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_recovery_gpu_supervisor.py": "689c0f3179e4fcd362e546d90214829a9e4f8739abdbdf74b560763e5a392f15",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_recovery_monitoring.py": "442c2a9c75e8c00b30a8f2e91cb52d87c466b19a80aea9128b10c021e61b922e",
|
||||
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1_continuation.py": "740da12c8bd36d812b281735cd2bde752d2c53dd2ce41f33cef7a7ae8f8df861",
|
||||
"/home/frank14f/optane/MeltMind/tools/track_a_h2_common.py": "79918e7de887d1f0a2de5523156b27d00dc2d85ac930a07efc34d817d904fe65",
|
||||
"/home/frank14f/optane/MeltMind/tools/track_a_h2_prepare.py": "f657be036dc48fc5936f56c4b8510a4b295d4787af6877cc5ccaad663107be17",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_fixed_io.py": "ea17fdfc7508f3f980e3cdf69534b1eb8ceb8a8e469c32451b2d5e16d0ada575",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_common.py": "308486507ae3589a0e6cc40565f6e60e6d2f63ae74c4d828bdacf33b3b8b8062",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_worker.py": "37dfd75fff50b3d65f4ef37f4255536616582fe5f3d01155a0e037325ab5c7cd",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_guard.py": "c394b7b8b12ba316910ac00abeba7ba0632a1d240d8e3fd03ddbf0cbc3b66ba1",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_queue.py": "ee858a52878032b49601e56e98e224640c62f86523495d85ceba644ce0c86172",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_worker.py": "03f33f3c0d30a3b51602a46422ecc8a0392bdc9435f335a90bcc1c1084a3f381",
|
||||
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1.py": "eed47b0cfba0861700f5e1731051bae92065e9bedec727a4f15e6f5c77ce367c",
|
||||
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1.py": "f6415a4f7d4778ae79c5886f0ff1d6080b4ef32f493080e4bcb626ed7c7fd3e5",
|
||||
"/home/frank14f/optane/MeltMind/tools/test_phase2_h2_s2.py": "4235890fd194b7778e937f0be9245b964512df2c75d962ef703e3a988c5b8805",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_models/network.py": "5137e845d81d1223afc13bf87e03ccba1132a3baa75d5189a8f320b2752be071",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_guard.py": "6bc7624b0cfe850591f77153cd2034eefdd62f7e63b93211ccdb87eb8aba346c",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_worker.py": "ecee04b183ef2b93b7b39a447d48c40a5843d3190e48d2d7eb4939811a5db9b0",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_prepare.py": "964fae13ac95e4de4dc50b3c0a80791d86349ce3a9f3c5a6e7dc3cf8ca041fad",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_prepare.py": "3a31af0766f628a80f38e0b7286beb11ca3109188eabc4adfdb2ecb0eef3dc27",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_queue.py": "d82dfb365956e08250758bb4c651a19fbecce92fe209072f5f8ed35970450735",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_launch.py": "480fac25dea5a2006bb84130a4b0a0b253b3040b2150a56c38eada62b2c259e6",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_guard.py": "ae94711adc17a7630f398e8d4120fa3cc8f60979a00122c4cd2dc364c382ced5",
|
||||
"/home/frank14f/optane/MeltMind/docs/phase2-h2-s2-execution-contract-2026-11-17.md": "1e8e5ef31650519737cd6cd247abc0bd04744dd93ffb013eff88866d42f03073",
|
||||
"/home/frank14f/optane/MeltMind/docs/phase3-seed1-continuation-2026-11-18.md": "3d0148f416ea825d6a5410ec9f50b28f4e6fb0abba9511e01a6493de004ffa1a",
|
||||
"/home/frank14f/optane/MeltMind/artifacts/phase3-seed1-continuation-acceptance-2026-11-18/attempt-01.log": "43adedbbacb318957ed45d8829db9ff7a41adab51c6f76c39654920f91060695"
|
||||
},
|
||||
"git_paths": {
|
||||
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/contract.json": "home-runs/phase3-seed1-continuation-2026-11-18/contract.json",
|
||||
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/comparison-contract.json": "home-runs/phase3-seed1-continuation-2026-11-18/comparison-contract.json",
|
||||
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/preparation-audit.json": "home-runs/phase3-seed1-continuation-2026-11-18/preparation-audit.json",
|
||||
"/home/frank14f/optane/MeltMind/tools/freeze_g1_components.py": "tools/freeze_g1_components.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/gre24_models.py": "tools/gre24_models.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_gate_outcome_review.py": "tools/phase2_gate_outcome_review.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h1_worker_v2.py": "tools/phase2_h1_worker_v2.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_data.py": "tools/phase2_h2_data.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_model.py": "tools/phase2_h2_model.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_pilot_worker.py": "tools/phase2_h2_pilot_worker.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_common.py": "tools/phase2_h2_s1_common.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_home_freeze_io.py": "tools/phase2_home_freeze_io.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_local_freeze_io.py": "tools/phase2_local_freeze_io.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_residual_math.py": "tools/phase2_residual_math.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_common.py": "tools/phase3_seed1_common.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_common.py": "tools/phase3_seed1_continuation_common.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_guard.py": "tools/phase3_seed1_continuation_guard.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_queue.py": "tools/phase3_seed1_continuation_queue.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_worker.py": "tools/phase3_seed1_continuation_worker.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1_continuation.py": "tools/prepare_phase3_seed1_continuation.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_gpu_idle_gate.py": "tools/s2_gpu_idle_gate.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_main_memory_guard.py": "tools/s2_main_memory_guard.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py": "tools/s2_pilot_train.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_precision_continuation_guard.py": "tools/s2_precision_continuation_guard.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_recovery_gpu_supervisor.py": "tools/s2_recovery_gpu_supervisor.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_recovery_monitoring.py": "tools/s2_recovery_monitoring.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1_continuation.py": "tools/test_phase3_seed1_continuation.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/track_a_h2_common.py": "tools/track_a_h2_common.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/track_a_h2_prepare.py": "tools/track_a_h2_prepare.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_fixed_io.py": "tools/phase2_fixed_io.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_common.py": "tools/phase2_h2_s2_common.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_worker.py": "tools/phase2_h2_s2_worker.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_guard.py": "tools/phase3_seed1_guard.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_queue.py": "tools/phase3_seed1_queue.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_worker.py": "tools/phase3_seed1_worker.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1.py": "tools/prepare_phase3_seed1.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1.py": "tools/test_phase3_seed1.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/test_phase2_h2_s2.py": "tools/test_phase2_h2_s2.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/s2_models/network.py": "tools/s2_models/network.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_guard.py": "tools/phase2_h2_s2_guard.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_worker.py": "tools/phase2_h2_s1_worker.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_prepare.py": "tools/phase2_h2_s2_prepare.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_prepare.py": "tools/phase2_h2_prepare.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_queue.py": "tools/phase2_h2_s2_queue.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_launch.py": "tools/phase2_h2_s1_launch.py",
|
||||
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_guard.py": "tools/phase2_h2_s1_guard.py",
|
||||
"/home/frank14f/optane/MeltMind/docs/phase2-h2-s2-execution-contract-2026-11-17.md": "docs/phase2-h2-s2-execution-contract-2026-11-17.md",
|
||||
"/home/frank14f/optane/MeltMind/docs/phase3-seed1-continuation-2026-11-18.md": "docs/phase3-seed1-continuation-2026-11-18.md",
|
||||
"/home/frank14f/optane/MeltMind/artifacts/phase3-seed1-continuation-acceptance-2026-11-18/attempt-01.log": "artifacts/phase3-seed1-continuation-acceptance-2026-11-18/attempt-01.log"
|
||||
},
|
||||
"remote_published": false,
|
||||
"local_repository": "/tmp/meltmind-phase2-local-fixed-20261117"
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"status": "PREPARED_PENDING_NARROW_REVIEW",
|
||||
"contract_sha256": "e90bc1b5e3781b22d2aed4b7a9b4f1f6b64ca0eebdcb1c6fa7eb42696d9752b0",
|
||||
"code_files": 44,
|
||||
"source_files": 251,
|
||||
"source_pins": "PASS",
|
||||
"old_stop_freeze": "a556b58340410dbe3faeb2dc2143b3708a7e724b",
|
||||
"old_seed1_gpu_once": 539.0720570000121,
|
||||
"new_train_gpu_cap": 159460.927943,
|
||||
"new_gpu_cap": 166660.927943,
|
||||
"cumulative_seed1_gpu_cap": 167200,
|
||||
"new_fold_caps": [
|
||||
31730.463887257967,
|
||||
31730.46405574202,
|
||||
32000,
|
||||
32000,
|
||||
32000
|
||||
],
|
||||
"phase3_prior_gpu_once": 72960.38805638684,
|
||||
"metadata_cpu_seconds": 3.318162066,
|
||||
"scientific_arrays_read": false,
|
||||
"model_weights_read": false,
|
||||
"test_targets_read": false,
|
||||
"new_fits_launched": false,
|
||||
"changes": "Remaining budget/contract accounting; isolated entry names; original science, projection, idle/first-epoch gates unchanged"
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Five original grouped folds, new seed1 only; fixed scope and bounded parallel cost gates."""
|
||||
import math
|
||||
from pathlib import Path
|
||||
from phase2_h1_worker_v2 import checked,sha
|
||||
from phase2_home_freeze_io import verify
|
||||
from track_a_h2_common import read
|
||||
|
||||
GPUS=['GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c','GPU-f1c9fd5a-e917-56e0-21fb-24035ba44a36','GPU-b8c11c01-c91f-aa0b-abf6-dad35085cc76','GPU-8b71b032-1b75-2891-935c-ce106a53034a']
|
||||
|
||||
|
||||
def execution_command(args):
|
||||
return any(Path(x).name in {'phase3_seed1_queue.py','phase3_seed1_guard.py','phase3_seed1_worker.py','phase3_seed1_continuation_queue.py','phase3_seed1_continuation_guard.py','phase3_seed1_continuation_worker.py'} for x in args)
|
||||
|
||||
|
||||
def fold_view(pilot,fold):
|
||||
if type(fold) is not int or fold not in range(5):raise PermissionError('Exactly five original grouped folds')
|
||||
f=pilot['folds'][fold]
|
||||
if f['fold']!=fold:raise ValueError('Actual original fold identity')
|
||||
roles=[set(f[k]) for k in ('learning','validation','test')]
|
||||
if any(roles[i]&roles[j] for i in range(3) for j in range(i)):raise ValueError('Role overlap')
|
||||
return dict(pilot,folds=[f])
|
||||
|
||||
|
||||
def config(root):
|
||||
c=read(root/'contract.json');verify(root,'implementation')
|
||||
if c['seed']!=1 or c['epochs']!=200 or c['new_fits']!=5 or [j['id'] for j in c['jobs']]!=list(range(5)) or not c['no_retry']:raise PermissionError('Seed1 five-fold scope')
|
||||
if c['dispatch_approval']!='1794983024.445549' or c['approval']!='1794984996.938059' or c['accounting_approval']!='1794985731.591689' or c['gpu_cap']!=167200-c['old_seed1_gpu_once'] or c['train_gpu_cap']!=160000-c['old_seed1_gpu_once'] or c['postfit_gpu_reserve']!=7200 or c['phase3_gpu_cap']!=612000:raise PermissionError('Frozen seed1 and phase3 ceilings')
|
||||
for p,h in {**c['source_sha256'],**c['code_sha256']}.items():checked(p,h)
|
||||
old=Path(c['original_seed1_root'])
|
||||
if str(old)!='/home/frank14f/MeltMind-runs/phase3-seed1-2026-11-18' or root.resolve()!=old.with_name('phase3-seed1-continuation-2026-11-18'):raise PermissionError('Sole labelled continuation root')
|
||||
if c['continuation_number']!=1 or c['fresh_restart_folds']!=[0,1] or c['previously_unattempted_folds']!=[2,3,4]:raise PermissionError('Fixed continuation scope')
|
||||
d=read(old/'complete.json');oc=read(old/'contract.json')
|
||||
if d['status']!='STOPPED' or d['never_attempted']!=[2,3,4] or [e['id'] for e in d['entries']]!=[0,1]:raise PermissionError('Immutable original stop')
|
||||
spent={e['id']:e['gpu_seconds_charged'] for e in d['entries']}
|
||||
if c['old_seed1_gpu_once']!=d['training_gpu_seconds'] or sum(spent.values())!=c['old_seed1_gpu_once']:raise ValueError('Old seed1 charged exactly once')
|
||||
if c['phase3_prior_gpu_once']!=oc['phase3_prior_gpu_once']+c['old_seed1_gpu_once']:raise ValueError('Phase3 original item plus old seed1 once')
|
||||
if c['cumulative_seed1_gpu_cap']!=167200 or c['cumulative_fold_gpu_cap']!=32000:raise ValueError('Final cumulative ceilings')
|
||||
if any(j['gpu_seconds']!=32000-spent.get(j['id'],0) or j['cpu_seconds']!=j['gpu_seconds'] for j in c['jobs']):raise ValueError('Per-fold remaining caps')
|
||||
if c['comparison_contract_sha256']!=oc['comparison_contract_sha256']:raise ValueError('Scientific comparison unchanged')
|
||||
pilot=read(c['pilot_contract'])
|
||||
import numpy as np,torch
|
||||
if np.__version__!=pilot['versions']['numpy'] or torch.__version__!=pilot['versions']['torch']:raise ValueError('Original library versions')
|
||||
if sum(j['gpu_seconds'] for j in c['jobs'])!=c['train_gpu_cap']:raise ValueError('Partitioned training budget')
|
||||
if c['phase3_prior_gpu_once']+c['gpu_cap']>c['phase3_gpu_cap']:raise ValueError('Aggregate phase3 reserve')
|
||||
tests=[]
|
||||
for j in c['jobs']:
|
||||
f=fold_view(pilot,j['id'])['folds'][0];tests.extend(f['test'])
|
||||
for k in ['learning','validation','test','learning_nodes','validation_nodes','clock_path','clock_sha256','c']:
|
||||
if j[k]!=f[k]:raise ValueError('Original roster/clock/geometry changed: '+k)
|
||||
if j['target_steps']!=200*len(f['learning']):raise ValueError('Approved GPU assignment and all epochs')
|
||||
if len(tests)!=552 or len(set(tests))!=552:raise ValueError('Original test partition')
|
||||
return c,pilot
|
||||
|
||||
|
||||
def project_cell(job,observation):
|
||||
e=observation['epoch'];train=observation['max_epoch_seconds'];charged=observation['charged']
|
||||
validation=max(job['validation_seconds_bound'],observation.get('max_validation_seconds',0))
|
||||
if type(e) is not int or e not in range(1,201) or any(not math.isfinite(x) or x<0 for x in [train,charged,validation]) or train<=0:raise ValueError('Measured epoch timing')
|
||||
future=1.25*((200-e)*train+(20-e//10)*validation)+120
|
||||
value=charged+future
|
||||
return dict(id=job['id'],projected_gpu_seconds=value,validation_seconds_bound=validation,within_cap=value<=job['gpu_seconds'])
|
||||
|
||||
|
||||
def project_total(c,estimates,terminal,active_charges=None):
|
||||
active_charges=active_charges or {};ids={j['id'] for j in c['jobs']}
|
||||
if (set(estimates)|set(terminal)|set(active_charges))-ids:raise ValueError('Unknown job')
|
||||
rows=[];total=c['postfit_gpu_reserve']
|
||||
for j in c['jobs']:
|
||||
i=j['id'];v=terminal[i]['gpu_seconds_charged'] if i in terminal else max(active_charges.get(i,0),estimates.get(i,{}).get('projected_gpu_seconds',j['gpu_seconds']))
|
||||
if not math.isfinite(v) or v<0:raise ValueError('Invalid ledger')
|
||||
total+=v;rows.append(dict(id=i,projected_gpu_seconds=v,within_cap=v<=j['gpu_seconds']))
|
||||
ok=total<=c['gpu_cap'] and c['phase3_prior_gpu_once']+total<=c['phase3_gpu_cap'] and all(r['within_cap'] for r in rows)
|
||||
return dict(status='PASS' if ok else 'STOP_BUDGET_PROJECTION',rows=rows,projected_gpu_seconds=total,phase3_prior_gpu_once=c['phase3_prior_gpu_once'],projected_phase3_gpu_seconds=c['phase3_prior_gpu_once']+total,postfit_reserve=c['postfit_gpu_reserve'])
|
||||
|
||||
|
||||
def first_epoch_release(root,jobid):
|
||||
if (root/'global-stop.json').exists():raise RuntimeError('Seed1 global stop')
|
||||
p=root/'jobs'/str(jobid)/'first-epoch-projection.json'
|
||||
if not p.exists():return False
|
||||
r=read(p)
|
||||
if r['status']!='PASS' or r['contract_sha256']!=sha(root/'contract.json'):raise RuntimeError('Cost gate rejected')
|
||||
return True
|
||||
|
||||
|
||||
def dispatchable(pending,active,available):
|
||||
occupied={j['gpu_uuid'] for j in active}
|
||||
if len(occupied)!=len(active) or len(active)>4 or not occupied<=set(GPUS) or not set(available)<=set(GPUS):raise ValueError('GPU collision or unknown GPU')
|
||||
free=[u for u in GPUS if u in available and u not in occupied]
|
||||
return [dict(j,gpu_uuid=u) for j,u in zip(pending,free)]
|
||||
|
||||
|
||||
def idle_eligible_gpus(eligible,query=None):
|
||||
if not set(eligible)<=set(GPUS):raise ValueError('Unknown eligible GPU')
|
||||
if query is None:
|
||||
from s2_precision_continuation_guard import device_usage
|
||||
query=device_usage
|
||||
idle=set();observations=[]
|
||||
for u in GPUS:
|
||||
if u not in eligible:continue
|
||||
mem,util=query(u)
|
||||
if not all(math.isfinite(x) and x>=0 for x in [mem,util]):raise ValueError('Invalid device occupancy')
|
||||
ok=mem<=128*2**20 and util==0
|
||||
observations.append(dict(gpu_uuid=u,memory_bytes=mem,utilization=util,idle=ok))
|
||||
if ok:idle.add(u)
|
||||
return idle,observations
|
||||
|
||||
|
||||
def assigned_job(root,c,index,digest):
|
||||
p=checked(root/'jobs'/str(index)/'assignment.json',digest);a=read(p)
|
||||
if a['id']!=index or a['contract_sha256']!=sha(root/'contract.json') or a['gpu_uuid'] not in GPUS:raise ValueError('Actual assignment identity')
|
||||
return dict(c['jobs'][index],gpu_uuid=a['gpu_uuid'])
|
||||
|
||||
|
||||
def available_gpus(c):
|
||||
import psutil
|
||||
available={GPUS[0]}
|
||||
item=Path(c['item1_recovery_root']);track=Path(c['track_a_training_root'])
|
||||
names={Path(a).name for p in psutil.process_iter(['cmdline']) for a in p.info['cmdline'] or []}
|
||||
if (item/'complete.json').exists() and not names & {'phase3_item1_postfit_worker.py','phase3_item1_resume.py'}:
|
||||
available.add(GPUS[1])
|
||||
if (track/'complete.json').exists() and not names & {'track_a_h2_queue.py','track_a_h2_guard.py','track_a_h2_worker.py'}:
|
||||
# GPU2 stays reserved for the already-prioritized Track A postfit until its terminal.
|
||||
post=Path(c['track_a_postfit_root'])
|
||||
available.add(GPUS[3])
|
||||
if (post/'complete.json').exists() and not names & {'track_a_postfit.py','track_a_postfit_worker.py'}:
|
||||
available.add(GPUS[2])
|
||||
return available
|
||||
@@ -0,0 +1,88 @@
|
||||
"""Independent bounded SEED1 fold guard; shared global stop and immutable S1."""
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import psutil
|
||||
from phase3_seed1_continuation_common import config,assigned_job
|
||||
from phase2_h1_worker_v2 import save,sha
|
||||
from s2_main_memory_guard import PeakRecords
|
||||
from s2_precision_continuation_guard import device_usage
|
||||
from s2_recovery_gpu_supervisor import supervise,tree_usage,stop_group,disk_usage
|
||||
|
||||
def run(queue,fold_id,assignment_sha):
|
||||
c,_=config(queue);job=assigned_job(queue,c,fold_id,assignment_sha);c=dict(c,**job);root=queue/'jobs'/str(fold_id);os.sched_setaffinity(0,c['guard_cpus'])
|
||||
with (root/'fold.lock').open('a') as lock, Path('/tmp/meltmind-seed1-'+c['gpu_uuid']+'.lock').open('a') as gpu_lock:
|
||||
fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
|
||||
fcntl.flock(gpu_lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
|
||||
if (root/'started.json').exists():raise FileExistsError('Single SEED1 fold launch, no retry')
|
||||
start=time.monotonic();cpu=time.process_time();requested=[];reasons=[];errors=[];samples=[];done=threading.Event();tracker=PeakRecords();receipt=None
|
||||
save(root/'started.json',dict(pid=os.getpid(),unix=time.time(),contract_sha256=sha(queue/'contract.json'),gpu_uuid=c['gpu_uuid'],setup_gpu_seconds=c['setup_gpu_seconds']))
|
||||
def elapsed():return c['setup_gpu_seconds']+time.monotonic()-start
|
||||
def reason(x):
|
||||
if x not in reasons:reasons.append(x)
|
||||
command=[sys.executable,'tools/phase3_seed1_continuation_worker.py',str(queue),str(fold_id),assignment_sha]
|
||||
def kill_worker():
|
||||
for p in psutil.Process().children(recursive=True):
|
||||
try:
|
||||
if p.cmdline()==command:stop_group(p.pid)
|
||||
except psutil.NoSuchProcess:pass
|
||||
old={s:signal.signal(s,lambda signum,frame:requested.append('COORDINATOR_SIGNAL_'+str(signum))) for s in [signal.SIGINT,signal.SIGTERM]}
|
||||
def scan():
|
||||
if tracker.scan(root/'worker'):reason('RECORDED_VRAM_LIMIT')
|
||||
def watch():
|
||||
while not done.is_set():
|
||||
try:
|
||||
scan();mem,util=device_usage(c['gpu_uuid']);_,rss,_=tree_usage(os.getpid())
|
||||
if mem>c['vram_bytes']:reason('DEVICE_VRAM_LIMIT')
|
||||
if rss>c['rss_bytes']:reason('WHOLE_TREE_RSS_LIMIT')
|
||||
if disk_usage(root)>c['disk_bytes']:reason('OUTPUT_DISK_LIMIT')
|
||||
if elapsed()>=c['gpu_seconds']-5:reason('SEED1_FOLD_TIME_LIMIT')
|
||||
if (queue/'global-stop.json').exists():reason('SEED1_GLOBAL_STOP')
|
||||
for x in requested:reason(x)
|
||||
save(root/'live-progress.json',dict(stage='SEED1_TRAINING',fold=fold_id,elapsed_gpu_seconds=elapsed(),device_bytes=mem,device_utilization=util,tree_rss_bytes=rss,recorded_peaks=tracker.maxima,reasons=reasons,unix=time.time()))
|
||||
except Exception as exc:reason('MONITOR_ERROR');errors.append(dict(type=type(exc).__name__,reason=str(exc)))
|
||||
if reasons:kill_worker()
|
||||
done.wait(.5)
|
||||
thread=None
|
||||
try:
|
||||
for i in range(3):
|
||||
mem,util=device_usage(c['gpu_uuid']);samples.append(dict(unix=time.time(),memory_bytes=mem,utilization=util))
|
||||
if mem>128*2**20 or util!=0:raise RuntimeError('SELECTED_GPU_BUSY')
|
||||
if i<2:time.sleep(2)
|
||||
save(root/'idle-polls.json',samples)
|
||||
if requested or (queue/'global-stop.json').exists():raise RuntimeError('STOP_BEFORE_WORKER_START')
|
||||
thread=threading.Thread(target=watch,daemon=True);thread.start()
|
||||
remaining=c['gpu_seconds']-elapsed()-10
|
||||
receipt=supervise(command,root/'guards','train',dict(total_cpu_seconds=c['cpu_seconds']-60,job_cpu_seconds=c['cpu_seconds']-60,job_wall_seconds=remaining,tree_rss_bytes=c['rss_bytes'],disk_bytes=c['disk_bytes'],max_processes=1),poll=.2,gpu_uuid=c['gpu_uuid'])
|
||||
except BaseException as exc:reason(type(exc).__name__+': '+str(exc));kill_worker()
|
||||
finally:
|
||||
done.set()
|
||||
if thread:thread.join(timeout=4)
|
||||
if thread and thread.is_alive():reason('MONITOR_DID_NOT_EXIT')
|
||||
try:scan()
|
||||
except Exception as exc:reason('FINAL_PEAK_SCAN_ERROR');errors.append(dict(reason=str(exc)))
|
||||
for s,handler in old.items():signal.signal(s,handler)
|
||||
value=None;p=root/'worker/result.json'
|
||||
try:
|
||||
if p.exists():value=json.loads(p.read_text())
|
||||
except Exception as exc:reason('RESULT_READ_ERROR');errors.append(dict(reason=str(exc)))
|
||||
if not receipt or receipt['status']!='PASS':reason('WORKER_NOT_PASS')
|
||||
if receipt and receipt.get('monitor_errors'):reason('WORKER_MONITOR_ERRORS')
|
||||
if not value or value.get('status')!='COMPLETE':reason('NO_COMPLETE_TRAIN_RESULT')
|
||||
if not all(k in tracker.witnesses for k in ['peak_allocated_bytes','peak_reserved_bytes']):reason('MISSING_CUDA_PEAK_RECORDS')
|
||||
totalcpu=60+(receipt['cpu_seconds'] if receipt else 0)+time.process_time()-cpu
|
||||
if elapsed()>c['gpu_seconds'] or totalcpu>c['cpu_seconds']:reason('FINAL_RESOURCE_LIMIT')
|
||||
try:config(queue)
|
||||
except Exception as exc:reason('FINAL_SOURCE_IDENTITY: '+str(exc))
|
||||
if value and (value.get('optimizer_steps')!=c['target_steps'] or value.get('fold')!=fold_id or value.get('completed_epochs')!=200):reason('INCOMPLETE_EPOCH_OR_FOLD_COUNT')
|
||||
save(root/'complete.json',dict(status='PASS' if not reasons else 'STOPPED_OR_FAILED',decision='TRAIN_COMPLETE_CHECKPOINT_FREEZE_PENDING' if not reasons else 'SEED1_FOLD_INCOMPLETE',reasons=reasons,monitor_errors=errors,
|
||||
worker=receipt,recorded_memory=tracker.as_dict(),gpu_seconds_charged=elapsed(),fold=fold_id,cpu_seconds_charged=totalcpu,
|
||||
worker_result_sha256=sha(p) if p.exists() else None,prediction_started=False,scoring_started=False,seed1_started=True,no_retry=True))
|
||||
return 0 if not reasons else 1
|
||||
|
||||
if __name__=='__main__':raise SystemExit(run(Path(sys.argv[1]),int(sys.argv[2]),sys.argv[3]))
|
||||
@@ -0,0 +1,212 @@
|
||||
"""Five original grouped folds on released GPUs; train and freeze only, never score."""
|
||||
import fcntl
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
import psutil
|
||||
from phase2_gate_outcome_review import validate_review
|
||||
from phase2_h1_worker_v2 import checked, save, sha
|
||||
from phase2_home_freeze_io import freeze
|
||||
from s2_recovery_gpu_supervisor import disk_usage, tree_usage
|
||||
from phase3_seed1_continuation_common import config, dispatchable, execution_command, project_cell, project_total, read, available_gpus, idle_eligible_gpus
|
||||
|
||||
|
||||
def run(root):
|
||||
c, _ = config(root)
|
||||
validate_review(root)
|
||||
lock = (root / 'queue.lock').open('a')
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
if (root / 'started.json').exists():
|
||||
raise FileExistsError('One Seed1 attempt; no retry')
|
||||
for p in psutil.process_iter(['pid', 'cmdline']):
|
||||
if p.pid != os.getpid() and execution_command(p.info['cmdline'] or []):
|
||||
raise RuntimeError('Existing Seed1 execution')
|
||||
os.sched_setaffinity(0, c['coordinator_cpus'])
|
||||
os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'expandable_segments:True'
|
||||
start = time.monotonic()
|
||||
cpu = time.process_time()
|
||||
active, terminal, estimates, last_epoch = {}, {}, {}, {}
|
||||
pending = list(c['jobs'])
|
||||
requested = []
|
||||
|
||||
def stop(reason):
|
||||
if not (root / 'global-stop.json').exists():
|
||||
save(root / 'global-stop.json', dict(reason=reason, unix=time.time(), no_retry=True))
|
||||
|
||||
def active_charges():
|
||||
return {i: c['jobs'][i]['setup_gpu_seconds'] + time.monotonic() - v['start']
|
||||
for i, v in active.items()}
|
||||
|
||||
def charged():
|
||||
return sum(v['gpu_seconds_charged'] for v in terminal.values()) + sum(active_charges().values())
|
||||
|
||||
def terminate(v):
|
||||
if v.get('stop_sent') is None:
|
||||
if v['process'].poll() is None:
|
||||
v['process'].send_signal(signal.SIGTERM)
|
||||
v['stop_sent'] = time.monotonic()
|
||||
elif time.monotonic() - v['stop_sent'] > 20:
|
||||
try:
|
||||
parent = psutil.Process(v['process'].pid)
|
||||
for p in reversed(parent.children(recursive=True)):
|
||||
try:
|
||||
p.kill()
|
||||
except psutil.NoSuchProcess:
|
||||
pass
|
||||
parent.kill()
|
||||
except psutil.NoSuchProcess:
|
||||
pass
|
||||
|
||||
def collect(i, v):
|
||||
rc = v['process'].poll()
|
||||
if rc is None:
|
||||
return False
|
||||
p = root / 'jobs' / str(i) / 'complete.json'
|
||||
r = read(p) if p.exists() else None
|
||||
observed = c['jobs'][i]['setup_gpu_seconds'] + time.monotonic() - v['start']
|
||||
ok = (rc == 0 and r is not None and r['status'] == 'PASS'
|
||||
and not r.get('monitor_errors') and r.get('worker') is not None
|
||||
and not r['worker'].get('monitor_errors'))
|
||||
terminal[i] = dict(id=i, status='PASS' if ok else 'STOPPED_OR_FAILED', returncode=rc,
|
||||
gpu_seconds_charged=max(observed, r['gpu_seconds_charged'] if r else 0),
|
||||
guard_complete_sha256=sha(p) if r else None, gpu_uuid=v['job']['gpu_uuid'], assignment_sha256=sha(p.parent/'assignment.json'))
|
||||
v['log'].close()
|
||||
del active[i]
|
||||
save(root / 'terminal-ledger.json', dict(entries=list(terminal.values()),
|
||||
settled_gpu_seconds=sum(x['gpu_seconds_charged'] for x in terminal.values())))
|
||||
if not ok:
|
||||
stop('JOB_' + str(i) + '_FAILED')
|
||||
return True
|
||||
|
||||
def dispatch():
|
||||
eligible=available_gpus(c)-{v['job']['gpu_uuid'] for v in active.values()}
|
||||
idle,observations=idle_eligible_gpus(eligible)
|
||||
observed_unix=time.time()
|
||||
save(root/'dispatch-idle-observation.json',dict(unix=observed_unix,eligible=sorted(eligible),observations=observations))
|
||||
for j in dispatchable(pending, [v['job'] for v in active.values()], idle):
|
||||
i = j['id']
|
||||
d = root / 'jobs' / str(i)
|
||||
d.mkdir(parents=True, exist_ok=False)
|
||||
log = (d / 'entry.log').open('x')
|
||||
begin = time.monotonic()
|
||||
save(d / 'assignment.json', dict(id=i, gpu_uuid=j['gpu_uuid'], contract_sha256=sha(root/'contract.json'), unix=time.time(), release_evidence=sorted(eligible), idle_observations=observations, idle_observed_unix=observed_unix))
|
||||
cmd = [sys.executable, 'tools/phase3_seed1_continuation_guard.py', str(root), str(i), sha(d/'assignment.json')]
|
||||
try:
|
||||
p = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT, start_new_session=True)
|
||||
except BaseException:
|
||||
log.close()
|
||||
raise
|
||||
active[i] = dict(process=p, log=log, start=begin, stop_sent=None, job=j)
|
||||
pending.remove(c['jobs'][i])
|
||||
save(d / 'dispatch.json', dict(pid=p.pid, command=cmd, unix=time.time(), gpu_uuid=j['gpu_uuid']))
|
||||
|
||||
handlers = {s: signal.signal(s, lambda n, f: requested.append('SIGNAL_' + str(n)))
|
||||
for s in (signal.SIGTERM, signal.SIGINT)}
|
||||
save(root / 'started.json', dict(pid=os.getpid(), unix=time.time(), new_fits=5,
|
||||
contract_sha256=sha(root / 'contract.json'), review_sha256=sha(root / 'review.json'),
|
||||
gpu_cap=c['gpu_cap'], A_refit=False, test_targets_read=False))
|
||||
try:
|
||||
while active or pending:
|
||||
for i, v in list(active.items()):
|
||||
collect(i, v)
|
||||
for reason in requested:
|
||||
stop(reason)
|
||||
amount = charged()
|
||||
_, rss, _ = tree_usage(os.getpid())
|
||||
if amount >= c['train_gpu_cap'] - 80:
|
||||
stop('CUMULATIVE_TRAIN_GPU_BOUNDARY')
|
||||
if rss > c['aggregate_rss_bytes'] or disk_usage(root) > c['aggregate_disk_bytes']:
|
||||
stop('AGGREGATE_MEMORY_DISK_LIMIT')
|
||||
if not (root / 'global-stop.json').exists():
|
||||
for i in active:
|
||||
files = sorted((root / 'jobs' / str(i) / 'worker/history').glob('*.json'))
|
||||
rows = [read(p) for p in files]
|
||||
if not rows:
|
||||
continue
|
||||
epoch = rows[-1]['epoch']
|
||||
# Recalculate every epoch: waiting, setup and observed cost stay charged.
|
||||
if epoch > last_epoch.get(i, 0):
|
||||
observation = dict(epoch=epoch, charged=active_charges()[i],
|
||||
max_epoch_seconds=max(r['train_epoch_seconds'] for r in rows),
|
||||
max_validation_seconds=max(r.get('validation_seconds',0) for r in rows))
|
||||
cell = project_cell(c['jobs'][i], observation)
|
||||
estimates[i] = cell
|
||||
projection = dict(**project_total(c, estimates, terminal, active_charges()),
|
||||
cell=cell, contract_sha256=sha(root / 'contract.json'), unix=time.time())
|
||||
name = 'first-epoch-projection.json' if i not in last_epoch else 'latest-cost-projection.json'
|
||||
save(root / 'jobs' / str(i) / name, projection)
|
||||
save(root / 'latest-cost-projection.json', projection)
|
||||
last_epoch[i] = epoch
|
||||
if projection['status'] != 'PASS':
|
||||
stop('PROJECTED_SEED1_BUDGET')
|
||||
if not (root / 'global-stop.json').exists():
|
||||
dispatch()
|
||||
if (root / 'global-stop.json').exists():
|
||||
for v in active.values():
|
||||
terminate(v)
|
||||
save(root / 'live-progress.json', dict(stage='STOPPING' if (root / 'global-stop.json').exists()
|
||||
else 'SEED1_TRAINING', unix=time.time(), active=[dict(id=i, guard_pid=v['process'].pid)
|
||||
for i, v in active.items()], terminal=list(terminal.values()), pending=[j['id'] for j in pending],
|
||||
charged_training_gpu_seconds=charged(), postfit_gpu_reserve=c['postfit_gpu_reserve']))
|
||||
if (root / 'global-stop.json').exists() and not active:
|
||||
break
|
||||
if active or pending:
|
||||
time.sleep(1)
|
||||
if (root / 'global-stop.json').exists() or len(terminal) != 5 or any(v['status'] != 'PASS' for v in terminal.values()):
|
||||
raise RuntimeError('Seed1 training incomplete')
|
||||
config(root)
|
||||
models = []
|
||||
for j in c['jobs']:
|
||||
w = root / 'jobs' / str(j['id']) / 'worker'
|
||||
r = read(w / 'result.json')
|
||||
stats = read(checked(w / 'standardization.json', r['standardization_sha256']))
|
||||
if (r['optimizer_steps'] != j['target_steps'] or r['completed_epochs'] != 200
|
||||
or stats['parts'] != j['learning'] or r['test_targets_read']):
|
||||
raise ValueError('Terminal model identity')
|
||||
checked(r['checkpoint'], r['checkpoint_sha256'])
|
||||
models.append(dict(job=j['id'], path=r['checkpoint'], sha256=r['checkpoint_sha256'],
|
||||
standardization=str(w / 'standardization.json'),
|
||||
standardization_sha256=r['standardization_sha256']))
|
||||
if charged() > c['train_gpu_cap']:
|
||||
raise ValueError('Terminal cumulative cost')
|
||||
save(root / 'models-terminal.json', dict(models=models, new_fits=5, A_refit=False, test_prediction_started=False))
|
||||
freeze(root, 'models', [root / 'contract.json', root / 'models-terminal.json', root / 'terminal-ledger.json',
|
||||
*[Path(m['path']) for m in models], *[Path(m['standardization']) for m in models]])
|
||||
save(root / 'complete.json', dict(status='TRAIN_COMPLETE_POSTFIT_REVIEW_PENDING', entries=list(terminal.values()),
|
||||
training_gpu_seconds=charged(), postfit_gpu_reserve=c['postfit_gpu_reserve'], gpu_cap=c['gpu_cap'],
|
||||
coordinator_cpu_seconds=time.process_time() - cpu, wall_seconds=time.monotonic() - start,
|
||||
phase3_prior_gpu_once=c['phase3_prior_gpu_once'], cumulative_phase3_training_gpu_seconds=c['phase3_prior_gpu_once']+charged(),
|
||||
no_retry=True, test_targets_read=False, postfit_started=False))
|
||||
freeze(root, 'terminal', [root / 'contract.json', root / 'complete.json', root / 'models-terminal.json'])
|
||||
return 0
|
||||
except BaseException as exc:
|
||||
stop(type(exc).__name__ + ': ' + str(exc))
|
||||
while active:
|
||||
for i, v in list(active.items()):
|
||||
try:
|
||||
if not collect(i, v):
|
||||
terminate(v)
|
||||
except Exception:
|
||||
terminate(v)
|
||||
if v['process'].poll() is not None:
|
||||
terminal[i] = dict(id=i, status='MALFORMED_TERMINAL',
|
||||
gpu_seconds_charged=c['jobs'][i]['setup_gpu_seconds'] + time.monotonic() - v['start'])
|
||||
v['log'].close()
|
||||
del active[i]
|
||||
if active:
|
||||
time.sleep(.5)
|
||||
save(root / 'complete.json', dict(status='STOPPED', reason=str(exc), entries=list(terminal.values()),
|
||||
never_attempted=[j['id'] for j in pending], training_gpu_seconds=charged(),
|
||||
postfit_started=False, no_retry=True))
|
||||
return 1
|
||||
finally:
|
||||
for s, handler in handlers.items():
|
||||
signal.signal(s, handler)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(run(Path(sys.argv[1])))
|
||||
@@ -0,0 +1,67 @@
|
||||
"""One new S2 fold, same frozen recipe with seed1/200epochs; no test prediction or target reads."""
|
||||
import json,os,sys,time,traceback,random
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import torch
|
||||
from phase2_h2_s1_common import selected_checkpoint
|
||||
from phase3_seed1_continuation_common import config,fold_view,first_epoch_release,assigned_job
|
||||
from phase2_h2_data import standardization
|
||||
from phase2_h2_data import load_part,to_device,arguments,cached
|
||||
from phase2_h2_model import H2Geo,optimizer,lr_epoch
|
||||
from phase2_h2_pilot_worker import peaks,train_step
|
||||
from phase2_residual_math import residual_target
|
||||
from phase2_h1_worker_v2 import save,sha
|
||||
|
||||
def run(root,fold_id,assignment_sha):
|
||||
c,pilot=config(root);old=fold_view(pilot,fold_id);job=assigned_job(root,c,fold_id,assignment_sha);out=root/'jobs'/str(fold_id)/'worker';out.mkdir(exist_ok=False);(out/'history').mkdir()
|
||||
os.sched_setaffinity(0,job['worker_cpus']);torch.set_num_threads(1);torch.set_num_interop_threads(1)
|
||||
start=time.monotonic();cpu=time.process_time();stage='SETUP';epoch=0;steps=0
|
||||
def record(item,epoch_file=False):
|
||||
item=dict(fold=fold_id,stage=stage,epoch=epoch,optimizer_steps=steps,wall_seconds=time.monotonic()-start,cpu_seconds=time.process_time()-cpu,**item,**peaks())
|
||||
with (out/'history.jsonl').open('a') as f:f.write(json.dumps(item,allow_nan=False)+'\n')
|
||||
save(out/'progress.json',item)
|
||||
if epoch_file:save(out/'history'/f'{epoch:03d}.json',item)
|
||||
return item
|
||||
try:
|
||||
if os.environ.get('CUDA_VISIBLE_DEVICES')!=job['gpu_uuid'] or os.environ.get('PYTORCH_CUDA_ALLOC_CONF')!='expandable_segments:True':raise ValueError('GPU/allocator identity')
|
||||
stage='LEARNING_ONLY_NORMALIZATION';normal=out/'normalization';normal.mkdir();stats=standardization(old,normal);save(out/'standardization.json',stats);save(out/'fold-identity.json',dict(actual_fold=fold_id,learning=old['folds'][0]['learning'],validation=old['folds'][0]['validation'],test=old['folds'][0]['test'],clock_sha256=old['folds'][0]['clock_sha256']))
|
||||
torch.backends.cuda.matmul.allow_tf32=False;torch.backends.cudnn.allow_tf32=False
|
||||
torch.cuda.set_device(0);torch.cuda.init();torch.cuda.reset_peak_memory_stats();random.seed(1);np.random.seed(1);torch.manual_seed(1);torch.cuda.manual_seed_all(1)
|
||||
model=H2Geo().cuda();opt=optimizer(model);fold=old['folds'][0];parts=len(fold['learning']);total=fold['learning_nodes']
|
||||
save(out/'environment.json',dict(torch=torch.__version__,gpu_uuid=job['gpu_uuid'],allocator_settings=str(torch.cuda.memory._snapshot().get('allocator_settings',{})),model_dtype='FP32',autocast=False,grad_scaler=False,fresh_seed=1,pilot_weights_reused=False,parameter_count=sum(p.numel() for p in model.parameters())))
|
||||
for epoch in range(1,201):
|
||||
stage='TRAIN';begin=time.monotonic();seen=0;nodes=0;loss_sum=0.
|
||||
if any(abs(g['lr']-lr_epoch(epoch))>1e-15 for g in opt.param_groups):raise ValueError('Learning rate epoch mismatch')
|
||||
order=np.random.default_rng(np.random.SeedSequence([1,epoch])).permutation(parts)
|
||||
for index in order:
|
||||
case=fold['learning'][int(index)];data=to_device(load_part(old,case,stats,'learning',out),'cuda')
|
||||
result=train_step(model,opt,data,parts,total);seen+=1;steps+=1;nodes+=data['nodes'];loss_sum+=result['loss']
|
||||
record(dict(case=case,epoch_parts=seen,epoch_nodes=nodes,lr=lr_epoch(epoch),loss=result['loss'],gradient_norm_before_clip=result['gradient_norm_before_clip']))
|
||||
del data
|
||||
if seen!=parts or nodes!=total:raise ValueError('Incomplete full-node learning epoch')
|
||||
metrics=dict(train_mean_weighted_loss=loss_sum/parts,train_epoch_seconds=time.monotonic()-begin,epoch_parts=seen,epoch_nodes=nodes,lr=lr_epoch(epoch))
|
||||
if epoch%10==0:
|
||||
stage='VALIDATION_DIAGNOSTIC';model.eval();val_loss=0.;val_nodes=0;vstart=time.monotonic()
|
||||
with torch.no_grad():
|
||||
for case in fold['validation']:
|
||||
data=to_device(load_part(old,case,stats,'validation',out),'cuda');p=model(*arguments(data),checkpoint_blocks=False)
|
||||
if not bool(torch.isfinite(p).all()):raise ValueError('Nonfinite validation output')
|
||||
with (out/'validation-target-access.jsonl').open('a') as f:f.write(json.dumps(dict(case=case,epoch=epoch,role='validation_diagnostic_only',target_sha256=old['parts'][case]['output_sha256']['target.npy']))+'\n')
|
||||
target=residual_target(cached(old,case,'target'),data['F'],fold['c'],fit_dtype='float32')
|
||||
delta=p.cpu().numpy().astype(np.float64)-target;val_loss+=float(delta@delta);val_nodes+=data['nodes'];del data,p
|
||||
torch.cuda.synchronize()
|
||||
if val_nodes!=fold['validation_nodes']:raise ValueError('Validation coverage')
|
||||
metrics.update(validation_mse=val_loss/val_nodes,validation_nodes=val_nodes,validation_seconds=time.monotonic()-vstart,validation_used_for_selection=False)
|
||||
stage='EPOCH_COMPLETE';record(metrics,epoch_file=True)
|
||||
if epoch==1:
|
||||
stage='WAIT_THIS_FOLD_FIRST_EPOCH_COST_GATE';record(dict(epoch_parts=seen,epoch_nodes=nodes))
|
||||
while not first_epoch_release(root,fold_id):time.sleep(.5)
|
||||
if epoch<200:
|
||||
for group in opt.param_groups:group['lr']=lr_epoch(epoch+1)
|
||||
stage='FINAL_CHECKPOINT';name=selected_checkpoint(epoch)
|
||||
torch.save(dict(model=model.state_dict(),optimizer=opt.state_dict(),completed_epochs=epoch,optimizer_steps=steps,seed=1,fold=fold_id,contract_sha256=sha(root/'contract.json'),role='last_epoch_only'),out/name)
|
||||
save(out/'result.json',dict(status='COMPLETE',fold=fold_id,completed_epochs=epoch,optimizer_steps=steps,learning_parts=parts,learning_nodes_per_epoch=total,checkpoint=str(out/name),checkpoint_sha256=sha(out/name),standardization_sha256=sha(out/'standardization.json'),test_targets_read=False,test_prediction_started=False,validation_used_for_selection=False,worker_wall_seconds=time.monotonic()-start,worker_cpu_seconds=time.process_time()-cpu,**peaks()))
|
||||
except BaseException as exc:
|
||||
memory=peaks() if torch.cuda.is_initialized() else dict(peak_allocated_bytes=0,peak_reserved_bytes=0)
|
||||
save(out/'failure.json',dict(status='FAILED',stage=stage,epoch=epoch,optimizer_steps=steps,type=type(exc).__name__,reason=str(exc),traceback=traceback.format_exc(),worker_wall_seconds=time.monotonic()-start,worker_cpu_seconds=time.process_time()-cpu,**memory));raise
|
||||
if __name__=='__main__':run(Path(sys.argv[1]),int(sys.argv[2]),sys.argv[3])
|
||||
@@ -0,0 +1,43 @@
|
||||
"""One labelled continuation: metadata-only, original STOP and scientific sources immutable."""
|
||||
import copy
|
||||
import shutil
|
||||
import time
|
||||
from pathlib import Path
|
||||
from phase2_h1_worker_v2 import checked,sha,save
|
||||
from phase2_home_freeze_io import verify
|
||||
from phase3_seed1_common import config as original_config,read
|
||||
from track_a_h2_prepare import local_code_closure
|
||||
B=Path(__file__).resolve().parents[1]
|
||||
OLD=Path('/home/frank14f/MeltMind-runs/phase3-seed1-2026-11-18')
|
||||
ROOT=OLD.with_name('phase3-seed1-continuation-2026-11-18')
|
||||
|
||||
def remaining_contract(original,terminal):
|
||||
c=copy.deepcopy(original)
|
||||
spent={e['id']:e['gpu_seconds_charged'] for e in terminal['entries']}
|
||||
old_charge=terminal['training_gpu_seconds']
|
||||
if terminal['status']!='STOPPED' or terminal['never_attempted']!=[2,3,4] or list(spent)!=[0,1] or sum(spent.values())!=old_charge:raise ValueError('Original stopped scope and accounting')
|
||||
c.update(version='phase3-seed1-labelled-continuation-v1',approval='1794984996.938059',accounting_approval='1794985731.591689',original_seed1_root=str(OLD),original_stop_publication='a45e375d0e8797e4d63258e446647191ef253585',continuation_number=1,fresh_restart_folds=[0,1],previously_unattempted_folds=[2,3,4],old_seed1_gpu_once=old_charge,cumulative_seed1_gpu_cap=167200,cumulative_fold_gpu_cap=32000,gpu_cap=167200-old_charge,train_gpu_cap=160000-old_charge,phase3_prior_gpu_once=original['phase3_prior_gpu_once']+old_charge,accounting='Runtime counters are new-run only. Add old_seed1_gpu_once once for cumulative seed1; phase3_prior already includes it. Never add it again to phase3. Remaining per-fold caps debit original fold charges; old STOP remains immutable.',remaining_caps_final=True)
|
||||
for j in c['jobs']:
|
||||
j['gpu_seconds']=32000-spent.get(j['id'],0);j['cpu_seconds']=j['gpu_seconds']
|
||||
return c
|
||||
|
||||
def main():
|
||||
start=time.process_time()
|
||||
if ROOT.exists():raise FileExistsError('Existing sole continuation root')
|
||||
oc,_=original_config(OLD);proof=verify(OLD,'budget-stop')
|
||||
if proof['commit']!='a556b58340410dbe3faeb2dc2143b3708a7e724b':raise ValueError('Original STOP freeze')
|
||||
c=remaining_contract(oc,read(OLD/'complete.json'))
|
||||
source=dict(oc['source_sha256']);source.update(oc['code_sha256']);source.update(proof['files'])
|
||||
for p in [OLD/'budget-stop-git-freeze.json',OLD/'budget-stop-authorization.json',OLD/'expected-review-binding.json']:
|
||||
source[str(p)]=sha(p)
|
||||
files=[B/'tools'/n for n in ['phase3_seed1_continuation_common.py','phase3_seed1_continuation_queue.py','phase3_seed1_continuation_guard.py','phase3_seed1_continuation_worker.py','prepare_phase3_seed1_continuation.py','test_phase3_seed1_continuation.py']]
|
||||
codes=local_code_closure(files)
|
||||
for p,h in oc['code_sha256'].items():codes.setdefault(p,h)
|
||||
for p,h in source.items():checked(p,h)
|
||||
ROOT.mkdir()
|
||||
shutil.copyfile(OLD/'comparison-contract.json',ROOT/'comparison-contract.json')
|
||||
c.update(source_sha256=source,code_sha256=codes,comparison_contract=str(ROOT/'comparison-contract.json'))
|
||||
save(ROOT/'contract.json',c)
|
||||
save(ROOT/'preparation-audit.json',dict(status='PREPARED_PENDING_NARROW_REVIEW',contract_sha256=sha(ROOT/'contract.json'),code_files=len(codes),source_files=len(source),source_pins='PASS',old_stop_freeze=proof['commit'],old_seed1_gpu_once=c['old_seed1_gpu_once'],new_train_gpu_cap=c['train_gpu_cap'],new_gpu_cap=c['gpu_cap'],cumulative_seed1_gpu_cap=c['cumulative_seed1_gpu_cap'],new_fold_caps=[j['gpu_seconds'] for j in c['jobs']],phase3_prior_gpu_once=c['phase3_prior_gpu_once'],metadata_cpu_seconds=time.process_time()-start,scientific_arrays_read=False,model_weights_read=False,test_targets_read=False,new_fits_launched=False,changes='Remaining budget/contract accounting; isolated entry names; original science, projection, idle/first-epoch gates unchanged'))
|
||||
print(read(ROOT/'preparation-audit.json'))
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Narrow CPU-only continuation delta checks; no datasets or CUDA execution."""
|
||||
import ast
|
||||
import copy
|
||||
import inspect
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from phase3_seed1_common import project_cell, project_total
|
||||
|
||||
TOOLS=Path(__file__).resolve().parent
|
||||
|
||||
class ContinuationTests(unittest.TestCase):
|
||||
def test_worker_guard_queue_only_entry_names_change(self):
|
||||
for suffix in ['worker','guard','queue']:
|
||||
original=(TOOLS/f'phase3_seed1_{suffix}.py').read_text()
|
||||
new=(TOOLS/f'phase3_seed1_continuation_{suffix}.py').read_text()
|
||||
restored=new.replace('phase3_seed1_continuation_common','phase3_seed1_common').replace('tools/phase3_seed1_continuation_guard.py','tools/phase3_seed1_guard.py').replace('tools/phase3_seed1_continuation_worker.py','tools/phase3_seed1_worker.py')
|
||||
self.assertEqual(original,restored)
|
||||
|
||||
def test_common_projection_idle_and_release_functions_unchanged(self):
|
||||
import phase3_seed1_common as old
|
||||
import phase3_seed1_continuation_common as new
|
||||
for name in ['fold_view','project_cell','project_total','first_epoch_release','dispatchable','idle_eligible_gpus','assigned_job','available_gpus']:
|
||||
self.assertEqual(inspect.getsource(getattr(old,name)),inspect.getsource(getattr(new,name)))
|
||||
|
||||
def test_duplicate_detection_includes_both_generations(self):
|
||||
from phase3_seed1_continuation_common import execution_command
|
||||
for prefix in ['phase3_seed1','phase3_seed1_continuation']:
|
||||
for suffix in ['queue','guard','worker']:
|
||||
self.assertTrue(execution_command(['python',f'tools/{prefix}_{suffix}.py']))
|
||||
self.assertFalse(execution_command(['python','pilot.py']))
|
||||
|
||||
def budget_fixture(self):
|
||||
from prepare_phase3_seed1_continuation import remaining_contract
|
||||
old=dict(jobs=[dict(id=i,gpu_seconds=22320,cpu_seconds=22320,validation_seconds_bound=12) for i in range(5)],phase3_prior_gpu_once=72421.31599938683,phase3_gpu_cap=612000,postfit_gpu_reserve=7200)
|
||||
terminal=dict(status='STOPPED',never_attempted=[2,3,4],entries=[dict(id=0,gpu_seconds_charged=269.53611274203286),dict(id=1,gpu_seconds_charged=269.5359442579793)],training_gpu_seconds=539.0720570000121)
|
||||
original=copy.deepcopy(old);c=remaining_contract(old,terminal);self.assertEqual(old,original)
|
||||
return c
|
||||
|
||||
def test_full_pending_caps_include_old_cost_once(self):
|
||||
c=self.budget_fixture();p=project_total(c,{}, {})
|
||||
self.assertEqual(p['status'],'PASS')
|
||||
self.assertAlmostEqual(p['projected_gpu_seconds']+c['old_seed1_gpu_once'],167200,places=8)
|
||||
self.assertAlmostEqual(p['projected_phase3_gpu_seconds'],72421.31599938683+167200,places=8)
|
||||
self.assertEqual(sum(j['gpu_seconds'] for j in c['jobs']),c['train_gpu_cap'])
|
||||
self.assertEqual(c['postfit_gpu_reserve'],7200)
|
||||
|
||||
def test_settled_new_cost_does_not_double_charge_old(self):
|
||||
c=self.budget_fixture();terminal={i:dict(gpu_seconds_charged=10000+i) for i in range(5)}
|
||||
p=project_total(c,{},terminal)
|
||||
self.assertAlmostEqual(p['projected_gpu_seconds'],57210)
|
||||
self.assertAlmostEqual(p['projected_phase3_gpu_seconds'],72421.31599938683+539.0720570000121+57210)
|
||||
self.assertEqual(p['status'],'PASS')
|
||||
|
||||
def test_final_fold_or_total_overrun_stops(self):
|
||||
c=self.budget_fixture();estimates={0:dict(projected_gpu_seconds=c['jobs'][0]['gpu_seconds']+.01)}
|
||||
self.assertEqual(project_total(c,estimates,{})['status'],'STOP_BUDGET_PROJECTION')
|
||||
observation=dict(epoch=2,charged=270,max_epoch_seconds=104.409218447,max_validation_seconds=0)
|
||||
self.assertTrue(project_cell(c['jobs'][1],observation)['within_cap'])
|
||||
observation['max_epoch_seconds']=140
|
||||
self.assertFalse(project_cell(c['jobs'][1],observation)['within_cap'])
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
Reference in new issue
Block a user