Freeze single labelled seed1 continuation with old cost inside final ceilings

This commit is contained in:
MeltAstra committed 2026-11-18 15:12:21 +08:00
1 parent a45e375d0e
commit 9535201ac7
16 files changed
+6961

No files matched your search

@@ -0,0 +1,11 @@
test_common_projection_idle_and_release_functions_unchanged (test_phase3_seed1_continuation.ContinuationTests) ... ok
test_duplicate_detection_includes_both_generations (test_phase3_seed1_continuation.ContinuationTests) ... ok
test_final_fold_or_total_overrun_stops (test_phase3_seed1_continuation.ContinuationTests) ... ok
test_full_pending_caps_include_old_cost_once (test_phase3_seed1_continuation.ContinuationTests) ... ok
test_settled_new_cost_does_not_double_charge_old (test_phase3_seed1_continuation.ContinuationTests) ... ok
test_worker_guard_queue_only_entry_names_change (test_phase3_seed1_continuation.ContinuationTests) ... ok
----------------------------------------------------------------------
Ran 6 tests in 0.025s
OK
@@ -647,3 +647,8 @@ Claude1794983369已明确撤回初次g方向误读并订正设计记录3c05a46
## Seed1每epoch投影保护终态(2026-11-18)
原seed1两首门PASS后,fold1第二轮104.409218s触发固定最大时长×1.25投影,26515.733643>单fold22320;总120793.704506>118800,14:46:49保护停。两折真实只完成2epoch,其余三折未尝试,无seed1性能结果;实际GPU539.072057而非耗尽预算。guard/worker监控均零,源不变,全部进程退出,证据a556b583和budget-stop报告已提交Claude。耗时上升原因尚无归因证据;不称科学数值失败,不能据首门PASS自动恢复/调整帽/重试。后续依赖seed1结果的1b/c门目前未得到结果,Track A继续其原授权工作。
## Seed1一次接续裁量与旧账(2026-11-18)
Claude1794984996及1794985731已实读:一次独立root接续,同seed1/200epoch/原科学配方;fold0/1新鲜重启,2/3/4首次尝试。累计每fold32000、seed1总167200含7200postfit,旧539.072057从原fold和新总帽扣除,phase3_prior72960.388056含旧一次;612000不变。投影max×1.25、真实idle、每fold首门和Track A GPU2优先保持。新75eff37b/e90bc1b5已固定及6CPU检查,需Assist本次窄审实际PASS后唯一启动;不是重复科学主审。当前未启动。另一次超帽即停止报告,无自动扩帽/重试;原STOP和a45e375d证据永久保留。后续1b仍待seed1结果。
+4
View File
@@ -1,3 +1,7 @@
## 最新 seed1单次接续固定待窄审(2026-11-18)
Claude1794984996授权一次labelled continuation;1794985731实读确认旧539.072057 GPU-s在167200总帽内计一次。新root `/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18`,implementation75eff37b/contracte90bc1b5/proof9188a5fd/comparisoncba51291/44pins,6CPU检查PASS、config PASS,尚无新review/started/训练。Assist窄审已经TRON请求;旧PASS不替代。新train159460.927943、gpu166660.927943含7200postfit;fold0/1新余额31730.463887/31730.464056,余三折32000;phase3_prior72960.388056已含旧seed1一次,不再重复加。fold0/1从头同配方,2/3/4首次尝试;再超帽停止不自动延长/重试。原root/STOP/a45e375d/108冻结文件保持,旧科学/投影/idle/首门/GPU2优先不改,独立入口仅名称路由及合同核验。说明docs/phase3-seed1-continuation-2026-11-18.md。Track A1262364仍在,启动前重核主机负载和idle,不根据当前外来进程解释旧慢轮。
## 最新 seed1已STOPPED,禁止按首门快照续称运行(2026-11-18 14:46:49)
原1365333队列/1365362/1365363/1365444/1365428均退出,service exit1。fold1 epoch2训练104.409218s(首轮84.753675),投影26515.733643>22320;总120793.704506>118800,触发PROJECTED_SEED1_BUDGET。两折完成2epoch、epoch3中途被保护停止;2/3/4 never_attempted,无checkpoint/postfit/预测/评分。底层monitor_errors各integer0、wrapper[];无数值failure或内存越界,worker−9符合guard停止路径。实际GPU539.072057,phase3旧72421.315999一次,实际累计72960.388056,不是投影193215.020505实际消耗。coordinator停止路径CPU未序列化,保留账务限制。
@@ -0,0 +1,18 @@
# Seed1 one labelled continuation
Claude 1794984996.938059 authorizes one fresh restart of folds 0/1 and first attempts of 2/3/4, with unchanged seed1/200 epochs, science recipe, max-epoch ×1.25 projection, per-fold first-epoch gate and actual idle dispatch. GPU2 remains reserved for Track A postfit. A further cap exceed stops and reports; no automatic extension or retry.
Claude 1794985731.591689 explicitly confirms the old 539.0720570000121 GPU-s is **inside** the final cumulative seed1 ceiling 167200, not an extra allowance:
- Cumulative per-fold cap: 32000 GPU-s. New fold0 cap is 32000−269.53611274203286; fold1 is 32000−269.5359442579793; folds2/3/4 remain32000.
- New-run training cap: 160000−539.0720570000121 =159460.927943 GPU-s. New-run total cap: 167200−539.0720570000121 =166660.927943, including unchanged7200 postfit reserve.
- Runtime receipts and projections describe **new-run cost**. Add old539.072057 once to report cumulative seed1. Phase3 prior already equals original item1 72421.31599938683 + old seed1 =72960.38805638684; add only new-run cost/projection to it. Never add old seed1 again. Phase3 ceiling612000 remains unchanged.
- Original STOP has no serialized coordinator CPU total; this limitation remains. It is not filled with estimated CPU.
Original root `/home/frank14f/MeltMind-runs/phase3-seed1-2026-11-18`, implementation ac45e2de, contract95d23536, budget-stop freeze a556b583 and published a45e375d0e8797e4d63258e446647191ef253585 are immutable. The old STOP, partial two epochs, original PASS/BLOCKED reviews and all failures remain. Both old attempted folds start fresh, without partial-checkpoint adoption. Scientific comparison SHA remains cba51291811c224966ccab82ff64a9b0690c83eaf2482316853ec9c656322493.
New root is exclusively `/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18`. Isolated worker/guard/queue copies differ only by their import/child-command names, preserving old pinned source. Common functions for scientific roles, projection, dispatch, real idle checks, GPU2 release priority and assignment validation are byte-identical. Common contract validation checks the new remaining constants against original STOP receipts and includes both entry generations in duplicate detection. GPU locks retain their original shared names. No unrelated counters, monitor logic or scientific fixes are included.
Six narrow CPU checks passed on first attempt: unchanged worker/guard/queue except entry names; unchanged projection/idle/release helpers; duplicate detection across both generations; full pending caps plus old charge exactly167200; settled new cost and phase3 old accounting once; fixed fold/total exceed STOP. Log: `artifacts/phase3-seed1-continuation-acceptance-2026-11-18/attempt-01.log`. No datasets, model weights or CUDA workloads were read/run by this preparation.
Status at preparation: no new run, review or launch. Require MeltAssist's actual new PASS bound to this implementation/contract/proof/comparison/full code map, then launch once with current host-load, RAM/disk, real idle and duplicate checks. The old PASS cannot authorize this binding. Foreign GPU jobs are avoided and never stopped; a current host-load snapshot does not establish the cause of the old slow epoch. Track A remains unchanged. No seed1 prediction/scoring or later item1b scope is implied.
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,52 @@
{
"reviewed_commit": "75eff37b9a2ba656adc3617e11a56f51384ff666",
"contract_sha256": "e90bc1b5e3781b22d2aed4b7a9b4f1f6b64ca0eebdcb1c6fa7eb42696d9752b0",
"implementation_proof_sha256": "9188a5fd7c2b26e45018b1901429a7660f8edc46628365937ac234a9c065b8b8",
"comparison_contract_sha256": "cba51291811c224966ccab82ff64a9b0690c83eaf2482316853ec9c656322493",
"code_sha256": {
"/home/frank14f/optane/MeltMind/tools/freeze_g1_components.py": "d0cee60d95243ac2c59ef3f2244ba818315718c4e3d1895f2f9f064f1f7f5ecd",
"/home/frank14f/optane/MeltMind/tools/gre24_models.py": "53b6fee8b700b9b982b06d29908ce2ce23e0c57512c46dbf74b2aade79e3ac3a",
"/home/frank14f/optane/MeltMind/tools/phase2_gate_outcome_review.py": "915617aeb8292d03c654e637bc34a014c254fd178a773f135d1806c4b41372a8",
"/home/frank14f/optane/MeltMind/tools/phase2_h1_worker_v2.py": "1543df85b615bb69b79bacd572074a7c77ae19dab110197601573a7e5805ed20",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_data.py": "7e03dd7a94e7bb43593277f183c222473d7cb7aaf2f60e53b45ab8d9499fcb09",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_model.py": "fab2470dd0654ca74c3b7d893f35ad1f3ae6c5c08ffe334ef6312cf6ec87892a",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_pilot_worker.py": "14651ca402e6d02466d8afdb77bc9e62fb2e752f0c0c3d20d98593157d5ebe81",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_common.py": "38a29a34663a5d6310571eeff53b5d18a0355e51b9fa58a08033f64fa379a9df",
"/home/frank14f/optane/MeltMind/tools/phase2_home_freeze_io.py": "bf09204b85ed2a7038acdd202257f618474d1110a9d6d772f105a3ffb334d3ff",
"/home/frank14f/optane/MeltMind/tools/phase2_local_freeze_io.py": "60adf3382577613c5234f5cf30779336b9951b66e90c075cb7cd0adab9cc42dd",
"/home/frank14f/optane/MeltMind/tools/phase2_residual_math.py": "ffbd9624dec79bdaabf95244fe87209055908f31a977e73ea4377045d6f8fe62",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_common.py": "a7b2b4c77b8646ab0b6616f1065390f91d2cc7ec386d52dd119cda2822d57c95",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_common.py": "2eec8859f97c47e3b5e6424f042384079f45303fba4411064b442c67e0c2c253",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_guard.py": "783ad2ce470a5453a238ca9c6699b30eec51cc92ef8c5ae6829304d1ff211d9c",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_queue.py": "5912e88b38c18cc9c895be51e6dfc932c7598f97ff58563b61bb482edb3893d3",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_worker.py": "5a07472a2d99c03daa6915fbc233f35f0251caac6667adc248d177bb52bf722e",
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1_continuation.py": "3c0289a06567a90077f8bb715ecf5e1e9f325338cc8b555c0a087488ff87e758",
"/home/frank14f/optane/MeltMind/tools/s2_gpu_idle_gate.py": "e2b1613e7e6ea398a8d555220a9365a1ad998ab4ef4b2bf8ec7cc4e31f8b4b9f",
"/home/frank14f/optane/MeltMind/tools/s2_main_memory_guard.py": "e094f5443e11518efe9018215c11718a8e18aa462a977bf43e974b2d0b2aebb1",
"/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py": "e9cceead51a5cc092aad23409ad98909c84a62e7e10571e750d1e9be1ad3b948",
"/home/frank14f/optane/MeltMind/tools/s2_precision_continuation_guard.py": "01235d897c7c03e5ffa109c58b14f5d24d128e8f1ba873e1f76cfb14e02f7a7f",
"/home/frank14f/optane/MeltMind/tools/s2_recovery_gpu_supervisor.py": "689c0f3179e4fcd362e546d90214829a9e4f8739abdbdf74b560763e5a392f15",
"/home/frank14f/optane/MeltMind/tools/s2_recovery_monitoring.py": "442c2a9c75e8c00b30a8f2e91cb52d87c466b19a80aea9128b10c021e61b922e",
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1_continuation.py": "740da12c8bd36d812b281735cd2bde752d2c53dd2ce41f33cef7a7ae8f8df861",
"/home/frank14f/optane/MeltMind/tools/track_a_h2_common.py": "79918e7de887d1f0a2de5523156b27d00dc2d85ac930a07efc34d817d904fe65",
"/home/frank14f/optane/MeltMind/tools/track_a_h2_prepare.py": "f657be036dc48fc5936f56c4b8510a4b295d4787af6877cc5ccaad663107be17",
"/home/frank14f/optane/MeltMind/tools/phase2_fixed_io.py": "ea17fdfc7508f3f980e3cdf69534b1eb8ceb8a8e469c32451b2d5e16d0ada575",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_common.py": "308486507ae3589a0e6cc40565f6e60e6d2f63ae74c4d828bdacf33b3b8b8062",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_worker.py": "37dfd75fff50b3d65f4ef37f4255536616582fe5f3d01155a0e037325ab5c7cd",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_guard.py": "c394b7b8b12ba316910ac00abeba7ba0632a1d240d8e3fd03ddbf0cbc3b66ba1",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_queue.py": "ee858a52878032b49601e56e98e224640c62f86523495d85ceba644ce0c86172",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_worker.py": "03f33f3c0d30a3b51602a46422ecc8a0392bdc9435f335a90bcc1c1084a3f381",
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1.py": "eed47b0cfba0861700f5e1731051bae92065e9bedec727a4f15e6f5c77ce367c",
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1.py": "f6415a4f7d4778ae79c5886f0ff1d6080b4ef32f493080e4bcb626ed7c7fd3e5",
"/home/frank14f/optane/MeltMind/tools/test_phase2_h2_s2.py": "4235890fd194b7778e937f0be9245b964512df2c75d962ef703e3a988c5b8805",
"/home/frank14f/optane/MeltMind/tools/s2_models/network.py": "5137e845d81d1223afc13bf87e03ccba1132a3baa75d5189a8f320b2752be071",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_guard.py": "6bc7624b0cfe850591f77153cd2034eefdd62f7e63b93211ccdb87eb8aba346c",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_worker.py": "ecee04b183ef2b93b7b39a447d48c40a5843d3190e48d2d7eb4939811a5db9b0",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_prepare.py": "964fae13ac95e4de4dc50b3c0a80791d86349ce3a9f3c5a6e7dc3cf8ca041fad",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_prepare.py": "3a31af0766f628a80f38e0b7286beb11ca3109188eabc4adfdb2ecb0eef3dc27",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_queue.py": "d82dfb365956e08250758bb4c651a19fbecce92fe209072f5f8ed35970450735",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_launch.py": "480fac25dea5a2006bb84130a4b0a0b253b3040b2150a56c38eada62b2c259e6",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_guard.py": "ae94711adc17a7630f398e8d4120fa3cc8f60979a00122c4cd2dc364c382ced5",
"/home/frank14f/optane/MeltMind/docs/phase2-h2-s2-execution-contract-2026-11-17.md": "1e8e5ef31650519737cd6cd247abc0bd04744dd93ffb013eff88866d42f03073"
}
}
@@ -0,0 +1,4 @@
{
"contract_sha256": "e90bc1b5e3781b22d2aed4b7a9b4f1f6b64ca0eebdcb1c6fa7eb42696d9752b0",
"proof_sha256": "9188a5fd7c2b26e45018b1901429a7660f8edc46628365937ac234a9c065b8b8"
}
@@ -0,0 +1,107 @@
{
"commit": "75eff37b9a2ba656adc3617e11a56f51384ff666",
"files": {
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/contract.json": "e90bc1b5e3781b22d2aed4b7a9b4f1f6b64ca0eebdcb1c6fa7eb42696d9752b0",
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/comparison-contract.json": "cba51291811c224966ccab82ff64a9b0690c83eaf2482316853ec9c656322493",
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/preparation-audit.json": "5579b033d1d9659aa33c055d870d2b0db922d64298da95f0277c8a74c258465f",
"/home/frank14f/optane/MeltMind/tools/freeze_g1_components.py": "d0cee60d95243ac2c59ef3f2244ba818315718c4e3d1895f2f9f064f1f7f5ecd",
"/home/frank14f/optane/MeltMind/tools/gre24_models.py": "53b6fee8b700b9b982b06d29908ce2ce23e0c57512c46dbf74b2aade79e3ac3a",
"/home/frank14f/optane/MeltMind/tools/phase2_gate_outcome_review.py": "915617aeb8292d03c654e637bc34a014c254fd178a773f135d1806c4b41372a8",
"/home/frank14f/optane/MeltMind/tools/phase2_h1_worker_v2.py": "1543df85b615bb69b79bacd572074a7c77ae19dab110197601573a7e5805ed20",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_data.py": "7e03dd7a94e7bb43593277f183c222473d7cb7aaf2f60e53b45ab8d9499fcb09",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_model.py": "fab2470dd0654ca74c3b7d893f35ad1f3ae6c5c08ffe334ef6312cf6ec87892a",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_pilot_worker.py": "14651ca402e6d02466d8afdb77bc9e62fb2e752f0c0c3d20d98593157d5ebe81",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_common.py": "38a29a34663a5d6310571eeff53b5d18a0355e51b9fa58a08033f64fa379a9df",
"/home/frank14f/optane/MeltMind/tools/phase2_home_freeze_io.py": "bf09204b85ed2a7038acdd202257f618474d1110a9d6d772f105a3ffb334d3ff",
"/home/frank14f/optane/MeltMind/tools/phase2_local_freeze_io.py": "60adf3382577613c5234f5cf30779336b9951b66e90c075cb7cd0adab9cc42dd",
"/home/frank14f/optane/MeltMind/tools/phase2_residual_math.py": "ffbd9624dec79bdaabf95244fe87209055908f31a977e73ea4377045d6f8fe62",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_common.py": "a7b2b4c77b8646ab0b6616f1065390f91d2cc7ec386d52dd119cda2822d57c95",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_common.py": "2eec8859f97c47e3b5e6424f042384079f45303fba4411064b442c67e0c2c253",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_guard.py": "783ad2ce470a5453a238ca9c6699b30eec51cc92ef8c5ae6829304d1ff211d9c",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_queue.py": "5912e88b38c18cc9c895be51e6dfc932c7598f97ff58563b61bb482edb3893d3",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_worker.py": "5a07472a2d99c03daa6915fbc233f35f0251caac6667adc248d177bb52bf722e",
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1_continuation.py": "3c0289a06567a90077f8bb715ecf5e1e9f325338cc8b555c0a087488ff87e758",
"/home/frank14f/optane/MeltMind/tools/s2_gpu_idle_gate.py": "e2b1613e7e6ea398a8d555220a9365a1ad998ab4ef4b2bf8ec7cc4e31f8b4b9f",
"/home/frank14f/optane/MeltMind/tools/s2_main_memory_guard.py": "e094f5443e11518efe9018215c11718a8e18aa462a977bf43e974b2d0b2aebb1",
"/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py": "e9cceead51a5cc092aad23409ad98909c84a62e7e10571e750d1e9be1ad3b948",
"/home/frank14f/optane/MeltMind/tools/s2_precision_continuation_guard.py": "01235d897c7c03e5ffa109c58b14f5d24d128e8f1ba873e1f76cfb14e02f7a7f",
"/home/frank14f/optane/MeltMind/tools/s2_recovery_gpu_supervisor.py": "689c0f3179e4fcd362e546d90214829a9e4f8739abdbdf74b560763e5a392f15",
"/home/frank14f/optane/MeltMind/tools/s2_recovery_monitoring.py": "442c2a9c75e8c00b30a8f2e91cb52d87c466b19a80aea9128b10c021e61b922e",
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1_continuation.py": "740da12c8bd36d812b281735cd2bde752d2c53dd2ce41f33cef7a7ae8f8df861",
"/home/frank14f/optane/MeltMind/tools/track_a_h2_common.py": "79918e7de887d1f0a2de5523156b27d00dc2d85ac930a07efc34d817d904fe65",
"/home/frank14f/optane/MeltMind/tools/track_a_h2_prepare.py": "f657be036dc48fc5936f56c4b8510a4b295d4787af6877cc5ccaad663107be17",
"/home/frank14f/optane/MeltMind/tools/phase2_fixed_io.py": "ea17fdfc7508f3f980e3cdf69534b1eb8ceb8a8e469c32451b2d5e16d0ada575",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_common.py": "308486507ae3589a0e6cc40565f6e60e6d2f63ae74c4d828bdacf33b3b8b8062",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_worker.py": "37dfd75fff50b3d65f4ef37f4255536616582fe5f3d01155a0e037325ab5c7cd",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_guard.py": "c394b7b8b12ba316910ac00abeba7ba0632a1d240d8e3fd03ddbf0cbc3b66ba1",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_queue.py": "ee858a52878032b49601e56e98e224640c62f86523495d85ceba644ce0c86172",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_worker.py": "03f33f3c0d30a3b51602a46422ecc8a0392bdc9435f335a90bcc1c1084a3f381",
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1.py": "eed47b0cfba0861700f5e1731051bae92065e9bedec727a4f15e6f5c77ce367c",
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1.py": "f6415a4f7d4778ae79c5886f0ff1d6080b4ef32f493080e4bcb626ed7c7fd3e5",
"/home/frank14f/optane/MeltMind/tools/test_phase2_h2_s2.py": "4235890fd194b7778e937f0be9245b964512df2c75d962ef703e3a988c5b8805",
"/home/frank14f/optane/MeltMind/tools/s2_models/network.py": "5137e845d81d1223afc13bf87e03ccba1132a3baa75d5189a8f320b2752be071",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_guard.py": "6bc7624b0cfe850591f77153cd2034eefdd62f7e63b93211ccdb87eb8aba346c",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_worker.py": "ecee04b183ef2b93b7b39a447d48c40a5843d3190e48d2d7eb4939811a5db9b0",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_prepare.py": "964fae13ac95e4de4dc50b3c0a80791d86349ce3a9f3c5a6e7dc3cf8ca041fad",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_prepare.py": "3a31af0766f628a80f38e0b7286beb11ca3109188eabc4adfdb2ecb0eef3dc27",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_queue.py": "d82dfb365956e08250758bb4c651a19fbecce92fe209072f5f8ed35970450735",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_launch.py": "480fac25dea5a2006bb84130a4b0a0b253b3040b2150a56c38eada62b2c259e6",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_guard.py": "ae94711adc17a7630f398e8d4120fa3cc8f60979a00122c4cd2dc364c382ced5",
"/home/frank14f/optane/MeltMind/docs/phase2-h2-s2-execution-contract-2026-11-17.md": "1e8e5ef31650519737cd6cd247abc0bd04744dd93ffb013eff88866d42f03073",
"/home/frank14f/optane/MeltMind/docs/phase3-seed1-continuation-2026-11-18.md": "3d0148f416ea825d6a5410ec9f50b28f4e6fb0abba9511e01a6493de004ffa1a",
"/home/frank14f/optane/MeltMind/artifacts/phase3-seed1-continuation-acceptance-2026-11-18/attempt-01.log": "43adedbbacb318957ed45d8829db9ff7a41adab51c6f76c39654920f91060695"
},
"git_paths": {
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/contract.json": "home-runs/phase3-seed1-continuation-2026-11-18/contract.json",
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/comparison-contract.json": "home-runs/phase3-seed1-continuation-2026-11-18/comparison-contract.json",
"/home/frank14f/MeltMind-runs/phase3-seed1-continuation-2026-11-18/preparation-audit.json": "home-runs/phase3-seed1-continuation-2026-11-18/preparation-audit.json",
"/home/frank14f/optane/MeltMind/tools/freeze_g1_components.py": "tools/freeze_g1_components.py",
"/home/frank14f/optane/MeltMind/tools/gre24_models.py": "tools/gre24_models.py",
"/home/frank14f/optane/MeltMind/tools/phase2_gate_outcome_review.py": "tools/phase2_gate_outcome_review.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h1_worker_v2.py": "tools/phase2_h1_worker_v2.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_data.py": "tools/phase2_h2_data.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_model.py": "tools/phase2_h2_model.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_pilot_worker.py": "tools/phase2_h2_pilot_worker.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_common.py": "tools/phase2_h2_s1_common.py",
"/home/frank14f/optane/MeltMind/tools/phase2_home_freeze_io.py": "tools/phase2_home_freeze_io.py",
"/home/frank14f/optane/MeltMind/tools/phase2_local_freeze_io.py": "tools/phase2_local_freeze_io.py",
"/home/frank14f/optane/MeltMind/tools/phase2_residual_math.py": "tools/phase2_residual_math.py",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_common.py": "tools/phase3_seed1_common.py",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_common.py": "tools/phase3_seed1_continuation_common.py",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_guard.py": "tools/phase3_seed1_continuation_guard.py",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_queue.py": "tools/phase3_seed1_continuation_queue.py",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_continuation_worker.py": "tools/phase3_seed1_continuation_worker.py",
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1_continuation.py": "tools/prepare_phase3_seed1_continuation.py",
"/home/frank14f/optane/MeltMind/tools/s2_gpu_idle_gate.py": "tools/s2_gpu_idle_gate.py",
"/home/frank14f/optane/MeltMind/tools/s2_main_memory_guard.py": "tools/s2_main_memory_guard.py",
"/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py": "tools/s2_pilot_train.py",
"/home/frank14f/optane/MeltMind/tools/s2_precision_continuation_guard.py": "tools/s2_precision_continuation_guard.py",
"/home/frank14f/optane/MeltMind/tools/s2_recovery_gpu_supervisor.py": "tools/s2_recovery_gpu_supervisor.py",
"/home/frank14f/optane/MeltMind/tools/s2_recovery_monitoring.py": "tools/s2_recovery_monitoring.py",
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1_continuation.py": "tools/test_phase3_seed1_continuation.py",
"/home/frank14f/optane/MeltMind/tools/track_a_h2_common.py": "tools/track_a_h2_common.py",
"/home/frank14f/optane/MeltMind/tools/track_a_h2_prepare.py": "tools/track_a_h2_prepare.py",
"/home/frank14f/optane/MeltMind/tools/phase2_fixed_io.py": "tools/phase2_fixed_io.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_common.py": "tools/phase2_h2_s2_common.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_worker.py": "tools/phase2_h2_s2_worker.py",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_guard.py": "tools/phase3_seed1_guard.py",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_queue.py": "tools/phase3_seed1_queue.py",
"/home/frank14f/optane/MeltMind/tools/phase3_seed1_worker.py": "tools/phase3_seed1_worker.py",
"/home/frank14f/optane/MeltMind/tools/prepare_phase3_seed1.py": "tools/prepare_phase3_seed1.py",
"/home/frank14f/optane/MeltMind/tools/test_phase3_seed1.py": "tools/test_phase3_seed1.py",
"/home/frank14f/optane/MeltMind/tools/test_phase2_h2_s2.py": "tools/test_phase2_h2_s2.py",
"/home/frank14f/optane/MeltMind/tools/s2_models/network.py": "tools/s2_models/network.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_guard.py": "tools/phase2_h2_s2_guard.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_worker.py": "tools/phase2_h2_s1_worker.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_prepare.py": "tools/phase2_h2_s2_prepare.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_prepare.py": "tools/phase2_h2_prepare.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s2_queue.py": "tools/phase2_h2_s2_queue.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_launch.py": "tools/phase2_h2_s1_launch.py",
"/home/frank14f/optane/MeltMind/tools/phase2_h2_s1_guard.py": "tools/phase2_h2_s1_guard.py",
"/home/frank14f/optane/MeltMind/docs/phase2-h2-s2-execution-contract-2026-11-17.md": "docs/phase2-h2-s2-execution-contract-2026-11-17.md",
"/home/frank14f/optane/MeltMind/docs/phase3-seed1-continuation-2026-11-18.md": "docs/phase3-seed1-continuation-2026-11-18.md",
"/home/frank14f/optane/MeltMind/artifacts/phase3-seed1-continuation-acceptance-2026-11-18/attempt-01.log": "artifacts/phase3-seed1-continuation-acceptance-2026-11-18/attempt-01.log"
},
"remote_published": false,
"local_repository": "/tmp/meltmind-phase2-local-fixed-20261117"
}
@@ -0,0 +1,26 @@
{
"status": "PREPARED_PENDING_NARROW_REVIEW",
"contract_sha256": "e90bc1b5e3781b22d2aed4b7a9b4f1f6b64ca0eebdcb1c6fa7eb42696d9752b0",
"code_files": 44,
"source_files": 251,
"source_pins": "PASS",
"old_stop_freeze": "a556b58340410dbe3faeb2dc2143b3708a7e724b",
"old_seed1_gpu_once": 539.0720570000121,
"new_train_gpu_cap": 159460.927943,
"new_gpu_cap": 166660.927943,
"cumulative_seed1_gpu_cap": 167200,
"new_fold_caps": [
31730.463887257967,
31730.46405574202,
32000,
32000,
32000
],
"phase3_prior_gpu_once": 72960.38805638684,
"metadata_cpu_seconds": 3.318162066,
"scientific_arrays_read": false,
"model_weights_read": false,
"test_targets_read": false,
"new_fits_launched": false,
"changes": "Remaining budget/contract accounting; isolated entry names; original science, projection, idle/first-epoch gates unchanged"
}
+127
View File
@@ -0,0 +1,127 @@
"""Five original grouped folds, new seed1 only; fixed scope and bounded parallel cost gates."""
import math
from pathlib import Path
from phase2_h1_worker_v2 import checked,sha
from phase2_home_freeze_io import verify
from track_a_h2_common import read
GPUS=['GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c','GPU-f1c9fd5a-e917-56e0-21fb-24035ba44a36','GPU-b8c11c01-c91f-aa0b-abf6-dad35085cc76','GPU-8b71b032-1b75-2891-935c-ce106a53034a']
def execution_command(args):
return any(Path(x).name in {'phase3_seed1_queue.py','phase3_seed1_guard.py','phase3_seed1_worker.py','phase3_seed1_continuation_queue.py','phase3_seed1_continuation_guard.py','phase3_seed1_continuation_worker.py'} for x in args)
def fold_view(pilot,fold):
if type(fold) is not int or fold not in range(5):raise PermissionError('Exactly five original grouped folds')
f=pilot['folds'][fold]
if f['fold']!=fold:raise ValueError('Actual original fold identity')
roles=[set(f[k]) for k in ('learning','validation','test')]
if any(roles[i]&roles[j] for i in range(3) for j in range(i)):raise ValueError('Role overlap')
return dict(pilot,folds=[f])
def config(root):
c=read(root/'contract.json');verify(root,'implementation')
if c['seed']!=1 or c['epochs']!=200 or c['new_fits']!=5 or [j['id'] for j in c['jobs']]!=list(range(5)) or not c['no_retry']:raise PermissionError('Seed1 five-fold scope')
if c['dispatch_approval']!='1794983024.445549' or c['approval']!='1794984996.938059' or c['accounting_approval']!='1794985731.591689' or c['gpu_cap']!=167200-c['old_seed1_gpu_once'] or c['train_gpu_cap']!=160000-c['old_seed1_gpu_once'] or c['postfit_gpu_reserve']!=7200 or c['phase3_gpu_cap']!=612000:raise PermissionError('Frozen seed1 and phase3 ceilings')
for p,h in {**c['source_sha256'],**c['code_sha256']}.items():checked(p,h)
old=Path(c['original_seed1_root'])
if str(old)!='/home/frank14f/MeltMind-runs/phase3-seed1-2026-11-18' or root.resolve()!=old.with_name('phase3-seed1-continuation-2026-11-18'):raise PermissionError('Sole labelled continuation root')
if c['continuation_number']!=1 or c['fresh_restart_folds']!=[0,1] or c['previously_unattempted_folds']!=[2,3,4]:raise PermissionError('Fixed continuation scope')
d=read(old/'complete.json');oc=read(old/'contract.json')
if d['status']!='STOPPED' or d['never_attempted']!=[2,3,4] or [e['id'] for e in d['entries']]!=[0,1]:raise PermissionError('Immutable original stop')
spent={e['id']:e['gpu_seconds_charged'] for e in d['entries']}
if c['old_seed1_gpu_once']!=d['training_gpu_seconds'] or sum(spent.values())!=c['old_seed1_gpu_once']:raise ValueError('Old seed1 charged exactly once')
if c['phase3_prior_gpu_once']!=oc['phase3_prior_gpu_once']+c['old_seed1_gpu_once']:raise ValueError('Phase3 original item plus old seed1 once')
if c['cumulative_seed1_gpu_cap']!=167200 or c['cumulative_fold_gpu_cap']!=32000:raise ValueError('Final cumulative ceilings')
if any(j['gpu_seconds']!=32000-spent.get(j['id'],0) or j['cpu_seconds']!=j['gpu_seconds'] for j in c['jobs']):raise ValueError('Per-fold remaining caps')
if c['comparison_contract_sha256']!=oc['comparison_contract_sha256']:raise ValueError('Scientific comparison unchanged')
pilot=read(c['pilot_contract'])
import numpy as np,torch
if np.__version__!=pilot['versions']['numpy'] or torch.__version__!=pilot['versions']['torch']:raise ValueError('Original library versions')
if sum(j['gpu_seconds'] for j in c['jobs'])!=c['train_gpu_cap']:raise ValueError('Partitioned training budget')
if c['phase3_prior_gpu_once']+c['gpu_cap']>c['phase3_gpu_cap']:raise ValueError('Aggregate phase3 reserve')
tests=[]
for j in c['jobs']:
f=fold_view(pilot,j['id'])['folds'][0];tests.extend(f['test'])
for k in ['learning','validation','test','learning_nodes','validation_nodes','clock_path','clock_sha256','c']:
if j[k]!=f[k]:raise ValueError('Original roster/clock/geometry changed: '+k)
if j['target_steps']!=200*len(f['learning']):raise ValueError('Approved GPU assignment and all epochs')
if len(tests)!=552 or len(set(tests))!=552:raise ValueError('Original test partition')
return c,pilot
def project_cell(job,observation):
e=observation['epoch'];train=observation['max_epoch_seconds'];charged=observation['charged']
validation=max(job['validation_seconds_bound'],observation.get('max_validation_seconds',0))
if type(e) is not int or e not in range(1,201) or any(not math.isfinite(x) or x<0 for x in [train,charged,validation]) or train<=0:raise ValueError('Measured epoch timing')
future=1.25*((200-e)*train+(20-e//10)*validation)+120
value=charged+future
return dict(id=job['id'],projected_gpu_seconds=value,validation_seconds_bound=validation,within_cap=value<=job['gpu_seconds'])
def project_total(c,estimates,terminal,active_charges=None):
active_charges=active_charges or {};ids={j['id'] for j in c['jobs']}
if (set(estimates)|set(terminal)|set(active_charges))-ids:raise ValueError('Unknown job')
rows=[];total=c['postfit_gpu_reserve']
for j in c['jobs']:
i=j['id'];v=terminal[i]['gpu_seconds_charged'] if i in terminal else max(active_charges.get(i,0),estimates.get(i,{}).get('projected_gpu_seconds',j['gpu_seconds']))
if not math.isfinite(v) or v<0:raise ValueError('Invalid ledger')
total+=v;rows.append(dict(id=i,projected_gpu_seconds=v,within_cap=v<=j['gpu_seconds']))
ok=total<=c['gpu_cap'] and c['phase3_prior_gpu_once']+total<=c['phase3_gpu_cap'] and all(r['within_cap'] for r in rows)
return dict(status='PASS' if ok else 'STOP_BUDGET_PROJECTION',rows=rows,projected_gpu_seconds=total,phase3_prior_gpu_once=c['phase3_prior_gpu_once'],projected_phase3_gpu_seconds=c['phase3_prior_gpu_once']+total,postfit_reserve=c['postfit_gpu_reserve'])
def first_epoch_release(root,jobid):
if (root/'global-stop.json').exists():raise RuntimeError('Seed1 global stop')
p=root/'jobs'/str(jobid)/'first-epoch-projection.json'
if not p.exists():return False
r=read(p)
if r['status']!='PASS' or r['contract_sha256']!=sha(root/'contract.json'):raise RuntimeError('Cost gate rejected')
return True
def dispatchable(pending,active,available):
occupied={j['gpu_uuid'] for j in active}
if len(occupied)!=len(active) or len(active)>4 or not occupied<=set(GPUS) or not set(available)<=set(GPUS):raise ValueError('GPU collision or unknown GPU')
free=[u for u in GPUS if u in available and u not in occupied]
return [dict(j,gpu_uuid=u) for j,u in zip(pending,free)]
def idle_eligible_gpus(eligible,query=None):
if not set(eligible)<=set(GPUS):raise ValueError('Unknown eligible GPU')
if query is None:
from s2_precision_continuation_guard import device_usage
query=device_usage
idle=set();observations=[]
for u in GPUS:
if u not in eligible:continue
mem,util=query(u)
if not all(math.isfinite(x) and x>=0 for x in [mem,util]):raise ValueError('Invalid device occupancy')
ok=mem<=128*2**20 and util==0
observations.append(dict(gpu_uuid=u,memory_bytes=mem,utilization=util,idle=ok))
if ok:idle.add(u)
return idle,observations
def assigned_job(root,c,index,digest):
p=checked(root/'jobs'/str(index)/'assignment.json',digest);a=read(p)
if a['id']!=index or a['contract_sha256']!=sha(root/'contract.json') or a['gpu_uuid'] not in GPUS:raise ValueError('Actual assignment identity')
return dict(c['jobs'][index],gpu_uuid=a['gpu_uuid'])
def available_gpus(c):
import psutil
available={GPUS[0]}
item=Path(c['item1_recovery_root']);track=Path(c['track_a_training_root'])
names={Path(a).name for p in psutil.process_iter(['cmdline']) for a in p.info['cmdline'] or []}
if (item/'complete.json').exists() and not names & {'phase3_item1_postfit_worker.py','phase3_item1_resume.py'}:
available.add(GPUS[1])
if (track/'complete.json').exists() and not names & {'track_a_h2_queue.py','track_a_h2_guard.py','track_a_h2_worker.py'}:
# GPU2 stays reserved for the already-prioritized Track A postfit until its terminal.
post=Path(c['track_a_postfit_root'])
available.add(GPUS[3])
if (post/'complete.json').exists() and not names & {'track_a_postfit.py','track_a_postfit_worker.py'}:
available.add(GPUS[2])
return available
+88
View File
@@ -0,0 +1,88 @@
"""Independent bounded SEED1 fold guard; shared global stop and immutable S1."""
import fcntl
import json
import os
from pathlib import Path
import signal
import sys
import threading
import time
import psutil
from phase3_seed1_continuation_common import config,assigned_job
from phase2_h1_worker_v2 import save,sha
from s2_main_memory_guard import PeakRecords
from s2_precision_continuation_guard import device_usage
from s2_recovery_gpu_supervisor import supervise,tree_usage,stop_group,disk_usage
def run(queue,fold_id,assignment_sha):
c,_=config(queue);job=assigned_job(queue,c,fold_id,assignment_sha);c=dict(c,**job);root=queue/'jobs'/str(fold_id);os.sched_setaffinity(0,c['guard_cpus'])
with (root/'fold.lock').open('a') as lock, Path('/tmp/meltmind-seed1-'+c['gpu_uuid']+'.lock').open('a') as gpu_lock:
fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
fcntl.flock(gpu_lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
if (root/'started.json').exists():raise FileExistsError('Single SEED1 fold launch, no retry')
start=time.monotonic();cpu=time.process_time();requested=[];reasons=[];errors=[];samples=[];done=threading.Event();tracker=PeakRecords();receipt=None
save(root/'started.json',dict(pid=os.getpid(),unix=time.time(),contract_sha256=sha(queue/'contract.json'),gpu_uuid=c['gpu_uuid'],setup_gpu_seconds=c['setup_gpu_seconds']))
def elapsed():return c['setup_gpu_seconds']+time.monotonic()-start
def reason(x):
if x not in reasons:reasons.append(x)
command=[sys.executable,'tools/phase3_seed1_continuation_worker.py',str(queue),str(fold_id),assignment_sha]
def kill_worker():
for p in psutil.Process().children(recursive=True):
try:
if p.cmdline()==command:stop_group(p.pid)
except psutil.NoSuchProcess:pass
old={s:signal.signal(s,lambda signum,frame:requested.append('COORDINATOR_SIGNAL_'+str(signum))) for s in [signal.SIGINT,signal.SIGTERM]}
def scan():
if tracker.scan(root/'worker'):reason('RECORDED_VRAM_LIMIT')
def watch():
while not done.is_set():
try:
scan();mem,util=device_usage(c['gpu_uuid']);_,rss,_=tree_usage(os.getpid())
if mem>c['vram_bytes']:reason('DEVICE_VRAM_LIMIT')
if rss>c['rss_bytes']:reason('WHOLE_TREE_RSS_LIMIT')
if disk_usage(root)>c['disk_bytes']:reason('OUTPUT_DISK_LIMIT')
if elapsed()>=c['gpu_seconds']-5:reason('SEED1_FOLD_TIME_LIMIT')
if (queue/'global-stop.json').exists():reason('SEED1_GLOBAL_STOP')
for x in requested:reason(x)
save(root/'live-progress.json',dict(stage='SEED1_TRAINING',fold=fold_id,elapsed_gpu_seconds=elapsed(),device_bytes=mem,device_utilization=util,tree_rss_bytes=rss,recorded_peaks=tracker.maxima,reasons=reasons,unix=time.time()))
except Exception as exc:reason('MONITOR_ERROR');errors.append(dict(type=type(exc).__name__,reason=str(exc)))
if reasons:kill_worker()
done.wait(.5)
thread=None
try:
for i in range(3):
mem,util=device_usage(c['gpu_uuid']);samples.append(dict(unix=time.time(),memory_bytes=mem,utilization=util))
if mem>128*2**20 or util!=0:raise RuntimeError('SELECTED_GPU_BUSY')
if i<2:time.sleep(2)
save(root/'idle-polls.json',samples)
if requested or (queue/'global-stop.json').exists():raise RuntimeError('STOP_BEFORE_WORKER_START')
thread=threading.Thread(target=watch,daemon=True);thread.start()
remaining=c['gpu_seconds']-elapsed()-10
receipt=supervise(command,root/'guards','train',dict(total_cpu_seconds=c['cpu_seconds']-60,job_cpu_seconds=c['cpu_seconds']-60,job_wall_seconds=remaining,tree_rss_bytes=c['rss_bytes'],disk_bytes=c['disk_bytes'],max_processes=1),poll=.2,gpu_uuid=c['gpu_uuid'])
except BaseException as exc:reason(type(exc).__name__+': '+str(exc));kill_worker()
finally:
done.set()
if thread:thread.join(timeout=4)
if thread and thread.is_alive():reason('MONITOR_DID_NOT_EXIT')
try:scan()
except Exception as exc:reason('FINAL_PEAK_SCAN_ERROR');errors.append(dict(reason=str(exc)))
for s,handler in old.items():signal.signal(s,handler)
value=None;p=root/'worker/result.json'
try:
if p.exists():value=json.loads(p.read_text())
except Exception as exc:reason('RESULT_READ_ERROR');errors.append(dict(reason=str(exc)))
if not receipt or receipt['status']!='PASS':reason('WORKER_NOT_PASS')
if receipt and receipt.get('monitor_errors'):reason('WORKER_MONITOR_ERRORS')
if not value or value.get('status')!='COMPLETE':reason('NO_COMPLETE_TRAIN_RESULT')
if not all(k in tracker.witnesses for k in ['peak_allocated_bytes','peak_reserved_bytes']):reason('MISSING_CUDA_PEAK_RECORDS')
totalcpu=60+(receipt['cpu_seconds'] if receipt else 0)+time.process_time()-cpu
if elapsed()>c['gpu_seconds'] or totalcpu>c['cpu_seconds']:reason('FINAL_RESOURCE_LIMIT')
try:config(queue)
except Exception as exc:reason('FINAL_SOURCE_IDENTITY: '+str(exc))
if value and (value.get('optimizer_steps')!=c['target_steps'] or value.get('fold')!=fold_id or value.get('completed_epochs')!=200):reason('INCOMPLETE_EPOCH_OR_FOLD_COUNT')
save(root/'complete.json',dict(status='PASS' if not reasons else 'STOPPED_OR_FAILED',decision='TRAIN_COMPLETE_CHECKPOINT_FREEZE_PENDING' if not reasons else 'SEED1_FOLD_INCOMPLETE',reasons=reasons,monitor_errors=errors,
worker=receipt,recorded_memory=tracker.as_dict(),gpu_seconds_charged=elapsed(),fold=fold_id,cpu_seconds_charged=totalcpu,
worker_result_sha256=sha(p) if p.exists() else None,prediction_started=False,scoring_started=False,seed1_started=True,no_retry=True))
return 0 if not reasons else 1
if __name__=='__main__':raise SystemExit(run(Path(sys.argv[1]),int(sys.argv[2]),sys.argv[3]))
+212
View File
@@ -0,0 +1,212 @@
"""Five original grouped folds on released GPUs; train and freeze only, never score."""
import fcntl
import os
from pathlib import Path
import signal
import subprocess
import sys
import time
import psutil
from phase2_gate_outcome_review import validate_review
from phase2_h1_worker_v2 import checked, save, sha
from phase2_home_freeze_io import freeze
from s2_recovery_gpu_supervisor import disk_usage, tree_usage
from phase3_seed1_continuation_common import config, dispatchable, execution_command, project_cell, project_total, read, available_gpus, idle_eligible_gpus
def run(root):
c, _ = config(root)
validate_review(root)
lock = (root / 'queue.lock').open('a')
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
if (root / 'started.json').exists():
raise FileExistsError('One Seed1 attempt; no retry')
for p in psutil.process_iter(['pid', 'cmdline']):
if p.pid != os.getpid() and execution_command(p.info['cmdline'] or []):
raise RuntimeError('Existing Seed1 execution')
os.sched_setaffinity(0, c['coordinator_cpus'])
os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'expandable_segments:True'
start = time.monotonic()
cpu = time.process_time()
active, terminal, estimates, last_epoch = {}, {}, {}, {}
pending = list(c['jobs'])
requested = []
def stop(reason):
if not (root / 'global-stop.json').exists():
save(root / 'global-stop.json', dict(reason=reason, unix=time.time(), no_retry=True))
def active_charges():
return {i: c['jobs'][i]['setup_gpu_seconds'] + time.monotonic() - v['start']
for i, v in active.items()}
def charged():
return sum(v['gpu_seconds_charged'] for v in terminal.values()) + sum(active_charges().values())
def terminate(v):
if v.get('stop_sent') is None:
if v['process'].poll() is None:
v['process'].send_signal(signal.SIGTERM)
v['stop_sent'] = time.monotonic()
elif time.monotonic() - v['stop_sent'] > 20:
try:
parent = psutil.Process(v['process'].pid)
for p in reversed(parent.children(recursive=True)):
try:
p.kill()
except psutil.NoSuchProcess:
pass
parent.kill()
except psutil.NoSuchProcess:
pass
def collect(i, v):
rc = v['process'].poll()
if rc is None:
return False
p = root / 'jobs' / str(i) / 'complete.json'
r = read(p) if p.exists() else None
observed = c['jobs'][i]['setup_gpu_seconds'] + time.monotonic() - v['start']
ok = (rc == 0 and r is not None and r['status'] == 'PASS'
and not r.get('monitor_errors') and r.get('worker') is not None
and not r['worker'].get('monitor_errors'))
terminal[i] = dict(id=i, status='PASS' if ok else 'STOPPED_OR_FAILED', returncode=rc,
gpu_seconds_charged=max(observed, r['gpu_seconds_charged'] if r else 0),
guard_complete_sha256=sha(p) if r else None, gpu_uuid=v['job']['gpu_uuid'], assignment_sha256=sha(p.parent/'assignment.json'))
v['log'].close()
del active[i]
save(root / 'terminal-ledger.json', dict(entries=list(terminal.values()),
settled_gpu_seconds=sum(x['gpu_seconds_charged'] for x in terminal.values())))
if not ok:
stop('JOB_' + str(i) + '_FAILED')
return True
def dispatch():
eligible=available_gpus(c)-{v['job']['gpu_uuid'] for v in active.values()}
idle,observations=idle_eligible_gpus(eligible)
observed_unix=time.time()
save(root/'dispatch-idle-observation.json',dict(unix=observed_unix,eligible=sorted(eligible),observations=observations))
for j in dispatchable(pending, [v['job'] for v in active.values()], idle):
i = j['id']
d = root / 'jobs' / str(i)
d.mkdir(parents=True, exist_ok=False)
log = (d / 'entry.log').open('x')
begin = time.monotonic()
save(d / 'assignment.json', dict(id=i, gpu_uuid=j['gpu_uuid'], contract_sha256=sha(root/'contract.json'), unix=time.time(), release_evidence=sorted(eligible), idle_observations=observations, idle_observed_unix=observed_unix))
cmd = [sys.executable, 'tools/phase3_seed1_continuation_guard.py', str(root), str(i), sha(d/'assignment.json')]
try:
p = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT, start_new_session=True)
except BaseException:
log.close()
raise
active[i] = dict(process=p, log=log, start=begin, stop_sent=None, job=j)
pending.remove(c['jobs'][i])
save(d / 'dispatch.json', dict(pid=p.pid, command=cmd, unix=time.time(), gpu_uuid=j['gpu_uuid']))
handlers = {s: signal.signal(s, lambda n, f: requested.append('SIGNAL_' + str(n)))
for s in (signal.SIGTERM, signal.SIGINT)}
save(root / 'started.json', dict(pid=os.getpid(), unix=time.time(), new_fits=5,
contract_sha256=sha(root / 'contract.json'), review_sha256=sha(root / 'review.json'),
gpu_cap=c['gpu_cap'], A_refit=False, test_targets_read=False))
try:
while active or pending:
for i, v in list(active.items()):
collect(i, v)
for reason in requested:
stop(reason)
amount = charged()
_, rss, _ = tree_usage(os.getpid())
if amount >= c['train_gpu_cap'] - 80:
stop('CUMULATIVE_TRAIN_GPU_BOUNDARY')
if rss > c['aggregate_rss_bytes'] or disk_usage(root) > c['aggregate_disk_bytes']:
stop('AGGREGATE_MEMORY_DISK_LIMIT')
if not (root / 'global-stop.json').exists():
for i in active:
files = sorted((root / 'jobs' / str(i) / 'worker/history').glob('*.json'))
rows = [read(p) for p in files]
if not rows:
continue
epoch = rows[-1]['epoch']
# Recalculate every epoch: waiting, setup and observed cost stay charged.
if epoch > last_epoch.get(i, 0):
observation = dict(epoch=epoch, charged=active_charges()[i],
max_epoch_seconds=max(r['train_epoch_seconds'] for r in rows),
max_validation_seconds=max(r.get('validation_seconds',0) for r in rows))
cell = project_cell(c['jobs'][i], observation)
estimates[i] = cell
projection = dict(**project_total(c, estimates, terminal, active_charges()),
cell=cell, contract_sha256=sha(root / 'contract.json'), unix=time.time())
name = 'first-epoch-projection.json' if i not in last_epoch else 'latest-cost-projection.json'
save(root / 'jobs' / str(i) / name, projection)
save(root / 'latest-cost-projection.json', projection)
last_epoch[i] = epoch
if projection['status'] != 'PASS':
stop('PROJECTED_SEED1_BUDGET')
if not (root / 'global-stop.json').exists():
dispatch()
if (root / 'global-stop.json').exists():
for v in active.values():
terminate(v)
save(root / 'live-progress.json', dict(stage='STOPPING' if (root / 'global-stop.json').exists()
else 'SEED1_TRAINING', unix=time.time(), active=[dict(id=i, guard_pid=v['process'].pid)
for i, v in active.items()], terminal=list(terminal.values()), pending=[j['id'] for j in pending],
charged_training_gpu_seconds=charged(), postfit_gpu_reserve=c['postfit_gpu_reserve']))
if (root / 'global-stop.json').exists() and not active:
break
if active or pending:
time.sleep(1)
if (root / 'global-stop.json').exists() or len(terminal) != 5 or any(v['status'] != 'PASS' for v in terminal.values()):
raise RuntimeError('Seed1 training incomplete')
config(root)
models = []
for j in c['jobs']:
w = root / 'jobs' / str(j['id']) / 'worker'
r = read(w / 'result.json')
stats = read(checked(w / 'standardization.json', r['standardization_sha256']))
if (r['optimizer_steps'] != j['target_steps'] or r['completed_epochs'] != 200
or stats['parts'] != j['learning'] or r['test_targets_read']):
raise ValueError('Terminal model identity')
checked(r['checkpoint'], r['checkpoint_sha256'])
models.append(dict(job=j['id'], path=r['checkpoint'], sha256=r['checkpoint_sha256'],
standardization=str(w / 'standardization.json'),
standardization_sha256=r['standardization_sha256']))
if charged() > c['train_gpu_cap']:
raise ValueError('Terminal cumulative cost')
save(root / 'models-terminal.json', dict(models=models, new_fits=5, A_refit=False, test_prediction_started=False))
freeze(root, 'models', [root / 'contract.json', root / 'models-terminal.json', root / 'terminal-ledger.json',
*[Path(m['path']) for m in models], *[Path(m['standardization']) for m in models]])
save(root / 'complete.json', dict(status='TRAIN_COMPLETE_POSTFIT_REVIEW_PENDING', entries=list(terminal.values()),
training_gpu_seconds=charged(), postfit_gpu_reserve=c['postfit_gpu_reserve'], gpu_cap=c['gpu_cap'],
coordinator_cpu_seconds=time.process_time() - cpu, wall_seconds=time.monotonic() - start,
phase3_prior_gpu_once=c['phase3_prior_gpu_once'], cumulative_phase3_training_gpu_seconds=c['phase3_prior_gpu_once']+charged(),
no_retry=True, test_targets_read=False, postfit_started=False))
freeze(root, 'terminal', [root / 'contract.json', root / 'complete.json', root / 'models-terminal.json'])
return 0
except BaseException as exc:
stop(type(exc).__name__ + ': ' + str(exc))
while active:
for i, v in list(active.items()):
try:
if not collect(i, v):
terminate(v)
except Exception:
terminate(v)
if v['process'].poll() is not None:
terminal[i] = dict(id=i, status='MALFORMED_TERMINAL',
gpu_seconds_charged=c['jobs'][i]['setup_gpu_seconds'] + time.monotonic() - v['start'])
v['log'].close()
del active[i]
if active:
time.sleep(.5)
save(root / 'complete.json', dict(status='STOPPED', reason=str(exc), entries=list(terminal.values()),
never_attempted=[j['id'] for j in pending], training_gpu_seconds=charged(),
postfit_started=False, no_retry=True))
return 1
finally:
for s, handler in handlers.items():
signal.signal(s, handler)
if __name__ == '__main__':
raise SystemExit(run(Path(sys.argv[1])))
+67
View File
@@ -0,0 +1,67 @@
"""One new S2 fold, same frozen recipe with seed1/200epochs; no test prediction or target reads."""
import json,os,sys,time,traceback,random
from pathlib import Path
import numpy as np
import torch
from phase2_h2_s1_common import selected_checkpoint
from phase3_seed1_continuation_common import config,fold_view,first_epoch_release,assigned_job
from phase2_h2_data import standardization
from phase2_h2_data import load_part,to_device,arguments,cached
from phase2_h2_model import H2Geo,optimizer,lr_epoch
from phase2_h2_pilot_worker import peaks,train_step
from phase2_residual_math import residual_target
from phase2_h1_worker_v2 import save,sha
def run(root,fold_id,assignment_sha):
c,pilot=config(root);old=fold_view(pilot,fold_id);job=assigned_job(root,c,fold_id,assignment_sha);out=root/'jobs'/str(fold_id)/'worker';out.mkdir(exist_ok=False);(out/'history').mkdir()
os.sched_setaffinity(0,job['worker_cpus']);torch.set_num_threads(1);torch.set_num_interop_threads(1)
start=time.monotonic();cpu=time.process_time();stage='SETUP';epoch=0;steps=0
def record(item,epoch_file=False):
item=dict(fold=fold_id,stage=stage,epoch=epoch,optimizer_steps=steps,wall_seconds=time.monotonic()-start,cpu_seconds=time.process_time()-cpu,**item,**peaks())
with (out/'history.jsonl').open('a') as f:f.write(json.dumps(item,allow_nan=False)+'\n')
save(out/'progress.json',item)
if epoch_file:save(out/'history'/f'{epoch:03d}.json',item)
return item
try:
if os.environ.get('CUDA_VISIBLE_DEVICES')!=job['gpu_uuid'] or os.environ.get('PYTORCH_CUDA_ALLOC_CONF')!='expandable_segments:True':raise ValueError('GPU/allocator identity')
stage='LEARNING_ONLY_NORMALIZATION';normal=out/'normalization';normal.mkdir();stats=standardization(old,normal);save(out/'standardization.json',stats);save(out/'fold-identity.json',dict(actual_fold=fold_id,learning=old['folds'][0]['learning'],validation=old['folds'][0]['validation'],test=old['folds'][0]['test'],clock_sha256=old['folds'][0]['clock_sha256']))
torch.backends.cuda.matmul.allow_tf32=False;torch.backends.cudnn.allow_tf32=False
torch.cuda.set_device(0);torch.cuda.init();torch.cuda.reset_peak_memory_stats();random.seed(1);np.random.seed(1);torch.manual_seed(1);torch.cuda.manual_seed_all(1)
model=H2Geo().cuda();opt=optimizer(model);fold=old['folds'][0];parts=len(fold['learning']);total=fold['learning_nodes']
save(out/'environment.json',dict(torch=torch.__version__,gpu_uuid=job['gpu_uuid'],allocator_settings=str(torch.cuda.memory._snapshot().get('allocator_settings',{})),model_dtype='FP32',autocast=False,grad_scaler=False,fresh_seed=1,pilot_weights_reused=False,parameter_count=sum(p.numel() for p in model.parameters())))
for epoch in range(1,201):
stage='TRAIN';begin=time.monotonic();seen=0;nodes=0;loss_sum=0.
if any(abs(g['lr']-lr_epoch(epoch))>1e-15 for g in opt.param_groups):raise ValueError('Learning rate epoch mismatch')
order=np.random.default_rng(np.random.SeedSequence([1,epoch])).permutation(parts)
for index in order:
case=fold['learning'][int(index)];data=to_device(load_part(old,case,stats,'learning',out),'cuda')
result=train_step(model,opt,data,parts,total);seen+=1;steps+=1;nodes+=data['nodes'];loss_sum+=result['loss']
record(dict(case=case,epoch_parts=seen,epoch_nodes=nodes,lr=lr_epoch(epoch),loss=result['loss'],gradient_norm_before_clip=result['gradient_norm_before_clip']))
del data
if seen!=parts or nodes!=total:raise ValueError('Incomplete full-node learning epoch')
metrics=dict(train_mean_weighted_loss=loss_sum/parts,train_epoch_seconds=time.monotonic()-begin,epoch_parts=seen,epoch_nodes=nodes,lr=lr_epoch(epoch))
if epoch%10==0:
stage='VALIDATION_DIAGNOSTIC';model.eval();val_loss=0.;val_nodes=0;vstart=time.monotonic()
with torch.no_grad():
for case in fold['validation']:
data=to_device(load_part(old,case,stats,'validation',out),'cuda');p=model(*arguments(data),checkpoint_blocks=False)
if not bool(torch.isfinite(p).all()):raise ValueError('Nonfinite validation output')
with (out/'validation-target-access.jsonl').open('a') as f:f.write(json.dumps(dict(case=case,epoch=epoch,role='validation_diagnostic_only',target_sha256=old['parts'][case]['output_sha256']['target.npy']))+'\n')
target=residual_target(cached(old,case,'target'),data['F'],fold['c'],fit_dtype='float32')
delta=p.cpu().numpy().astype(np.float64)-target;val_loss+=float(delta@delta);val_nodes+=data['nodes'];del data,p
torch.cuda.synchronize()
if val_nodes!=fold['validation_nodes']:raise ValueError('Validation coverage')
metrics.update(validation_mse=val_loss/val_nodes,validation_nodes=val_nodes,validation_seconds=time.monotonic()-vstart,validation_used_for_selection=False)
stage='EPOCH_COMPLETE';record(metrics,epoch_file=True)
if epoch==1:
stage='WAIT_THIS_FOLD_FIRST_EPOCH_COST_GATE';record(dict(epoch_parts=seen,epoch_nodes=nodes))
while not first_epoch_release(root,fold_id):time.sleep(.5)
if epoch<200:
for group in opt.param_groups:group['lr']=lr_epoch(epoch+1)
stage='FINAL_CHECKPOINT';name=selected_checkpoint(epoch)
torch.save(dict(model=model.state_dict(),optimizer=opt.state_dict(),completed_epochs=epoch,optimizer_steps=steps,seed=1,fold=fold_id,contract_sha256=sha(root/'contract.json'),role='last_epoch_only'),out/name)
save(out/'result.json',dict(status='COMPLETE',fold=fold_id,completed_epochs=epoch,optimizer_steps=steps,learning_parts=parts,learning_nodes_per_epoch=total,checkpoint=str(out/name),checkpoint_sha256=sha(out/name),standardization_sha256=sha(out/'standardization.json'),test_targets_read=False,test_prediction_started=False,validation_used_for_selection=False,worker_wall_seconds=time.monotonic()-start,worker_cpu_seconds=time.process_time()-cpu,**peaks()))
except BaseException as exc:
memory=peaks() if torch.cuda.is_initialized() else dict(peak_allocated_bytes=0,peak_reserved_bytes=0)
save(out/'failure.json',dict(status='FAILED',stage=stage,epoch=epoch,optimizer_steps=steps,type=type(exc).__name__,reason=str(exc),traceback=traceback.format_exc(),worker_wall_seconds=time.monotonic()-start,worker_cpu_seconds=time.process_time()-cpu,**memory));raise
if __name__=='__main__':run(Path(sys.argv[1]),int(sys.argv[2]),sys.argv[3])
@@ -0,0 +1,43 @@
"""One labelled continuation: metadata-only, original STOP and scientific sources immutable."""
import copy
import shutil
import time
from pathlib import Path
from phase2_h1_worker_v2 import checked,sha,save
from phase2_home_freeze_io import verify
from phase3_seed1_common import config as original_config,read
from track_a_h2_prepare import local_code_closure
B=Path(__file__).resolve().parents[1]
OLD=Path('/home/frank14f/MeltMind-runs/phase3-seed1-2026-11-18')
ROOT=OLD.with_name('phase3-seed1-continuation-2026-11-18')
def remaining_contract(original,terminal):
c=copy.deepcopy(original)
spent={e['id']:e['gpu_seconds_charged'] for e in terminal['entries']}
old_charge=terminal['training_gpu_seconds']
if terminal['status']!='STOPPED' or terminal['never_attempted']!=[2,3,4] or list(spent)!=[0,1] or sum(spent.values())!=old_charge:raise ValueError('Original stopped scope and accounting')
c.update(version='phase3-seed1-labelled-continuation-v1',approval='1794984996.938059',accounting_approval='1794985731.591689',original_seed1_root=str(OLD),original_stop_publication='a45e375d0e8797e4d63258e446647191ef253585',continuation_number=1,fresh_restart_folds=[0,1],previously_unattempted_folds=[2,3,4],old_seed1_gpu_once=old_charge,cumulative_seed1_gpu_cap=167200,cumulative_fold_gpu_cap=32000,gpu_cap=167200-old_charge,train_gpu_cap=160000-old_charge,phase3_prior_gpu_once=original['phase3_prior_gpu_once']+old_charge,accounting='Runtime counters are new-run only. Add old_seed1_gpu_once once for cumulative seed1; phase3_prior already includes it. Never add it again to phase3. Remaining per-fold caps debit original fold charges; old STOP remains immutable.',remaining_caps_final=True)
for j in c['jobs']:
j['gpu_seconds']=32000-spent.get(j['id'],0);j['cpu_seconds']=j['gpu_seconds']
return c
def main():
start=time.process_time()
if ROOT.exists():raise FileExistsError('Existing sole continuation root')
oc,_=original_config(OLD);proof=verify(OLD,'budget-stop')
if proof['commit']!='a556b58340410dbe3faeb2dc2143b3708a7e724b':raise ValueError('Original STOP freeze')
c=remaining_contract(oc,read(OLD/'complete.json'))
source=dict(oc['source_sha256']);source.update(oc['code_sha256']);source.update(proof['files'])
for p in [OLD/'budget-stop-git-freeze.json',OLD/'budget-stop-authorization.json',OLD/'expected-review-binding.json']:
source[str(p)]=sha(p)
files=[B/'tools'/n for n in ['phase3_seed1_continuation_common.py','phase3_seed1_continuation_queue.py','phase3_seed1_continuation_guard.py','phase3_seed1_continuation_worker.py','prepare_phase3_seed1_continuation.py','test_phase3_seed1_continuation.py']]
codes=local_code_closure(files)
for p,h in oc['code_sha256'].items():codes.setdefault(p,h)
for p,h in source.items():checked(p,h)
ROOT.mkdir()
shutil.copyfile(OLD/'comparison-contract.json',ROOT/'comparison-contract.json')
c.update(source_sha256=source,code_sha256=codes,comparison_contract=str(ROOT/'comparison-contract.json'))
save(ROOT/'contract.json',c)
save(ROOT/'preparation-audit.json',dict(status='PREPARED_PENDING_NARROW_REVIEW',contract_sha256=sha(ROOT/'contract.json'),code_files=len(codes),source_files=len(source),source_pins='PASS',old_stop_freeze=proof['commit'],old_seed1_gpu_once=c['old_seed1_gpu_once'],new_train_gpu_cap=c['train_gpu_cap'],new_gpu_cap=c['gpu_cap'],cumulative_seed1_gpu_cap=c['cumulative_seed1_gpu_cap'],new_fold_caps=[j['gpu_seconds'] for j in c['jobs']],phase3_prior_gpu_once=c['phase3_prior_gpu_once'],metadata_cpu_seconds=time.process_time()-start,scientific_arrays_read=False,model_weights_read=False,test_targets_read=False,new_fits_launched=False,changes='Remaining budget/contract accounting; isolated entry names; original science, projection, idle/first-epoch gates unchanged'))
print(read(ROOT/'preparation-audit.json'))
if __name__=='__main__':main()
+62
View File
@@ -0,0 +1,62 @@
"""Narrow CPU-only continuation delta checks; no datasets or CUDA execution."""
import ast
import copy
import inspect
import unittest
from pathlib import Path
from phase3_seed1_common import project_cell, project_total
TOOLS=Path(__file__).resolve().parent
class ContinuationTests(unittest.TestCase):
def test_worker_guard_queue_only_entry_names_change(self):
for suffix in ['worker','guard','queue']:
original=(TOOLS/f'phase3_seed1_{suffix}.py').read_text()
new=(TOOLS/f'phase3_seed1_continuation_{suffix}.py').read_text()
restored=new.replace('phase3_seed1_continuation_common','phase3_seed1_common').replace('tools/phase3_seed1_continuation_guard.py','tools/phase3_seed1_guard.py').replace('tools/phase3_seed1_continuation_worker.py','tools/phase3_seed1_worker.py')
self.assertEqual(original,restored)
def test_common_projection_idle_and_release_functions_unchanged(self):
import phase3_seed1_common as old
import phase3_seed1_continuation_common as new
for name in ['fold_view','project_cell','project_total','first_epoch_release','dispatchable','idle_eligible_gpus','assigned_job','available_gpus']:
self.assertEqual(inspect.getsource(getattr(old,name)),inspect.getsource(getattr(new,name)))
def test_duplicate_detection_includes_both_generations(self):
from phase3_seed1_continuation_common import execution_command
for prefix in ['phase3_seed1','phase3_seed1_continuation']:
for suffix in ['queue','guard','worker']:
self.assertTrue(execution_command(['python',f'tools/{prefix}_{suffix}.py']))
self.assertFalse(execution_command(['python','pilot.py']))
def budget_fixture(self):
from prepare_phase3_seed1_continuation import remaining_contract
old=dict(jobs=[dict(id=i,gpu_seconds=22320,cpu_seconds=22320,validation_seconds_bound=12) for i in range(5)],phase3_prior_gpu_once=72421.31599938683,phase3_gpu_cap=612000,postfit_gpu_reserve=7200)
terminal=dict(status='STOPPED',never_attempted=[2,3,4],entries=[dict(id=0,gpu_seconds_charged=269.53611274203286),dict(id=1,gpu_seconds_charged=269.5359442579793)],training_gpu_seconds=539.0720570000121)
original=copy.deepcopy(old);c=remaining_contract(old,terminal);self.assertEqual(old,original)
return c
def test_full_pending_caps_include_old_cost_once(self):
c=self.budget_fixture();p=project_total(c,{}, {})
self.assertEqual(p['status'],'PASS')
self.assertAlmostEqual(p['projected_gpu_seconds']+c['old_seed1_gpu_once'],167200,places=8)
self.assertAlmostEqual(p['projected_phase3_gpu_seconds'],72421.31599938683+167200,places=8)
self.assertEqual(sum(j['gpu_seconds'] for j in c['jobs']),c['train_gpu_cap'])
self.assertEqual(c['postfit_gpu_reserve'],7200)
def test_settled_new_cost_does_not_double_charge_old(self):
c=self.budget_fixture();terminal={i:dict(gpu_seconds_charged=10000+i) for i in range(5)}
p=project_total(c,{},terminal)
self.assertAlmostEqual(p['projected_gpu_seconds'],57210)
self.assertAlmostEqual(p['projected_phase3_gpu_seconds'],72421.31599938683+539.0720570000121+57210)
self.assertEqual(p['status'],'PASS')
def test_final_fold_or_total_overrun_stops(self):
c=self.budget_fixture();estimates={0:dict(projected_gpu_seconds=c['jobs'][0]['gpu_seconds']+.01)}
self.assertEqual(project_total(c,estimates,{})['status'],'STOP_BUDGET_PROJECTION')
observation=dict(epoch=2,charged=270,max_epoch_seconds=104.409218447,max_validation_seconds=0)
self.assertTrue(project_cell(c['jobs'][1],observation)['within_cap'])
observation['max_epoch_seconds']=140
self.assertFalse(project_cell(c['jobs'][1],observation)['within_cap'])
if __name__=='__main__':unittest.main()