Record verified N128 worker launch after three idle polls
This commit is contained in:
1 parent
ea5ac05191
commit
45abbcd50d
11 files changed
+449
No files matched your search
@@ -18,3 +18,5 @@
|
||||
- Terminal update (2026-11-15 06:14 UTC+8): recovery82149 and chain42022 have finished. Recovery134=131 PASS+3 wall_limit; canonical499=493 PASS+6 wall_limit. E256 probe passed; N128-seed0 failed at epoch10 with nonfinite encoder output. Pilot stopped without retry/width selection after296.023387s of7200s; other three runs never started. Read `docs/s2-recovery-pilot-stage-result-2026-11-15.md`; preserve all evidence,125-test seal,and source pins. Do not restart from stale progress. Claude received stop notice1794694508.930959; next changes require the existing collaboration review, not a repeat sharing approval.
|
||||
|
||||
- Precision-v2 terminal correction (2026-11-15 06:30 UTC+8): diagnostic41846 completed and reproduced fp16 slice overflow; fixed8a548cc preserves original sources and implements the approved shared FP32 aggregation. New v2 session26792 has already exited: probe PASS at187345 nodes, VRAM12586057728B, E_max42, but SELECTED_GPU_BUSY_BEFORE_RUN stopped before any training run (runs=[]), total11.143491s. Earlier inference from pilot-started to N128 running was incorrect and retracted. Do not restart or rerun probe without the engineering continuation ruling; read latest handoff and raw complete receipts.125 tests remain sealed.
|
||||
|
||||
- Latest engineering continuation (2026-11-15): Claude1794695421.177269 authorized reuse of v2 probe after its zero-run infrastructure stop. Fixedea5ac05; sole active session34632/root `artifacts/s2-pilot-v2-continuation-2026-11-15`, entryPID209380 and N128 workerPID209415 verified running. Three idle polls passed over20.285703s, sameGPU. ReuseE_max42/probe; original11.143491s plus all new waits/setup/runs share7200s. Keep v2 zero-run stop and original numerical failure immutable. Inspect real worker/start/progress/terminal receipts before any action; never duplicate or modify pinned code.125 tests remain sealed.
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"started_unix": 1794695673.7712483,
|
||||
"contract_sha256": "f6eaa3cb2fd6918e5bd940826a2cc954ffcee0a9a2d1578b2da1e5e1c72abd5c"
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"command": [
|
||||
"/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python",
|
||||
"tools/run_s2_pilot_training_v2.py",
|
||||
"artifacts/s2-pilot-v2-continuation-2026-11-15",
|
||||
"0"
|
||||
],
|
||||
"limits": {
|
||||
"total_cpu_seconds": 7200,
|
||||
"job_cpu_seconds": 1800,
|
||||
"job_wall_seconds": 1797.4587304769666,
|
||||
"tree_rss_bytes": 17179869184,
|
||||
"disk_bytes": 5368709120,
|
||||
"max_processes": 2
|
||||
},
|
||||
"started_unix": 1794695671.3665867
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"status": "IDLE_CONFIRMED",
|
||||
"uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c",
|
||||
"polls": 3,
|
||||
"consecutive_idle": 3,
|
||||
"wall_seconds": 20.285703044966795,
|
||||
"last": {
|
||||
"unix": 1794695671.319949,
|
||||
"uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c",
|
||||
"memory_MiB": 5,
|
||||
"utilization_percent": 0,
|
||||
"occupants": [],
|
||||
"idle": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{"unix": 1794695651.1208584, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 0, "occupants": [], "idle": true, "poll": 1, "consecutive_idle": 1, "elapsed_seconds": 0.08635188097832724}
|
||||
{"unix": 1794695661.2204115, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 0, "occupants": [], "idle": true, "poll": 2, "consecutive_idle": 2, "elapsed_seconds": 10.185906714003067}
|
||||
{"unix": 1794695671.319949, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 0, "occupants": [], "idle": true, "poll": 3, "consecutive_idle": 3, "elapsed_seconds": 20.285443256958388}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"started_unix": 1794695671.323467,
|
||||
"contract_sha256": "f6eaa3cb2fd6918e5bd940826a2cc954ffcee0a9a2d1578b2da1e5e1c72abd5c"
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"stage": "training",
|
||||
"epoch": 5,
|
||||
"parts_done": 113,
|
||||
"parts_total": 382,
|
||||
"updates": 2021,
|
||||
"elapsed_seconds": 148.30627381202066,
|
||||
"peak_allocated_bytes": 4706622464,
|
||||
"peak_reserved_bytes": 8243904512
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"status": "N128_WORKER_ACTUALLY_RUNNING",
|
||||
"observed_unix": 1794695823.3364732,
|
||||
"session": 34632,
|
||||
"entry_PID": 209380,
|
||||
"N128_worker_PID": 209415,
|
||||
"code_pins_match": true,
|
||||
"v2_source_bindings_match": true,
|
||||
"test_parts": 125,
|
||||
"test_manifest_overlap": 0,
|
||||
"probe_reused": true,
|
||||
"probe_rerun": false,
|
||||
"gpu_switched": false,
|
||||
"total_v2_limit_seconds": 7200,
|
||||
"prior_v2_wall_seconds": 11.143491394002922,
|
||||
"contract_sha256": "f6eaa3cb2fd6918e5bd940826a2cc954ffcee0a9a2d1578b2da1e5e1c72abd5c",
|
||||
"file_sha256": {
|
||||
"artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/started.json": "faafcb0a870d5de0524f6377dc11cf8be5bc9158a33bcc23e7ed16a241aea221",
|
||||
"artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.started.json": "1bc75baded2ffc1dc8aa53934e1be28a1e138782be37bd4ebaf2af644ecf0e5b",
|
||||
"artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/initial-selected.json": "0e7a31405a607b1913b9acacf49f1e79d3dd8004d183ef0bc077c5bfebd8cb50",
|
||||
"artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/initial-selected.jsonl": "2aafe2f73dba34188ac857533f4def13b14cdff2ad7263012fcabc957a6b805d",
|
||||
"artifacts/s2-pilot-v2-continuation-2026-11-15/pilot-started.json": "8787de8b0f95ea0d4aadb3f44e4d899cee7cfa3eb11a505c6b3e033661dcf0fd",
|
||||
"artifacts/s2-pilot-v2-continuation-2026-11-15/public/N128-actual-start-progress.json": "626b2efcbf5d38d3d5fffa24433a81698a55e7f93319f1b6c4fff02ae9b655cc",
|
||||
"artifacts/s2-pilot-v2-continuation-2026-11-15/started.json": "147318ce2fa0bc76488f14205eb6dca87747ff5234e6dda62fc46b28c1c4f77c",
|
||||
"artifacts/s2-pilot-v2-continuation-2026-11-15/training-contract.json": "f6eaa3cb2fd6918e5bd940826a2cc954ffcee0a9a2d1578b2da1e5e1c72abd5c"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"started_unix": 1794695650.9901605,
|
||||
"contract_sha256": "225e51f42d545b868964fc7aa793805d6c35ba821915fcf1bffb47d516eb3574",
|
||||
"prior_wall_seconds": 11.143491394002922
|
||||
}
|
||||
@@ -0,0 +1,356 @@
|
||||
{
|
||||
"fold": 0,
|
||||
"pool": 100,
|
||||
"seed": 0,
|
||||
"gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c",
|
||||
"code_sha256": {
|
||||
"tools/prepare_s2_recovered_pilot_training.py": "ab5a4274363598ff933ff873df2b00497f069b1fb45756a1d1dba8d723294f68",
|
||||
"tools/run_s2_recovered_pilot.py": "56962009bf5eaaee599ffea6a3d454372e0aea2842fa79eac609fd4a24ecbb39",
|
||||
"tools/run_s2_pilot_probe.py": "2360ab7b44b179144a5ecd2d691e33cd36a6ec4bfeac71452bf062e7f42f81eb",
|
||||
"tools/run_s2_pilot_training.py": "d5e543478543243fe1db3475a4d4cd135fbdcf631d3bce48c4b5af8b2edafb22",
|
||||
"tools/s2_pilot_train.py": "e9cceead51a5cc092aad23409ad98909c84a62e7e10571e750d1e9be1ad3b948",
|
||||
"tools/s2_recovered_pilot_gpu_guard.py": "9a27f47918c865ce37e46b0be54144dc3f9ab1058e4eddc9f6fbc771f4959eb0",
|
||||
"tools/s2_recovery_gpu_supervisor.py": "689c0f3179e4fcd362e546d90214829a9e4f8739abdbdf74b560763e5a392f15",
|
||||
"tools/s2_e1_supervisor.py": "6753c9ceb0b932047ba505bd15681f109e9407bdb58fd96c55e1431a3f1b4a02",
|
||||
"tools/s2_pilot_data.py": "82c1fd18ec51757479c8fd7dda2db4d5f38c366d77fedcca3216a08998214c97",
|
||||
"tools/s2_incident_recovery_manifest.py": "8cbb2e1067c2cc22762d8ae3cf192aa2c5fa7c7a199a22e7a39a25570ee1c3b1",
|
||||
"tools/freeze_g1_components.py": "d0cee60d95243ac2c59ef3f2244ba818315718c4e3d1895f2f9f064f1f7f5ecd",
|
||||
"tools/s2_recovery_monitoring.py": "442c2a9c75e8c00b30a8f2e91cb52d87c466b19a80aea9128b10c021e61b922e",
|
||||
"tools/prepare_s2_incident_recovery.py": "4f772573dedc2b7921986e25eaedb14f91d98b235421829c284f3b40b85710aa",
|
||||
"tools/run_s2_recovery_chain.py": "76d0808bdb2d624cfbaadf5a3c5e7e9a0c2be5d5ac41d4ac3fd9ccd48743933c",
|
||||
"tools/s2_models/__init__.py": "a0fa1323d3553172da66c5d442c5fc4fcb0d5055c886c776928604550ea97609",
|
||||
"tools/s2_models/attention.py": "7714adfccf6a0fc6b52a44aee3a50321a88898bf3d743680a1108c01e1ff8a61",
|
||||
"tools/s2_models/blocks.py": "b13e986cea5f1d7f325a1638c1505197b4a105baacff5a9a515f23fcaf28ea75",
|
||||
"tools/s2_models/features.py": "3b34ae61cf8012ebd58b7505a73d7aa43d8ee1890ac54f98d7917b67887a45a4",
|
||||
"tools/s2_models/network.py": "5137e845d81d1223afc13bf87e03ccba1132a3baa75d5189a8f320b2752be071",
|
||||
"tools/s2_models/normalization.py": "7180696d3845799e28d8c31ddef957d075de6a2e46d96a3a0f0df16fc05234a0",
|
||||
"tools/s2_models/objectives.py": "5b6527a45f08690055d572335470ce5979e8bb86e1cab00de594f79ae21316b1",
|
||||
"tools/prepare_s2_precision_pilot.py": "c3a71b0ff58bcea4433ff93e18ef14d3a00521438a76baaa69830a5ee8ac098a",
|
||||
"tools/run_s2_precision_pilot.py": "ac8347be1ca11f48cb67abbf5e5c78b39af46ed3a7c6f89b49098fd5e77a6904",
|
||||
"tools/s2_precision_pilot_gpu_guard.py": "5cf0f41de34b28394dbee4672b5bd34eb9c4fe2344822aa3d648f20561d96328",
|
||||
"tools/run_s2_pilot_probe_v2.py": "61c08499346d50405d0c254de54deea24b3522256bf1c8b0aec3e9f4ddab6390",
|
||||
"tools/run_s2_pilot_training_v2.py": "f461cb43bc602a32677d7dcfb8a322e206531ad152fa33bb4e36ea8546a1c0f2",
|
||||
"tools/run_s2_encoder_diagnostic.py": "8f31d312fb53bfdb9b1804af064e463d3989f1b33c4aa7279a967ca82dd2c0a4",
|
||||
"tools/s2_models_v2/__init__.py": "0302d333bcd8b33e85aaa6a4e7822e7e1ccfc66343136107fea6f4ca283ba1ec",
|
||||
"tools/s2_models_v2/attention.py": "3829e34ca30942c863699c4d43fb6fa104452c6c04c4d606c11019d3dde16ffa",
|
||||
"tools/s2_models_v2/blocks.py": "b13e986cea5f1d7f325a1638c1505197b4a105baacff5a9a515f23fcaf28ea75",
|
||||
"tools/s2_models_v2/network.py": "a443470caeecdf529504bffab3ca8cc0b70ce44b505bad60f9c63db9709dbb83",
|
||||
"tools/run_s2_precision_continuation.py": "9fb5a22007788f0c18c77a82b6a8b76fbb53a98b49cc9251bc49a7a5b8312f2f",
|
||||
"tools/s2_precision_continuation_guard.py": "01235d897c7c03e5ffa109c58b14f5d24d128e8f1ba873e1f76cfb14e02f7a7f",
|
||||
"tools/s2_gpu_idle_gate.py": "e2b1613e7e6ea398a8d555220a9365a1ad998ab4ef4b2bf8ec7cc4e31f8b4b9f"
|
||||
},
|
||||
"role_manifest": "artifacts/s2-pilot-inputs-2026-11-15/role-manifest.json",
|
||||
"role_manifest_sha256": "a6dce3ebf875dbbe2df349f06e728960140f131d4d55ebd00af03945a46ac35c",
|
||||
"success_manifest": "artifacts/s2-pilot-v2-2026-11-15/success-manifest.json",
|
||||
"success_manifest_sha256": "678ca725214c8b7469c55990ded3a16826fc62397ff1a3e5a6590b6ba309c2f3",
|
||||
"probe_cases": [
|
||||
"00001260_a4715d363f88400c8f0238d1_step_003",
|
||||
"00000290_58f8ff46b7564e989f9695b7_step_003",
|
||||
"00001557_16c191aef5034779aaab6e3f_step_000",
|
||||
"00000293_58f8ff46b7564e989f9695b7_step_006",
|
||||
"00001193_46b17dd88ed24f55bca2eb69_step_000",
|
||||
"00000105_52f7124261ae4c558f800775_step_002",
|
||||
"00001592_35f4abaacc9447e4ac0b964f_step_001",
|
||||
"00000140_2fc54fcd110d4f49969163c4_step_003",
|
||||
"00000166_2246cab58dbe4a2fb00fed50_step_000",
|
||||
"00001321_7cef9b79a6b04db894fd74e5_step_000",
|
||||
"00000462_516b1784884d4facbc4c0d74_step_010",
|
||||
"00000470_11951e51b89249dba38ded3e_step_005",
|
||||
"00001495_6c8ef6d2f5564d139a5077f8_step_006",
|
||||
"00001484_a45f6c99e2154f3b94955175_step_068",
|
||||
"00000784_a25fb37392a44447b9684f16_step_009",
|
||||
"00000314_3062bccff48e47a2b9de05e3_step_009",
|
||||
"00001186_507b60b57791425bbb73273e_step_001",
|
||||
"00001569_6fc579f3224d4d53a1b60710_step_000",
|
||||
"00000617_c710d414bbbc4340b682c58a_step_003",
|
||||
"00000666_6f67a59c16b949a4b66dcfe1_step_000"
|
||||
],
|
||||
"probe_selection": "20 equally spaced ranks in successful learning node-count order; includes min/max; tie by case",
|
||||
"learning_nodes": 11863762,
|
||||
"validation_nodes": 3330152,
|
||||
"successful": {
|
||||
"learning": {
|
||||
"parts": 382,
|
||||
"nodes": 11300812
|
||||
},
|
||||
"validation": {
|
||||
"parts": 111,
|
||||
"nodes": 3255106
|
||||
}
|
||||
},
|
||||
"extrapolation": "Train rate * all 387 planned learning nodes + FP32 inference rate * all 112 planned validation nodes; conservative if failed parts dropped",
|
||||
"runs": [
|
||||
{
|
||||
"name": "N128-seed0",
|
||||
"arm": "N",
|
||||
"width": 128,
|
||||
"seed": 0
|
||||
},
|
||||
{
|
||||
"name": "E128-seed0",
|
||||
"arm": "E",
|
||||
"width": 128,
|
||||
"seed": 0
|
||||
},
|
||||
{
|
||||
"name": "N256-seed0",
|
||||
"arm": "N",
|
||||
"width": 256,
|
||||
"seed": 0
|
||||
},
|
||||
{
|
||||
"name": "E256-seed0",
|
||||
"arm": "E",
|
||||
"width": 256,
|
||||
"seed": 0
|
||||
}
|
||||
],
|
||||
"limits": {
|
||||
"total_gpu_wall_seconds": 7200,
|
||||
"probe_wall_seconds": 300,
|
||||
"run_wall_seconds": 1800,
|
||||
"host_rss_bytes": 17179869184,
|
||||
"vram_bytes": 16106127360,
|
||||
"disk_bytes": 5368709120
|
||||
},
|
||||
"coverage_boundary": {
|
||||
"planned_max_learning_nodes": 187345,
|
||||
"max_successful_learning_nodes": 187345,
|
||||
"failed_parts": [
|
||||
{
|
||||
"case": "00000575_5a4cef68211d4706b1ec8586_step_004",
|
||||
"role": "validation",
|
||||
"nodes": 75046,
|
||||
"component_id": "component-3a3e340f4dfac391",
|
||||
"guard": {
|
||||
"status": "STOPPED_OR_FAILED",
|
||||
"command": [
|
||||
"/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python",
|
||||
"tools/s2_pilot_prepare.py",
|
||||
"artifacts/s2-pilot-inputs-2026-11-15",
|
||||
"181"
|
||||
],
|
||||
"returncode": -9,
|
||||
"stop_reason": "wall_limit",
|
||||
"cpu_seconds": 1799.766116,
|
||||
"wall_seconds": 1800.400393687014,
|
||||
"peak_tree_rss_bytes": 602218496,
|
||||
"peak_process_count": 1,
|
||||
"output_disk_bytes_before_receipt": 4096,
|
||||
"max_poll_gap_seconds": 1.128913332009688,
|
||||
"poll_target_seconds": 0.25,
|
||||
"reaped_orphan_cpu_seconds": 0.0,
|
||||
"supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure",
|
||||
"gpu_visible": false,
|
||||
"retry": false
|
||||
},
|
||||
"failure": null
|
||||
},
|
||||
{
|
||||
"case": "00000614_c710d414bbbc4340b682c58a_step_000",
|
||||
"role": "learning",
|
||||
"nodes": 145860,
|
||||
"component_id": "component-4bd089778094e746",
|
||||
"guard": {
|
||||
"status": "STOPPED_OR_FAILED",
|
||||
"command": [
|
||||
"/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python",
|
||||
"tools/s2_pilot_prepare.py",
|
||||
"artifacts/s2-pilot-inputs-2026-11-15",
|
||||
"201"
|
||||
],
|
||||
"returncode": -9,
|
||||
"stop_reason": "wall_limit",
|
||||
"cpu_seconds": 1799.796751,
|
||||
"wall_seconds": 1800.0567526929663,
|
||||
"peak_tree_rss_bytes": 629305344,
|
||||
"peak_process_count": 1,
|
||||
"output_disk_bytes_before_receipt": 4096,
|
||||
"max_poll_gap_seconds": 1.1576595069491304,
|
||||
"poll_target_seconds": 0.25,
|
||||
"reaped_orphan_cpu_seconds": 0.0,
|
||||
"supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure",
|
||||
"gpu_visible": false,
|
||||
"retry": false
|
||||
},
|
||||
"failure": null
|
||||
},
|
||||
{
|
||||
"case": "00000625_6af98f5458b347bb843f0562_step_000",
|
||||
"role": "learning",
|
||||
"nodes": 145707,
|
||||
"component_id": "component-4bd089778094e746",
|
||||
"guard": {
|
||||
"status": "STOPPED_OR_FAILED",
|
||||
"command": [
|
||||
"/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python",
|
||||
"tools/s2_pilot_prepare.py",
|
||||
"artifacts/s2-pilot-inputs-2026-11-15",
|
||||
"211"
|
||||
],
|
||||
"returncode": -9,
|
||||
"stop_reason": "wall_limit",
|
||||
"cpu_seconds": 1799.8864629999998,
|
||||
"wall_seconds": 1800.119237862993,
|
||||
"peak_tree_rss_bytes": 647880704,
|
||||
"peak_process_count": 1,
|
||||
"output_disk_bytes_before_receipt": 4096,
|
||||
"max_poll_gap_seconds": 1.159920622012578,
|
||||
"poll_target_seconds": 0.25,
|
||||
"reaped_orphan_cpu_seconds": 0.0,
|
||||
"supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure",
|
||||
"gpu_visible": false,
|
||||
"retry": false
|
||||
},
|
||||
"failure": null
|
||||
},
|
||||
{
|
||||
"case": "00001466_a45f6c99e2154f3b94955175_step_050",
|
||||
"role": "learning",
|
||||
"nodes": 77138,
|
||||
"component_id": "component-7fe5173e77c86f9d",
|
||||
"guard": {
|
||||
"status": "STOPPED_OR_FAILED",
|
||||
"command": [
|
||||
"/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python",
|
||||
"tools/s2_pilot_prepare.py",
|
||||
"artifacts/s2-pilot-recovery-inputs-2026-11-15",
|
||||
"83"
|
||||
],
|
||||
"returncode": -9,
|
||||
"stop_reason": "wall_limit",
|
||||
"cpu_seconds": 1799.926194,
|
||||
"wall_seconds": 1800.0714844280155,
|
||||
"peak_tree_rss_bytes": 383070208,
|
||||
"peak_process_count": 1,
|
||||
"output_disk_bytes_before_receipt": 4096,
|
||||
"max_poll_gap_seconds": 1.0109735240112059,
|
||||
"poll_target_seconds": 0.25,
|
||||
"reaped_orphan_cpu_seconds": 0.0,
|
||||
"supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure",
|
||||
"gpu_visible": false,
|
||||
"retry": false,
|
||||
"monitor_errors": 0,
|
||||
"last_monitor_error": null
|
||||
},
|
||||
"failure": null
|
||||
},
|
||||
{
|
||||
"case": "00001470_a45f6c99e2154f3b94955175_step_054",
|
||||
"role": "learning",
|
||||
"nodes": 124817,
|
||||
"component_id": "component-7fe5173e77c86f9d",
|
||||
"guard": {
|
||||
"status": "STOPPED_OR_FAILED",
|
||||
"command": [
|
||||
"/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python",
|
||||
"tools/s2_pilot_prepare.py",
|
||||
"artifacts/s2-pilot-recovery-inputs-2026-11-15",
|
||||
"87"
|
||||
],
|
||||
"returncode": -9,
|
||||
"stop_reason": "wall_limit",
|
||||
"cpu_seconds": 1799.664371,
|
||||
"wall_seconds": 1800.0443400070071,
|
||||
"peak_tree_rss_bytes": 578732032,
|
||||
"peak_process_count": 1,
|
||||
"output_disk_bytes_before_receipt": 4096,
|
||||
"max_poll_gap_seconds": 1.007104828953743,
|
||||
"poll_target_seconds": 0.25,
|
||||
"reaped_orphan_cpu_seconds": 0.0,
|
||||
"supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure",
|
||||
"gpu_visible": false,
|
||||
"retry": false,
|
||||
"monitor_errors": 0,
|
||||
"last_monitor_error": null
|
||||
},
|
||||
"failure": null
|
||||
},
|
||||
{
|
||||
"case": "00001473_a45f6c99e2154f3b94955175_step_057",
|
||||
"role": "learning",
|
||||
"nodes": 69428,
|
||||
"component_id": "component-7fe5173e77c86f9d",
|
||||
"guard": {
|
||||
"status": "STOPPED_OR_FAILED",
|
||||
"command": [
|
||||
"/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python",
|
||||
"tools/s2_pilot_prepare.py",
|
||||
"artifacts/s2-pilot-recovery-inputs-2026-11-15",
|
||||
"90"
|
||||
],
|
||||
"returncode": -9,
|
||||
"stop_reason": "wall_limit",
|
||||
"cpu_seconds": 1799.751432,
|
||||
"wall_seconds": 1800.2498029720155,
|
||||
"peak_tree_rss_bytes": 438853632,
|
||||
"peak_process_count": 1,
|
||||
"output_disk_bytes_before_receipt": 4096,
|
||||
"max_poll_gap_seconds": 0.8929723129840568,
|
||||
"poll_target_seconds": 0.25,
|
||||
"reaped_orphan_cpu_seconds": 0.0,
|
||||
"supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure",
|
||||
"gpu_visible": false,
|
||||
"retry": false,
|
||||
"monitor_errors": 0,
|
||||
"last_monitor_error": null
|
||||
},
|
||||
"failure": null
|
||||
}
|
||||
],
|
||||
"statement": "VRAM probe covers only the maximum successful learning part included; not failed/larger unprobed parts"
|
||||
},
|
||||
"authorizing_rulings": [
|
||||
"1794680129.303719",
|
||||
"1794681576.845789",
|
||||
"1794681807.376669",
|
||||
"1794689807.792819",
|
||||
"1794694544.819729",
|
||||
"1794694646.032819"
|
||||
],
|
||||
"test_labels_read": false,
|
||||
"precision_variant": {
|
||||
"version": "S2-pilot-v2-slice-FP32",
|
||||
"changed": "Only shared N/E slice aggregation numerator, denominator and normalization in FP32 with autocast disabled",
|
||||
"original_pilot": "artifacts/s2-pilot-training-2026-11-15",
|
||||
"original_contract_sha256": "f6987431f478275485d18f62240144c2626cab3fbeb75c3e130b68b5d57a1fd2",
|
||||
"original_status": "PILOT_INCOMPLETE",
|
||||
"diagnostic_root": "artifacts/s2-encoder-diagnostic-2026-11-15",
|
||||
"diagnostic_result_sha256": "69132454710b5e063567cd7424a87c23611719147ab45557f205855493ec3e41",
|
||||
"diagnostic_guard_sha256": "ec7b55e2cbca465c29aec84630fc02c9cfd9fed71e98226a90eb5b3fb2b45537",
|
||||
"from_scratch": true,
|
||||
"original_weights_reused": false,
|
||||
"preparation_repeated": false,
|
||||
"reprobe_max_nodes": 187345,
|
||||
"VRAM_limit_failure_no_batch_or_width_fallback": true
|
||||
},
|
||||
"E_max": 42,
|
||||
"cosine_T_max": 42,
|
||||
"projected_epoch_seconds": 35.118895331723465,
|
||||
"projected_epochs_seconds": 1474.9936039323854,
|
||||
"training_seconds_per_node": 2.5702122098117325e-06,
|
||||
"inference_seconds_per_node": 1.389278743139353e-06,
|
||||
"timing_probe_receipt": "artifacts/s2-pilot-v2-2026-11-15/timing-probe/result.json",
|
||||
"timing_probe_sha256": "96b95ec8dd0e37c38096dda9ec06257eb02662f8fd62cbe11b9ad58887d8376b",
|
||||
"timing_probe_guard_sha256": "45e9fe236d31dd7eabbbd46edce4fc7a80914c5a12088d1977de3633d79d6c2a",
|
||||
"probe_guard_wall_seconds": 11.099626450042706,
|
||||
"standardizer_fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03",
|
||||
"engineering_continuation": {
|
||||
"source": "artifacts/s2-pilot-v2-2026-11-15",
|
||||
"source_sha256": {
|
||||
"artifacts/s2-pilot-v2-2026-11-15/pilot-plan.json": "5a127d5ddfab366da49475867c9f14f2854f5000a733c7a3211df61bd7acecae",
|
||||
"artifacts/s2-pilot-v2-2026-11-15/training-contract.json": "0218b74f8d86ae4190e0b814f7c5c8a7224a616b94ad7587120156e755de2101",
|
||||
"artifacts/s2-pilot-v2-2026-11-15/timing-probe/result.json": "96b95ec8dd0e37c38096dda9ec06257eb02662f8fd62cbe11b9ad58887d8376b",
|
||||
"artifacts/s2-pilot-v2-2026-11-15/timing-probe-guard-receipt.json": "45e9fe236d31dd7eabbbd46edce4fc7a80914c5a12088d1977de3633d79d6c2a",
|
||||
"artifacts/s2-pilot-v2-2026-11-15/pilot-complete.json": "337ea9552569db135e22b93f3e0b1557e03c5b56669cdb4d1bbcd391e6f67982",
|
||||
"artifacts/s2-pilot-v2-2026-11-15/orchestrator-complete.json": "e2794244d0664cc10aa3707a8da4f2d078946f985a1095f82a2f12b5310128a9"
|
||||
},
|
||||
"prior_wall_seconds": 11.143491394002922,
|
||||
"probe_reused": true,
|
||||
"probe_rerun": false,
|
||||
"gpu_switched": false,
|
||||
"original_gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c",
|
||||
"selected_gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c",
|
||||
"ruling": "1794695421.177269"
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,11 @@
|
||||
# S2 当前执行接续记录
|
||||
|
||||
## 当前唯一运行:v2工程接续34632,N128 worker已实证启动
|
||||
|
||||
Claude1794695421.177269裁量后的新入口固定ea5ac05。session34632、入口PID209380、N128 worker PID209415实际存活;三次10秒间隔空闲检查共20.285703秒通过,同UUID未切换。根artifacts/s2-pilot-v2-continuation-2026-11-15,actual-start-audit及两级worker.started/进度证明首run实际启动。新probe不重跑,E_max42;预算累计原11.143491秒加全部新等待/setup/run仍≤7200秒。源码pin不改、125测试封存。未完成run不作完成或选宽声明;禁止重复启动。
|
||||
|
||||
旧26792保持零run的PILOT_INCOMPLETE;旧恢复/旧接续/诊断均结束,不重启。Slack发布首次遇连接错误,实际交付状态以之后成功链接为准。
|
||||
|
||||
## 最新终态更正:v2 probe通过,GPU空闲门停止,零训练run
|
||||
|
||||
session26792已结束。v2 probe PASS、187345节点/峰值12586057728B、E_max42均为实测;但probe后首run前GPU空闲门返回SELECTED_GPU_BUSY_BEFORE_RUN,pilot-complete的runs=[],四个训练run均未启动,总wall11.143491秒。此前将pilot-started入口标记误写为N128启动,现明确撤回;不得从该标记推断训练已运行。原始回执不改。
|
||||
|
||||
Reference in new issue
Block a user