diff --git a/AGENTS.md b/AGENTS.md index f8615c53a..9ee964528 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,3 +20,7 @@ - Precision-v2 terminal correction (2026-11-15 06:30 UTC+8): diagnostic41846 completed and reproduced fp16 slice overflow; fixed8a548cc preserves original sources and implements the approved shared FP32 aggregation. New v2 session26792 has already exited: probe PASS at187345 nodes, VRAM12586057728B, E_max42, but SELECTED_GPU_BUSY_BEFORE_RUN stopped before any training run (runs=[]), total11.143491s. Earlier inference from pilot-started to N128 running was incorrect and retracted. Do not restart or rerun probe without the engineering continuation ruling; read latest handoff and raw complete receipts.125 tests remain sealed. - Latest engineering continuation (2026-11-15): Claude1794695421.177269 authorized reuse of v2 probe after its zero-run infrastructure stop. Fixedea5ac05; sole active session34632/root `artifacts/s2-pilot-v2-continuation-2026-11-15`, entryPID209380 and N128 workerPID209415 verified running. Three idle polls passed over20.285703s, sameGPU. ReuseE_max42/probe; original11.143491s plus all new waits/setup/runs share7200s. Keep v2 zero-run stop and original numerical failure immutable. Inspect real worker/start/progress/terminal receipts before any action; never duplicate or modify pinned code.125 tests remain sealed. + +- Latest terminal update (2026-11-15 07:14:59 UTC+8): session34632 and all pilot workers have exited. N128/E128 completed42 epochs with guardPASS; N256 failed at epoch2 on4/382 AMP skips exceeding1%, E256 never started. PILOT_INCOMPLETE/no width; cumulative2439.912972s/7200. N256 history/progress also records reserved16416505856B above15GiB; the failed-result path did not inspect recorded peaks. Preserve this resource-boundary gap. Read `docs/s2-v2-pilot-stop-results-2026-11-15.md`; no restart, reprobe, threshold/source change or full matrix is authorized by a polling request.125 tests sealed; evidence submitted toClaude. + +- Claude1794698179.161599 records a separate feasibility decision: main-matrix width128, width effect not tested, no use of validation scores to justify selection. Original pilot selected_global_width staysnull. Before any main-matrix execution: fix recorded progress/result VRAM-peak enforcement with synthetic test, allocator configuration per ruling, unchangedAMP1% andE_max42, and budget/execution plan. This read-only stage audit has not implemented or launched that work. diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0-guard-receipt.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0-guard-receipt.json new file mode 100644 index 000000000..6c8520abd --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0-guard-receipt.json @@ -0,0 +1,34 @@ +{ + "status": "PASS", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "1" + ], + "returncode": 0, + "stop_reason": null, + "cpu_seconds": 1121.705467, + "wall_seconds": 1121.303854590049, + "peak_tree_rss_bytes": 2574909440, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 32768, + "max_poll_gap_seconds": 0.7979419510229491, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null, + "device_monitor": { + "peak_device_memory_bytes": 8660189184, + "peak_whole_run_tree_rss_bytes": 2981724160, + "max_poll_gap_seconds": 1.105505651968997, + "reason": null, + "samples": 1887 + }, + "total_run_wall_seconds": 1121.3519870990422, + "result_sha256": "2e1271dbcc00ba4ee4889aa2c9e192283c06a9bd3eabc10a973dabba2f7dad78" +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/best-validation-parts.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/best-validation-parts.json new file mode 100644 index 000000000..eaa08a65a --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/best-validation-parts.json @@ -0,0 +1,113 @@ +{ + "00000111_ce24e0cbec8e45c89735d148_step_003": 0.1343679197205029, + "00000115_ce24e0cbec8e45c89735d148_step_007": 0.13758672155870916, + "00000116_ce24e0cbec8e45c89735d148_step_008": 0.08463958347693151, + "00000117_ce24e0cbec8e45c89735d148_step_009": 0.15342898660856275, + "00000119_ce24e0cbec8e45c89735d148_step_011": 0.12557230144132203, + "00000120_ce24e0cbec8e45c89735d148_step_012": 0.1242337995576084, + "00000131_945244874ed84e599987daa9_step_000": 0.06182399460153422, + "00000132_438507a38e404bfdbd51a16b_step_000": 0.06756212086857138, + "00000133_08b739e0511f4305b161cd05_step_000": 0.05867959001176893, + "00000190_185ecbc783f34408aef52119_step_000": 0.06396081027660315, + "00000196_c9b71ecdd0654716891b0db8_step_000": 0.09056029498274536, + "00000206_27417003e762409185cb2b69_step_000": 0.054572541748064585, + "00000238_4829bc0645d6414d9b5cec31_step_001": 0.059581047385305355, + "00000241_4829bc0645d6414d9b5cec31_step_004": 0.212148626409705, + "00000243_4829bc0645d6414d9b5cec31_step_006": 0.098546246279916, + "00000248_e87554b09cdc48c3add39db9_step_000": 0.16216205679089105, + "00000257_845fb06d7f2f4802a806a600_step_002": 0.053997440018342856, + "00000258_845fb06d7f2f4802a806a600_step_003": 0.0846846326339309, + "00000259_845fb06d7f2f4802a806a600_step_004": 0.06713926921440924, + "00000261_845fb06d7f2f4802a806a600_step_006": 0.0940587224121968, + "00000262_845fb06d7f2f4802a806a600_step_007": 0.06757211385601955, + "00000264_845fb06d7f2f4802a806a600_step_009": 0.21263708843450632, + "00000265_845fb06d7f2f4802a806a600_step_010": 0.11556758153049056, + "00000266_845fb06d7f2f4802a806a600_step_011": 0.13911023743631398, + "00000267_845fb06d7f2f4802a806a600_step_012": 0.042584757370485136, + "00000269_845fb06d7f2f4802a806a600_step_014": 0.07822614441393157, + "00000391_fd490b564a44413884fab719_step_000": 0.05227786078318484, + "00000405_f3759bbb4682423caf207e8e_step_002": 0.11258255664186091, + "00000406_f3759bbb4682423caf207e8e_step_003": 0.06229348384702075, + "00000444_4ed4c78d6d754aac90876fc2_step_003": 0.06921304184598819, + "00000445_4ed4c78d6d754aac90876fc2_step_004": 0.08453830894144249, + "00000501_5499914343bd4f549b38046d_step_002": 0.047228706731313325, + "00000502_5499914343bd4f549b38046d_step_003": 0.06684364586460824, + "00000574_5a4cef68211d4706b1ec8586_step_003": 0.18500320265231116, + "00000578_5a4cef68211d4706b1ec8586_step_007": 0.1265959984438593, + "00000579_5a4cef68211d4706b1ec8586_step_008": 0.1284793581536923, + "00000580_5a4cef68211d4706b1ec8586_step_009": 0.14365772456255563, + "00000582_5a4cef68211d4706b1ec8586_step_011": 0.14166351727500343, + "00000583_5a4cef68211d4706b1ec8586_step_012": 0.1870464258759157, + "00000597_7fe894812c4244e18241aa08_step_003": 0.20977868673408676, + "00000601_7fe894812c4244e18241aa08_step_007": 0.1574996473849678, + "00000602_7fe894812c4244e18241aa08_step_008": 0.10666808275561115, + "00000603_7fe894812c4244e18241aa08_step_009": 0.16224906566189298, + "00000605_7fe894812c4244e18241aa08_step_011": 0.13997595089992187, + "00000606_7fe894812c4244e18241aa08_step_012": 0.18282196304827272, + "00000654_d87d11d9c3ca4ee5a88797ae_step_002": 0.10240866266514304, + "00000758_8ab50540e9804365a8717ec0_step_005": 0.11114758277527761, + "00000759_8ab50540e9804365a8717ec0_step_006": 0.06650833147278977, + "00000760_8ab50540e9804365a8717ec0_step_007": 0.09889873992289532, + "00000763_8ab50540e9804365a8717ec0_step_010": 0.10492979954060652, + "00000816_36e6258346004cabb4c091b2_step_004": 0.1231060912693525, + "00000817_36e6258346004cabb4c091b2_step_005": 0.16188817856025678, + "00000892_be26dc83f78947738b2bfabd_step_001": 0.12596579472072753, + "00000894_be26dc83f78947738b2bfabd_step_003": 0.21686587723250417, + "00000897_be26dc83f78947738b2bfabd_step_006": 0.12184301048995777, + "00000919_cdcebf32644443b9812097ba_step_003": 0.11896299877772587, + "00000923_cdcebf32644443b9812097ba_step_007": 0.14245565110609912, + "00000927_cdcebf32644443b9812097ba_step_011": 0.13109823780678218, + "00000928_cdcebf32644443b9812097ba_step_012": 0.17212010304533154, + "00000955_a741c7966d2c40a8a511263e_step_001": 0.06590048471438022, + "00000984_0dd1386597104265ab67e6cf_step_007": 0.10706089848001821, + "00000986_0dd1386597104265ab67e6cf_step_009": 0.08866535815879016, + "00000987_0dd1386597104265ab67e6cf_step_010": 0.15629526732098517, + "00000988_0dd1386597104265ab67e6cf_step_011": 0.15724320960318264, + "00000996_6d71695ba3fd475c927a00a7_step_000": 0.15057582617610374, + "00001019_ca4cf9693d6d4a98b515ba99_step_000": 0.04774444653645815, + "00001020_ca4cf9693d6d4a98b515ba99_step_001": 0.10910534722138153, + "00001021_ca4cf9693d6d4a98b515ba99_step_002": 0.05755607974166003, + "00001026_0f133d8e7fd448708a554e28_step_000": 0.045323071080638856, + "00001027_0f133d8e7fd448708a554e28_step_001": 0.30053344994439735, + "00001040_74c42207fa764831b03f7658_step_000": 0.049824264422328464, + "00001052_2fd6155fa34f418083d0bd0f_step_001": 0.05563831053172737, + "00001053_2fd6155fa34f418083d0bd0f_step_002": 0.05851200291575983, + "00001097_256ee7ec3a4b4972a8b87c8d_step_001": 0.16477683883506955, + "00001099_256ee7ec3a4b4972a8b87c8d_step_003": 0.1321636986325246, + "00001102_256ee7ec3a4b4972a8b87c8d_step_006": 0.16334411367030505, + "00001107_78566795c3144719804f8d34_step_000": 0.08398352807365266, + "00001167_107f61d0402f4c9a9d9d1342_step_000": 0.1694342606644836, + "00001168_107f61d0402f4c9a9d9d1342_step_001": 0.055258966951879, + "00001217_e517c61f1c49433bb81cf980_step_000": 0.15348692419370372, + "00001238_96503b6a26c64074ad598927_step_000": 0.07467456738745658, + "00001239_96503b6a26c64074ad598927_step_001": 0.08826290594772515, + "00001240_96503b6a26c64074ad598927_step_002": 0.06823067580490266, + "00001241_96503b6a26c64074ad598927_step_003": 0.20754891609907897, + "00001242_96503b6a26c64074ad598927_step_004": 0.03597726445900491, + "00001243_96503b6a26c64074ad598927_step_005": 0.036696770700500876, + "00001244_96503b6a26c64074ad598927_step_006": 0.07718243020201285, + "00001246_96503b6a26c64074ad598927_step_008": 0.1565289029376164, + "00001247_96503b6a26c64074ad598927_step_009": 0.09810734759150445, + "00001249_96503b6a26c64074ad598927_step_011": 0.17977696732976636, + "00001252_96503b6a26c64074ad598927_step_014": 0.0723641543257618, + "00001253_96503b6a26c64074ad598927_step_015": 0.13621596278429407, + "00001314_a9ed5e56a6a8414f91b39bdb_step_000": 0.09045590960846321, + "00001333_beab79941e884455a62af40c_step_001": 0.1526556149625648, + "00001334_beab79941e884455a62af40c_step_002": 0.20422838266087354, + "00001335_beab79941e884455a62af40c_step_003": 0.2800919958389591, + "00001336_beab79941e884455a62af40c_step_004": 0.25904183073111303, + "00001338_beab79941e884455a62af40c_step_006": 0.2080707126468865, + "00001339_beab79941e884455a62af40c_step_007": 0.1379616255518867, + "00001340_beab79941e884455a62af40c_step_008": 0.166847231292684, + "00001352_4e1d56bc2cef49ea84626b11_step_000": 0.07386029287122761, + "00001353_4e1d56bc2cef49ea84626b11_step_001": 0.09682523022231708, + "00001354_4e1d56bc2cef49ea84626b11_step_002": 0.07977990999653213, + "00001379_5b1d6e2599ea47d498a04256_step_000": 0.08625382360009703, + "00001387_60cf3ad636dd42e0bcec6946_step_000": 0.11734760544918385, + "00001511_5cbe58077e4b4885a5c213b3_step_004": 0.09860342187514147, + "00001512_5cbe58077e4b4885a5c213b3_step_005": 0.1057075930507414, + "00001525_e0d35a03b46745259cf8878f_step_000": 0.18077717517231642, + "00001532_9f0a4a641e8f4cc588eb4315_step_000": 0.09778661021356745, + "00001552_b0a9c4d836294846891615fe_step_000": 0.13618314127390238, + "00001553_adfa1c33ae4d4d0aa441ddc3_step_000": 0.1063776989722833 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/best.pt b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/best.pt new file mode 100644 index 000000000..26a291486 Binary files /dev/null and b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/best.pt differ diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-000-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-000-amp.json new file mode 100644 index 000000000..0d8e3bf8a --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-000-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 2, + "successful_updates": 380, + "skipped_fraction": 0.005235602094240838, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-001-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-001-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-001-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-002-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-002-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-002-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-003-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-003-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-003-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-004-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-004-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-004-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-005-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-005-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-005-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-006-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-006-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-006-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-007-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-007-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-007-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-008-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-008-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-008-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-009-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-009-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-009-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-010-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-010-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-010-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-011-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-011-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-011-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-012-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-012-amp.json new file mode 100644 index 000000000..f66f9a3d7 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-012-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-013-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-013-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-013-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-014-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-014-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-014-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-015-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-015-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-015-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-016-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-016-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-016-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-017-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-017-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-017-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-018-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-018-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-018-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-019-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-019-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-019-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-020-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-020-amp.json new file mode 100644 index 000000000..f66f9a3d7 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-020-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-021-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-021-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-021-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-022-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-022-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-022-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-023-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-023-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-023-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-024-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-024-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-024-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-025-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-025-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-025-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-026-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-026-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-026-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-027-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-027-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-027-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-028-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-028-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-028-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-029-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-029-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-029-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-030-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-030-amp.json new file mode 100644 index 000000000..f66f9a3d7 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-030-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-031-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-031-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-031-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-032-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-032-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-032-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-033-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-033-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-033-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-034-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-034-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-034-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-035-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-035-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-035-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-036-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-036-amp.json new file mode 100644 index 000000000..f66f9a3d7 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-036-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-037-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-037-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-037-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-038-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-038-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-038-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-039-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-039-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-039-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-040-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-040-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-040-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-041-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-041-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-041-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/history.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/history.json new file mode 100644 index 000000000..3489d6f72 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/history.json @@ -0,0 +1,716 @@ +[ + { + "epoch": 0, + "amp": { + "attempted_steps": 382, + "skipped_steps": 2, + "successful_updates": 380, + "skipped_fraction": 0.005235602094240838, + "consecutive_skipped": 0 + }, + "attempted_updates": 382, + "training_global_weighted_mse": 0.03777154701891729, + "validation_component_equal_relative_l2": 0.1583400171910542, + "updates": 380, + "elapsed_seconds": 33.279333626967855, + "peak_allocated_bytes": 4704985600, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 1, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 764, + "training_global_weighted_mse": 0.012727464129142607, + "validation_component_equal_relative_l2": 0.15742186052662527, + "updates": 762, + "elapsed_seconds": 58.68679470900679, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 2, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 1146, + "training_global_weighted_mse": 0.013003694076250736, + "validation_component_equal_relative_l2": 0.18959317082673424, + "updates": 1144, + "elapsed_seconds": 85.53299703699304, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 3, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 1528, + "training_global_weighted_mse": 0.012685709300448541, + "validation_component_equal_relative_l2": 0.17380365130863631, + "updates": 1526, + "elapsed_seconds": 112.21489061199827, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 4, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 1910, + "training_global_weighted_mse": 0.00975527722892566, + "validation_component_equal_relative_l2": 0.1370594002706418, + "updates": 1908, + "elapsed_seconds": 138.5898038489977, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 5, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 2292, + "training_global_weighted_mse": 0.00827810675071029, + "validation_component_equal_relative_l2": 0.1237315203605647, + "updates": 2290, + "elapsed_seconds": 165.71112991397968, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 6, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 2674, + "training_global_weighted_mse": 0.009841250099289885, + "validation_component_equal_relative_l2": 0.12105059629730207, + "updates": 2672, + "elapsed_seconds": 192.15261031896807, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 7, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 3056, + "training_global_weighted_mse": 0.007885463235147676, + "validation_component_equal_relative_l2": 0.1503962452531702, + "updates": 3054, + "elapsed_seconds": 218.62901735899504, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 8, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 3438, + "training_global_weighted_mse": 0.008254757676095617, + "validation_component_equal_relative_l2": 0.12984515420638335, + "updates": 3436, + "elapsed_seconds": 245.21401930297725, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 9, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 3820, + "training_global_weighted_mse": 0.0077437261751921805, + "validation_component_equal_relative_l2": 0.11645651227094547, + "updates": 3818, + "elapsed_seconds": 272.2327727829688, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 10, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 4202, + "training_global_weighted_mse": 0.007559133971814574, + "validation_component_equal_relative_l2": 0.11790679809707552, + "updates": 4200, + "elapsed_seconds": 298.9363094990258, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 11, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 4584, + "training_global_weighted_mse": 0.007314151204014396, + "validation_component_equal_relative_l2": 0.136636725191453, + "updates": 4582, + "elapsed_seconds": 325.57086828298634, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 12, + "amp": { + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 + }, + "attempted_updates": 4966, + "training_global_weighted_mse": 0.007513021904285222, + "validation_component_equal_relative_l2": 0.12895678195464605, + "updates": 4963, + "elapsed_seconds": 352.0781779229874, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 13, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 5348, + "training_global_weighted_mse": 0.007109822103769224, + "validation_component_equal_relative_l2": 0.1145335538377817, + "updates": 5345, + "elapsed_seconds": 379.0714644350228, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 14, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 5730, + "training_global_weighted_mse": 0.007158998076716597, + "validation_component_equal_relative_l2": 0.11095412889910025, + "updates": 5727, + "elapsed_seconds": 407.0874903440126, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 15, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 6112, + "training_global_weighted_mse": 0.006788285161946889, + "validation_component_equal_relative_l2": 0.11616125471010663, + "updates": 6109, + "elapsed_seconds": 433.32142731698696, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 16, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 6494, + "training_global_weighted_mse": 0.006726070689392478, + "validation_component_equal_relative_l2": 0.11081830780756512, + "updates": 6491, + "elapsed_seconds": 459.18691649497487, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 17, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 6876, + "training_global_weighted_mse": 0.006581763330992846, + "validation_component_equal_relative_l2": 0.1387859039638887, + "updates": 6873, + "elapsed_seconds": 484.80278230499243, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 18, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 7258, + "training_global_weighted_mse": 0.006622952473743685, + "validation_component_equal_relative_l2": 0.10736822987841725, + "updates": 7255, + "elapsed_seconds": 510.5525389779941, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 19, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 7640, + "training_global_weighted_mse": 0.006378984562520864, + "validation_component_equal_relative_l2": 0.1285029350341078, + "updates": 7637, + "elapsed_seconds": 537.6905506449984, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 20, + "amp": { + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 + }, + "attempted_updates": 8022, + "training_global_weighted_mse": 0.006323783354826031, + "validation_component_equal_relative_l2": 0.13450906937607846, + "updates": 8018, + "elapsed_seconds": 565.9920399599941, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 21, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 8404, + "training_global_weighted_mse": 0.006506983306171227, + "validation_component_equal_relative_l2": 0.11061497313039317, + "updates": 8400, + "elapsed_seconds": 591.672262986016, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 22, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 8786, + "training_global_weighted_mse": 0.006188833143557348, + "validation_component_equal_relative_l2": 0.11386371580205701, + "updates": 8782, + "elapsed_seconds": 618.5636462389957, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 23, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 9168, + "training_global_weighted_mse": 0.005998109882114321, + "validation_component_equal_relative_l2": 0.12015199454742402, + "updates": 9164, + "elapsed_seconds": 644.4918569899746, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 24, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 9550, + "training_global_weighted_mse": 0.006330224495605563, + "validation_component_equal_relative_l2": 0.11149255834059588, + "updates": 9546, + "elapsed_seconds": 670.9965526600135, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 25, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 9932, + "training_global_weighted_mse": 0.006017579651902518, + "validation_component_equal_relative_l2": 0.12460129235532239, + "updates": 9928, + "elapsed_seconds": 696.7369387699873, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 26, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 10314, + "training_global_weighted_mse": 0.005725823295106429, + "validation_component_equal_relative_l2": 0.10854158254618171, + "updates": 10310, + "elapsed_seconds": 723.6213204860105, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 27, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 10696, + "training_global_weighted_mse": 0.0057398792949967005, + "validation_component_equal_relative_l2": 0.11156128556087039, + "updates": 10692, + "elapsed_seconds": 750.1456139740185, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 28, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 11078, + "training_global_weighted_mse": 0.00560497407924036, + "validation_component_equal_relative_l2": 0.10841698191989019, + "updates": 11074, + "elapsed_seconds": 775.6729469029815, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 29, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 11460, + "training_global_weighted_mse": 0.0055617517901970625, + "validation_component_equal_relative_l2": 0.10689022970683579, + "updates": 11456, + "elapsed_seconds": 802.2463578279712, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 30, + "amp": { + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 + }, + "attempted_updates": 11842, + "training_global_weighted_mse": 0.005338570249425548, + "validation_component_equal_relative_l2": 0.10925306274794101, + "updates": 11837, + "elapsed_seconds": 828.243478601973, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 31, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 12224, + "training_global_weighted_mse": 0.005402832033942866, + "validation_component_equal_relative_l2": 0.11322502131082494, + "updates": 12219, + "elapsed_seconds": 855.0242627999978, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 32, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 12606, + "training_global_weighted_mse": 0.005210265068030128, + "validation_component_equal_relative_l2": 0.10769285384674371, + "updates": 12601, + "elapsed_seconds": 881.5072197879781, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 33, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 12988, + "training_global_weighted_mse": 0.005150678228065862, + "validation_component_equal_relative_l2": 0.10886442516513213, + "updates": 12983, + "elapsed_seconds": 907.640579976025, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 34, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 13370, + "training_global_weighted_mse": 0.005059083120523179, + "validation_component_equal_relative_l2": 0.11003817021855752, + "updates": 13365, + "elapsed_seconds": 934.9226679850253, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 35, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 13752, + "training_global_weighted_mse": 0.004975307062041979, + "validation_component_equal_relative_l2": 0.10624394418905878, + "updates": 13747, + "elapsed_seconds": 961.0181614760077, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 36, + "amp": { + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 + }, + "attempted_updates": 14134, + "training_global_weighted_mse": 0.004918266695160546, + "validation_component_equal_relative_l2": 0.10488759627346131, + "updates": 14128, + "elapsed_seconds": 986.9381958320155, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 37, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 14516, + "training_global_weighted_mse": 0.004851824022621996, + "validation_component_equal_relative_l2": 0.1057088498478092, + "updates": 14510, + "elapsed_seconds": 1013.6811215750058, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 38, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 14898, + "training_global_weighted_mse": 0.004790806054126673, + "validation_component_equal_relative_l2": 0.105547625627317, + "updates": 14892, + "elapsed_seconds": 1040.6714822470094, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 39, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 15280, + "training_global_weighted_mse": 0.004739985027523355, + "validation_component_equal_relative_l2": 0.10435484679758687, + "updates": 15274, + "elapsed_seconds": 1065.8401368879713, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 40, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 15662, + "training_global_weighted_mse": 0.004715351346469242, + "validation_component_equal_relative_l2": 0.104591054634982, + "updates": 15656, + "elapsed_seconds": 1091.7899562880048, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 41, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 16044, + "training_global_weighted_mse": 0.004690328458595227, + "validation_component_equal_relative_l2": 0.10452529978657271, + "updates": 16038, + "elapsed_seconds": 1117.918959765986, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512 + } +] diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/progress.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/progress.json new file mode 100644 index 000000000..c1e91c235 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/progress.json @@ -0,0 +1,19 @@ +{ + "stage": "epoch_complete", + "epoch": 41, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 16044, + "training_global_weighted_mse": 0.004690328458595227, + "validation_component_equal_relative_l2": 0.10452529978657271, + "updates": 16038, + "elapsed_seconds": 1117.918959765986, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512, + "patience_count": 2 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/result.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/result.json new file mode 100644 index 000000000..081c9efdb --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/result.json @@ -0,0 +1,210 @@ +{ + "status": "COMPLETE", + "run": { + "name": "E128-seed0", + "arm": "E", + "width": 128, + "seed": 0 + }, + "E_max": 42, + "cosine_T_max": 42, + "epochs": 42, + "attempted_updates": 16044, + "updates": 16038, + "best_epoch": 39, + "best_validation_score": 0.10435484679758687, + "learning_parts": 382, + "validation_parts": 111, + "learning_components": 137, + "validation_components": 35, + "failed_preparation_parts_dropped": [ + { + "case": "00000575_5a4cef68211d4706b1ec8586_step_004", + "role": "validation", + "nodes": 75046, + "component_id": "component-3a3e340f4dfac391", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-inputs-2026-11-15", + "181" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.766116, + "wall_seconds": 1800.400393687014, + "peak_tree_rss_bytes": 602218496, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.128913332009688, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false + }, + "failure": null + }, + { + "case": "00000614_c710d414bbbc4340b682c58a_step_000", + "role": "learning", + "nodes": 145860, + "component_id": "component-4bd089778094e746", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-inputs-2026-11-15", + "201" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.796751, + "wall_seconds": 1800.0567526929663, + "peak_tree_rss_bytes": 629305344, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.1576595069491304, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false + }, + "failure": null + }, + { + "case": "00000625_6af98f5458b347bb843f0562_step_000", + "role": "learning", + "nodes": 145707, + "component_id": "component-4bd089778094e746", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-inputs-2026-11-15", + "211" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.8864629999998, + "wall_seconds": 1800.119237862993, + "peak_tree_rss_bytes": 647880704, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.159920622012578, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false + }, + "failure": null + }, + { + "case": "00001466_a45f6c99e2154f3b94955175_step_050", + "role": "learning", + "nodes": 77138, + "component_id": "component-7fe5173e77c86f9d", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-recovery-inputs-2026-11-15", + "83" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.926194, + "wall_seconds": 1800.0714844280155, + "peak_tree_rss_bytes": 383070208, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.0109735240112059, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null + }, + "failure": null + }, + { + "case": "00001470_a45f6c99e2154f3b94955175_step_054", + "role": "learning", + "nodes": 124817, + "component_id": "component-7fe5173e77c86f9d", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-recovery-inputs-2026-11-15", + "87" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.664371, + "wall_seconds": 1800.0443400070071, + "peak_tree_rss_bytes": 578732032, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.007104828953743, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null + }, + "failure": null + }, + { + "case": "00001473_a45f6c99e2154f3b94955175_step_057", + "role": "learning", + "nodes": 69428, + "component_id": "component-7fe5173e77c86f9d", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-recovery-inputs-2026-11-15", + "90" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.751432, + "wall_seconds": 1800.2498029720155, + "peak_tree_rss_bytes": 438853632, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 0.8929723129840568, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null + }, + "failure": null + } + ], + "standardizer_fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03", + "wall_seconds": 1117.921224004007, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512, + "test_labels_read": false, + "test_predictions": false, + "capacity_selected": false, + "checkpoint_sha256": "910437bd645a965241786a90ad14325fe7af09092238bfd0be5ee5bddf8a9a24" +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/standardizer.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/standardizer.json new file mode 100644 index 000000000..11716fd2e --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/standardizer.json @@ -0,0 +1,429 @@ +{ + "fold": "0", + "pool": "100", + "learning_part_ids": [ + "00000003_1ffb81a71e5b402e966b9341_step_002", + "00000004_1ffb81a71e5b402e966b9341_step_003", + "00000021_ad34a3f60c4a4caa99646600_step_001", + "00000023_ad34a3f60c4a4caa99646600_step_003", + "00000024_ad34a3f60c4a4caa99646600_step_004", + "00000030_ad34a3f60c4a4caa99646600_step_010", + "00000031_ad34a3f60c4a4caa99646600_step_011", + "00000033_ad34a3f60c4a4caa99646600_step_013", + "00000049_aa208c45e8734584a147f595_step_000", + "00000057_666139e3bff64d4e8a6ce183_step_006", + "00000058_666139e3bff64d4e8a6ce183_step_007", + "00000060_767e4372b5f94a88a7a17d90_step_001", + "00000061_767e4372b5f94a88a7a17d90_step_002", + "00000065_767e4372b5f94a88a7a17d90_step_006", + "00000066_767e4372b5f94a88a7a17d90_step_007", + "00000067_767e4372b5f94a88a7a17d90_step_008", + "00000068_767e4372b5f94a88a7a17d90_step_009", + "00000070_767e4372b5f94a88a7a17d90_step_011", + "00000072_767e4372b5f94a88a7a17d90_step_013", + "00000077_767e4372b5f94a88a7a17d90_step_018", + "00000078_767e4372b5f94a88a7a17d90_step_019", + "00000079_767e4372b5f94a88a7a17d90_step_020", + "00000087_b4f9335a60a14cecaa599159_step_002", + "00000088_169bec34317b46e98eb1a403_step_000", + "00000093_169bec34317b46e98eb1a403_step_005", + "00000098_72cf0f9ca5324822a4d41a80_step_000", + "00000103_52f7124261ae4c558f800775_step_000", + "00000104_52f7124261ae4c558f800775_step_001", + "00000105_52f7124261ae4c558f800775_step_002", + "00000106_52f7124261ae4c558f800775_step_003", + "00000125_e18c7feacfb54f3880659b43_step_000", + "00000128_e18c7feacfb54f3880659b43_step_003", + "00000129_e18c7feacfb54f3880659b43_step_004", + "00000130_e18c7feacfb54f3880659b43_step_005", + "00000140_2fc54fcd110d4f49969163c4_step_003", + "00000142_2fc54fcd110d4f49969163c4_step_005", + "00000144_2fc54fcd110d4f49969163c4_step_007", + "00000145_2fc54fcd110d4f49969163c4_step_008", + "00000152_a951550b8a1043a68423c541_step_002", + "00000153_a951550b8a1043a68423c541_step_003", + "00000154_a951550b8a1043a68423c541_step_004", + "00000157_8208fd8b44b64dce91322f84_step_000", + "00000161_a583c6c3cdc143bab3f6c856_step_000", + "00000162_4dcda9fab6dd46cf98fb194e_step_000", + "00000163_ff9b2ed255484db78bb0e6da_step_000", + "00000164_91adbf180e114b698160c9a4_step_000", + "00000165_4070a99d6ebe45f5bd5d71eb_step_000", + "00000166_2246cab58dbe4a2fb00fed50_step_000", + "00000170_b21aa794b6b64d87a57c245e_step_000", + "00000171_b21aa794b6b64d87a57c245e_step_001", + "00000173_b21aa794b6b64d87a57c245e_step_003", + "00000174_b21aa794b6b64d87a57c245e_step_004", + "00000175_b21aa794b6b64d87a57c245e_step_005", + "00000176_b21aa794b6b64d87a57c245e_step_006", + "00000181_b21aa794b6b64d87a57c245e_step_011", + "00000198_826494e894454504b0971fd1_step_001", + "00000223_97b354a907114b7183faa9c4_step_001", + "00000224_97b354a907114b7183faa9c4_step_002", + "00000229_682d529d5e8c4387bc59b3a1_step_000", + "00000250_641f03cbe6a145508c6ae055_step_000", + "00000272_648a7c23a21f4e88bf25ffeb_step_000", + "00000273_648a7c23a21f4e88bf25ffeb_step_001", + "00000274_648a7c23a21f4e88bf25ffeb_step_002", + "00000275_648a7c23a21f4e88bf25ffeb_step_003", + "00000276_648a7c23a21f4e88bf25ffeb_step_004", + "00000280_5a6ef3285b89452ea6b390e7_step_001", + "00000282_5a6ef3285b89452ea6b390e7_step_003", + "00000285_d215b7bc60ae468288e5f1ea_step_000", + "00000287_58f8ff46b7564e989f9695b7_step_000", + "00000290_58f8ff46b7564e989f9695b7_step_003", + "00000291_58f8ff46b7564e989f9695b7_step_004", + "00000293_58f8ff46b7564e989f9695b7_step_006", + "00000295_52a153d2417747b5926ff9df_step_000", + "00000296_52a153d2417747b5926ff9df_step_001", + "00000301_52a153d2417747b5926ff9df_step_006", + "00000302_39b08d62a677414f86fdae52_step_000", + "00000305_3062bccff48e47a2b9de05e3_step_000", + "00000306_3062bccff48e47a2b9de05e3_step_001", + "00000307_3062bccff48e47a2b9de05e3_step_002", + "00000308_3062bccff48e47a2b9de05e3_step_003", + "00000309_3062bccff48e47a2b9de05e3_step_004", + "00000310_3062bccff48e47a2b9de05e3_step_005", + "00000311_3062bccff48e47a2b9de05e3_step_006", + "00000312_3062bccff48e47a2b9de05e3_step_007", + "00000314_3062bccff48e47a2b9de05e3_step_009", + "00000315_3062bccff48e47a2b9de05e3_step_010", + "00000316_3062bccff48e47a2b9de05e3_step_011", + "00000317_3062bccff48e47a2b9de05e3_step_012", + "00000318_3062bccff48e47a2b9de05e3_step_013", + "00000319_3062bccff48e47a2b9de05e3_step_014", + "00000321_3062bccff48e47a2b9de05e3_step_016", + "00000322_3062bccff48e47a2b9de05e3_step_017", + "00000323_3062bccff48e47a2b9de05e3_step_018", + "00000324_3062bccff48e47a2b9de05e3_step_019", + "00000325_3062bccff48e47a2b9de05e3_step_020", + "00000329_12a935cd9b684f879c0505fe_step_000", + "00000330_12a935cd9b684f879c0505fe_step_001", + "00000331_12a935cd9b684f879c0505fe_step_002", + "00000336_1a821c3883f7459a9615fca4_step_000", + "00000337_153efbd85ca64c4fb6b7ed5f_step_000", + "00000346_4185972a944744d8a7a0f2b4_step_002", + "00000348_4185972a944744d8a7a0f2b4_step_004", + "00000352_4185972a944744d8a7a0f2b4_step_008", + "00000355_68d1a5c2319a4a419ff625c6_step_001", + "00000356_68d1a5c2319a4a419ff625c6_step_002", + "00000357_68d1a5c2319a4a419ff625c6_step_003", + "00000364_38d899e2706a4d09960705dd_step_001", + "00000367_23c957e8e7b2428282a13ae2_step_000", + "00000368_23c957e8e7b2428282a13ae2_step_001", + "00000369_23c957e8e7b2428282a13ae2_step_002", + "00000374_23c957e8e7b2428282a13ae2_step_007", + "00000375_23c957e8e7b2428282a13ae2_step_008", + "00000376_23c957e8e7b2428282a13ae2_step_009", + "00000377_23c957e8e7b2428282a13ae2_step_010", + "00000378_23c957e8e7b2428282a13ae2_step_011", + "00000379_239bbf0e92974a32bd214366_step_000", + "00000382_239bbf0e92974a32bd214366_step_003", + "00000383_239bbf0e92974a32bd214366_step_004", + "00000386_239bbf0e92974a32bd214366_step_007", + "00000387_239bbf0e92974a32bd214366_step_008", + "00000388_239bbf0e92974a32bd214366_step_009", + "00000418_ce5b31679ee845a69ba89d6f_step_000", + "00000422_8e8a96d437584c77b3cfd84f_step_000", + "00000423_8e8a96d437584c77b3cfd84f_step_001", + "00000424_8e8a96d437584c77b3cfd84f_step_002", + "00000459_516b1784884d4facbc4c0d74_step_007", + "00000462_516b1784884d4facbc4c0d74_step_010", + "00000466_11951e51b89249dba38ded3e_step_001", + "00000469_11951e51b89249dba38ded3e_step_004", + "00000470_11951e51b89249dba38ded3e_step_005", + "00000474_a5639f2b84c445aea5da18ad_step_000", + "00000477_0fc8cc17c6af48dabe36c8f2_step_002", + "00000478_0fc8cc17c6af48dabe36c8f2_step_003", + "00000479_0fc8cc17c6af48dabe36c8f2_step_004", + "00000485_a7d0c851b5f24a97a7cac950_step_002", + "00000488_a7d0c851b5f24a97a7cac950_step_005", + "00000489_a7d0c851b5f24a97a7cac950_step_006", + "00000492_a7d0c851b5f24a97a7cac950_step_009", + "00000513_f78bf7cf156f4231988af062_step_001", + "00000519_5b8fa6b193d14279a77a0e50_step_000", + "00000523_1f8d7f343e634c11a80d23ac_step_000", + "00000526_f9d6559308dc4c85aab9fa38_step_000", + "00000530_bc0d6937206b43bc96521413_step_002", + "00000531_bc0d6937206b43bc96521413_step_003", + "00000543_8cf6554d440d40c4823dad6c_step_000", + "00000559_a1621e156ade4317ab6a5bc5_step_000", + "00000565_5b5119548f7b421e8f597438_step_000", + "00000587_2417b410de78416ebe50d460_step_000", + "00000589_2417b410de78416ebe50d460_step_002", + "00000590_2417b410de78416ebe50d460_step_003", + "00000591_2417b410de78416ebe50d460_step_004", + "00000609_04e799936ae446b9a3d8d2b8_step_000", + "00000610_04e799936ae446b9a3d8d2b8_step_001", + "00000611_04e799936ae446b9a3d8d2b8_step_002", + "00000613_189d1310aa8c4a2a92a21020_step_001", + "00000615_c710d414bbbc4340b682c58a_step_001", + "00000616_c710d414bbbc4340b682c58a_step_002", + "00000617_c710d414bbbc4340b682c58a_step_003", + "00000618_c710d414bbbc4340b682c58a_step_004", + "00000619_c710d414bbbc4340b682c58a_step_005", + "00000620_c710d414bbbc4340b682c58a_step_006", + "00000621_c710d414bbbc4340b682c58a_step_007", + "00000622_c710d414bbbc4340b682c58a_step_008", + "00000624_1eacb269d52a47f5b265c1ab_step_000", + "00000626_f686126d229f47aeb125b5ba_step_000", + "00000627_53bfa7e2e04c4d9dbf4a3db2_step_000", + "00000632_4f322b34e87c43289916c298_step_001", + "00000644_761a451dfea94bfa93e23cfe_step_000", + "00000658_a40da330321b44f286a27361_step_000", + "00000666_6f67a59c16b949a4b66dcfe1_step_000", + "00000683_8863c3497d6d4e05b0f41ddc_step_000", + "00000685_24615f5ff44d4cd5809e77a5_step_001", + "00000686_24615f5ff44d4cd5809e77a5_step_002", + "00000688_b9f3563b403143dbb86ab2b8_step_001", + "00000691_b9f3563b403143dbb86ab2b8_step_004", + "00000693_b9f3563b403143dbb86ab2b8_step_006", + "00000694_45f6100bfa3948e69405bb21_step_000", + "00000696_32309198fce9481896bd0a99_step_000", + "00000698_6f0fc559e8f3422c908b6060_step_001", + "00000699_cae88f0aaaf34de597863810_step_000", + "00000708_c08c1c78b86f4ee3bcb7d9de_step_004", + "00000709_c08c1c78b86f4ee3bcb7d9de_step_005", + "00000710_c08c1c78b86f4ee3bcb7d9de_step_006", + "00000721_9a461e8ae7dc4d33a55883eb_step_000", + "00000723_8ed79b6bcb024d1a9f2aa5e7_step_000", + "00000737_989e0f807cde438face0cb00_step_002", + "00000738_989e0f807cde438face0cb00_step_003", + "00000751_6f0d90054d40400593050930_step_000", + "00000752_6f0d90054d40400593050930_step_001", + "00000764_9cab8335065e4d439bbff471_step_000", + "00000765_9cab8335065e4d439bbff471_step_001", + "00000766_9cab8335065e4d439bbff471_step_002", + "00000767_9cab8335065e4d439bbff471_step_003", + "00000775_a25fb37392a44447b9684f16_step_000", + "00000776_a25fb37392a44447b9684f16_step_001", + "00000777_a25fb37392a44447b9684f16_step_002", + "00000778_a25fb37392a44447b9684f16_step_003", + "00000779_a25fb37392a44447b9684f16_step_004", + "00000780_a25fb37392a44447b9684f16_step_005", + "00000781_a25fb37392a44447b9684f16_step_006", + "00000782_a25fb37392a44447b9684f16_step_007", + "00000783_a25fb37392a44447b9684f16_step_008", + "00000784_a25fb37392a44447b9684f16_step_009", + "00000785_a25fb37392a44447b9684f16_step_010", + "00000786_a25fb37392a44447b9684f16_step_011", + "00000787_a25fb37392a44447b9684f16_step_012", + "00000788_a25fb37392a44447b9684f16_step_013", + "00000789_a25fb37392a44447b9684f16_step_014", + "00000790_a25fb37392a44447b9684f16_step_015", + "00000791_a25fb37392a44447b9684f16_step_016", + "00000792_a25fb37392a44447b9684f16_step_017", + "00000793_a25fb37392a44447b9684f16_step_018", + "00000797_e7533c87b9a347cdafd0c390_step_000", + "00000799_ad6107d9fc994ccca38cc4e4_step_001", + "00000800_ad6107d9fc994ccca38cc4e4_step_002", + "00000801_ad6107d9fc994ccca38cc4e4_step_003", + "00000802_ad6107d9fc994ccca38cc4e4_step_004", + "00000818_6bbad2756d7845bd8d731d60_step_000", + "00000819_6bbad2756d7845bd8d731d60_step_001", + "00000821_6bbad2756d7845bd8d731d60_step_003", + "00000822_c98978894f544f89aa10633f_step_000", + "00000823_c98978894f544f89aa10633f_step_001", + "00000827_b0d209f7a8ad4b02910c317f_step_003", + "00000829_889a065d51a24bb8ad704c69_step_000", + "00000830_889a065d51a24bb8ad704c69_step_001", + "00000832_889a065d51a24bb8ad704c69_step_003", + "00000834_889a065d51a24bb8ad704c69_step_005", + "00000837_b5409201f9a5495fa169af85_step_000", + "00000889_e2d3a4dc5f7c438bb240e9d1_step_000", + "00000890_e2d3a4dc5f7c438bb240e9d1_step_001", + "00000900_4df6ab9924674556bb4a509d_step_000", + "00000904_4df6ab9924674556bb4a509d_step_004", + "00000905_4df6ab9924674556bb4a509d_step_005", + "00000906_4df6ab9924674556bb4a509d_step_006", + "00000942_f297a24e1a4446cf9d74efd9_step_001", + "00000948_f297a24e1a4446cf9d74efd9_step_007", + "00000949_f297a24e1a4446cf9d74efd9_step_008", + "00000953_ccc14f0fcff54b5985cc9113_step_000", + "00000958_04906edf324a4bce8fc3f417_step_000", + "00000961_a98d481268ea4524a7769731_step_000", + "00000962_a98d481268ea4524a7769731_step_001", + "00000963_a98d481268ea4524a7769731_step_002", + "00000973_209426c5054945fab8a4af1b_step_000", + "00000976_b85ec2ff588148c4abd8c8d7_step_000", + "00001024_21adcacb9707488ea6655fd2_step_000", + "00001025_21adcacb9707488ea6655fd2_step_001", + "00001029_cde47df6e28543d1b434f206_step_001", + "00001030_cde47df6e28543d1b434f206_step_002", + "00001031_cde47df6e28543d1b434f206_step_003", + "00001033_18151464e6434759b7c2ece7_step_000", + "00001035_be3ed2c9696e4e6a91b34f79_step_000", + "00001036_be3ed2c9696e4e6a91b34f79_step_001", + "00001041_6baf25f5fe2947e8927797d8_step_000", + "00001048_2b2638f82b3946b1867f2d11_step_004", + "00001049_2b2638f82b3946b1867f2d11_step_005", + "00001050_2b2638f82b3946b1867f2d11_step_006", + "00001061_284effca5a19488fa8a228e2_step_002", + "00001062_c04956ddff6c4324b5cad67c_step_000", + "00001106_ba88e7579ced4a0aa2de629c_step_000", + "00001108_412675e2ea594e70bbd8cf6c_step_000", + "00001114_5f5cf6feacbc41fb88b2d873_step_000", + "00001121_da4037e40839451dbbc2bd1f_step_000", + "00001130_c2db313c2ce749d784c610d2_step_002", + "00001139_0b0ed9b0ccbb455d82c60d69_step_000", + "00001140_0b0ed9b0ccbb455d82c60d69_step_001", + "00001141_0b0ed9b0ccbb455d82c60d69_step_002", + "00001146_274866da478e4a009bed8750_step_001", + "00001159_f112d6f3e0c644ae9de2a724_step_000", + "00001165_4b65bb9a02f6494aa897a53e_step_000", + "00001169_0889c458f3f74dc288d2d531_step_000", + "00001170_0889c458f3f74dc288d2d531_step_001", + "00001182_bc0b5846e3d84904ad02e547_step_001", + "00001185_507b60b57791425bbb73273e_step_000", + "00001186_507b60b57791425bbb73273e_step_001", + "00001187_507b60b57791425bbb73273e_step_002", + "00001190_507b60b57791425bbb73273e_step_005", + "00001191_507b60b57791425bbb73273e_step_006", + "00001193_46b17dd88ed24f55bca2eb69_step_000", + "00001200_e44a0f9afbe84b918f789f0f_step_000", + "00001206_e44a0f9afbe84b918f789f0f_step_006", + "00001210_4be610c137ba42ae9933e69b_step_000", + "00001211_4be610c137ba42ae9933e69b_step_001", + "00001212_4be610c137ba42ae9933e69b_step_002", + "00001213_4be610c137ba42ae9933e69b_step_003", + "00001214_9edb90c9f417497fb70bddf6_step_000", + "00001228_2fd836ae38b44c41bfbd9584_step_002", + "00001232_73b40f9a4cd3419ab3b42b73_step_001", + "00001233_73b40f9a4cd3419ab3b42b73_step_002", + "00001235_17ea801126da4654b34f6c52_step_000", + "00001254_c2da273d137e475abb1ab1f4_step_000", + "00001255_3df9015915ff4fa7b57f0a71_step_000", + "00001257_a4715d363f88400c8f0238d1_step_000", + "00001260_a4715d363f88400c8f0238d1_step_003", + "00001266_b53ece83d8964b44bbf1f8ed_step_002", + "00001273_b53ece83d8964b44bbf1f8ed_step_009", + "00001274_b53ece83d8964b44bbf1f8ed_step_010", + "00001276_b53ece83d8964b44bbf1f8ed_step_012", + "00001285_b53ece83d8964b44bbf1f8ed_step_021", + "00001294_b53ece83d8964b44bbf1f8ed_step_030", + "00001298_b53ece83d8964b44bbf1f8ed_step_034", + "00001300_b9951da825954791b59b979d_step_000", + "00001301_a54ee779467b42ea8709e1ca_step_000", + "00001311_e95b8713cc0b4436909c78e9_step_000", + "00001319_6a5a7dbad03b4a57b33f2c0d_step_002", + "00001320_6a5a7dbad03b4a57b33f2c0d_step_003", + "00001321_7cef9b79a6b04db894fd74e5_step_000", + "00001324_5bf4a74af99f405392406e40_step_000", + "00001344_d8d97699a52842539e08d117_step_002", + "00001345_d8d97699a52842539e08d117_step_003", + "00001346_d8d97699a52842539e08d117_step_004", + "00001348_318b81a8e412438183201be7_step_000", + "00001349_27327ddba7f64417ae1e69a3_step_000", + "00001356_aa0de7c5e9e84d73816bdbde_step_000", + "00001380_c49484307f434da78fa06b1f_step_000", + "00001381_c49484307f434da78fa06b1f_step_001", + "00001389_21b525da345e4e62aebd906b_step_000", + "00001391_406eb61c554a429ebd8a4d4f_step_000", + "00001392_406eb61c554a429ebd8a4d4f_step_001", + "00001393_406eb61c554a429ebd8a4d4f_step_002", + "00001394_406eb61c554a429ebd8a4d4f_step_003", + "00001414_b38f661707674b8eb983d992_step_000", + "00001417_a45f6c99e2154f3b94955175_step_001", + "00001422_a45f6c99e2154f3b94955175_step_006", + "00001427_a45f6c99e2154f3b94955175_step_011", + "00001428_a45f6c99e2154f3b94955175_step_012", + "00001429_a45f6c99e2154f3b94955175_step_013", + "00001430_a45f6c99e2154f3b94955175_step_014", + "00001432_a45f6c99e2154f3b94955175_step_016", + "00001433_a45f6c99e2154f3b94955175_step_017", + "00001434_a45f6c99e2154f3b94955175_step_018", + "00001438_a45f6c99e2154f3b94955175_step_022", + "00001445_a45f6c99e2154f3b94955175_step_029", + "00001447_a45f6c99e2154f3b94955175_step_031", + "00001449_a45f6c99e2154f3b94955175_step_033", + "00001450_a45f6c99e2154f3b94955175_step_034", + "00001453_a45f6c99e2154f3b94955175_step_037", + "00001455_a45f6c99e2154f3b94955175_step_039", + "00001456_a45f6c99e2154f3b94955175_step_040", + "00001459_a45f6c99e2154f3b94955175_step_043", + "00001461_a45f6c99e2154f3b94955175_step_045", + "00001467_a45f6c99e2154f3b94955175_step_051", + "00001468_a45f6c99e2154f3b94955175_step_052", + "00001469_a45f6c99e2154f3b94955175_step_053", + "00001471_a45f6c99e2154f3b94955175_step_055", + "00001472_a45f6c99e2154f3b94955175_step_056", + "00001474_a45f6c99e2154f3b94955175_step_058", + "00001480_a45f6c99e2154f3b94955175_step_064", + "00001481_a45f6c99e2154f3b94955175_step_065", + "00001482_a45f6c99e2154f3b94955175_step_066", + "00001483_a45f6c99e2154f3b94955175_step_067", + "00001484_a45f6c99e2154f3b94955175_step_068", + "00001487_a45f6c99e2154f3b94955175_step_071", + "00001490_6c8ef6d2f5564d139a5077f8_step_001", + "00001491_6c8ef6d2f5564d139a5077f8_step_002", + "00001495_6c8ef6d2f5564d139a5077f8_step_006", + "00001496_eb4e2b01aaca4131bfc64adb_step_000", + "00001497_eb4e2b01aaca4131bfc64adb_step_001", + "00001500_eb4e2b01aaca4131bfc64adb_step_004", + "00001501_6a0f221e85e04411afe23f51_step_000", + "00001503_6a0f221e85e04411afe23f51_step_002", + "00001505_6d02705716e04d9081f4980b_step_000", + "00001527_9214869844f14e83a9dce3ec_step_000", + "00001534_e1d62958d7ca4ad18fefd54d_step_001", + "00001535_e1d62958d7ca4ad18fefd54d_step_002", + "00001538_e1d62958d7ca4ad18fefd54d_step_005", + "00001543_e1d62958d7ca4ad18fefd54d_step_010", + "00001550_a94d978cf62948ce80b417a9_step_000", + "00001554_30de00089a3f4a339eddb68d_step_000", + "00001555_30de00089a3f4a339eddb68d_step_001", + "00001556_30de00089a3f4a339eddb68d_step_002", + "00001557_16c191aef5034779aaab6e3f_step_000", + "00001563_2e9dc7e9a1a54dd7ab901478_step_000", + "00001567_9280741c840d4aa1af32658e_step_002", + "00001569_6fc579f3224d4d53a1b60710_step_000", + "00001578_a3b89691665043a08931897f_step_001", + "00001579_a3b89691665043a08931897f_step_002", + "00001582_a3b89691665043a08931897f_step_005", + "00001586_763da6a559544098acbf4154_step_002", + "00001592_35f4abaacc9447e4ac0b964f_step_001", + "00001594_2128dbbd4bf34879913b3a99_step_000", + "00001596_2128dbbd4bf34879913b3a99_step_002", + "00001597_13bd23e0e4c740969b0c71ab_step_000" + ], + "node_count": 11300812, + "mean": [ + -0.0023328224084797375, + 0.002493796834147401, + -0.001809911687675766, + 0.25742899061541036, + 1.3342682062027138, + 1.8059588831891302, + 3.111110856814303, + 9.224285832960385, + 1.0265793073404939, + -1.221687502467248 + ], + "std": [ + 0.6417830773452925, + 0.5878867083697583, + 0.5840912225689718, + 0.22858594008765615, + 0.6758364464754756, + 2.124656857116319, + 1.0612042542158557, + 2.4681213620763502, + 0.7631566124335305, + 1.6411759487011053 + ], + "columns": [ + "x_star", + "y_star", + "z_star", + "d_over_t", + "gate_distance_over_Rg", + "geodesic_over_Rg", + "log_Rg", + "log_volume", + "log_t_part", + "log_t_set" + ], + "weighting": "equal_learning_nodes", + "ddof": 0, + "constant_column_scale": 1.0, + "fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03" +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/started.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/started.json new file mode 100644 index 000000000..656464446 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/started.json @@ -0,0 +1,4 @@ +{ + "started_unix": 1794696833.528147, + "contract_sha256": "f6eaa3cb2fd6918e5bd940826a2cc954ffcee0a9a2d1578b2da1e5e1c72abd5c" +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0-guard-receipt.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0-guard-receipt.json new file mode 100644 index 000000000..8376d18ec --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0-guard-receipt.json @@ -0,0 +1,34 @@ +{ + "status": "PASS", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "0" + ], + "returncode": 0, + "stop_reason": null, + "cpu_seconds": 1129.584761, + "wall_seconds": 1129.2963322199648, + "peak_tree_rss_bytes": 2574286848, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 12288, + "max_poll_gap_seconds": 0.8972175170201808, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null, + "device_monitor": { + "peak_device_memory_bytes": 8662286336, + "peak_whole_run_tree_rss_bytes": 2980745216, + "max_poll_gap_seconds": 1.473451861005742, + "reason": null, + "samples": 1896 + }, + "total_run_wall_seconds": 1129.3405424560187, + "result_sha256": "f75ded80d92c4e9835ca8bcda6b9772fd1fe56b131c9964097fdfea7ea8f1a44" +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/best-validation-parts.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/best-validation-parts.json new file mode 100644 index 000000000..3bc2cd0fe --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/best-validation-parts.json @@ -0,0 +1,113 @@ +{ + "00000111_ce24e0cbec8e45c89735d148_step_003": 0.1418326986805392, + "00000115_ce24e0cbec8e45c89735d148_step_007": 0.13499011979614386, + "00000116_ce24e0cbec8e45c89735d148_step_008": 0.09001265818381358, + "00000117_ce24e0cbec8e45c89735d148_step_009": 0.15049204735091037, + "00000119_ce24e0cbec8e45c89735d148_step_011": 0.12714211544006057, + "00000120_ce24e0cbec8e45c89735d148_step_012": 0.10739385897229049, + "00000131_945244874ed84e599987daa9_step_000": 0.06561638461790519, + "00000132_438507a38e404bfdbd51a16b_step_000": 0.07042763008898459, + "00000133_08b739e0511f4305b161cd05_step_000": 0.06044987283890425, + "00000190_185ecbc783f34408aef52119_step_000": 0.0645608273773318, + "00000196_c9b71ecdd0654716891b0db8_step_000": 0.08909182609494609, + "00000206_27417003e762409185cb2b69_step_000": 0.05173494264830453, + "00000238_4829bc0645d6414d9b5cec31_step_001": 0.06189846220013667, + "00000241_4829bc0645d6414d9b5cec31_step_004": 0.21256467328062745, + "00000243_4829bc0645d6414d9b5cec31_step_006": 0.09960978018234652, + "00000248_e87554b09cdc48c3add39db9_step_000": 0.15333624470653126, + "00000257_845fb06d7f2f4802a806a600_step_002": 0.05481768723087835, + "00000258_845fb06d7f2f4802a806a600_step_003": 0.08446282170570607, + "00000259_845fb06d7f2f4802a806a600_step_004": 0.060935314579890786, + "00000261_845fb06d7f2f4802a806a600_step_006": 0.08786963364042194, + "00000262_845fb06d7f2f4802a806a600_step_007": 0.06685207183750448, + "00000264_845fb06d7f2f4802a806a600_step_009": 0.21276797350358556, + "00000265_845fb06d7f2f4802a806a600_step_010": 0.1152459026318642, + "00000266_845fb06d7f2f4802a806a600_step_011": 0.13908850412125517, + "00000267_845fb06d7f2f4802a806a600_step_012": 0.038459567348800315, + "00000269_845fb06d7f2f4802a806a600_step_014": 0.07998723872452156, + "00000391_fd490b564a44413884fab719_step_000": 0.053289409278953176, + "00000405_f3759bbb4682423caf207e8e_step_002": 0.11693916201765817, + "00000406_f3759bbb4682423caf207e8e_step_003": 0.06354150271191995, + "00000444_4ed4c78d6d754aac90876fc2_step_003": 0.06630675132082518, + "00000445_4ed4c78d6d754aac90876fc2_step_004": 0.08509133030182592, + "00000501_5499914343bd4f549b38046d_step_002": 0.042797312965140825, + "00000502_5499914343bd4f549b38046d_step_003": 0.06110567784500328, + "00000574_5a4cef68211d4706b1ec8586_step_003": 0.1799449352618915, + "00000578_5a4cef68211d4706b1ec8586_step_007": 0.12486475222208533, + "00000579_5a4cef68211d4706b1ec8586_step_008": 0.12399657022129994, + "00000580_5a4cef68211d4706b1ec8586_step_009": 0.14633975991850098, + "00000582_5a4cef68211d4706b1ec8586_step_011": 0.1449923620128791, + "00000583_5a4cef68211d4706b1ec8586_step_012": 0.18491147076012035, + "00000597_7fe894812c4244e18241aa08_step_003": 0.21573004354242584, + "00000601_7fe894812c4244e18241aa08_step_007": 0.1572722213759513, + "00000602_7fe894812c4244e18241aa08_step_008": 0.10159354554243094, + "00000603_7fe894812c4244e18241aa08_step_009": 0.1588393365134127, + "00000605_7fe894812c4244e18241aa08_step_011": 0.13886339776337586, + "00000606_7fe894812c4244e18241aa08_step_012": 0.17459572150269845, + "00000654_d87d11d9c3ca4ee5a88797ae_step_002": 0.10299334066133657, + "00000758_8ab50540e9804365a8717ec0_step_005": 0.11024779630443561, + "00000759_8ab50540e9804365a8717ec0_step_006": 0.06372706814866537, + "00000760_8ab50540e9804365a8717ec0_step_007": 0.09364000343350842, + "00000763_8ab50540e9804365a8717ec0_step_010": 0.10341671622882422, + "00000816_36e6258346004cabb4c091b2_step_004": 0.1259031151697205, + "00000817_36e6258346004cabb4c091b2_step_005": 0.16138226171626582, + "00000892_be26dc83f78947738b2bfabd_step_001": 0.127611145100154, + "00000894_be26dc83f78947738b2bfabd_step_003": 0.19958599743054536, + "00000897_be26dc83f78947738b2bfabd_step_006": 0.12253258600065273, + "00000919_cdcebf32644443b9812097ba_step_003": 0.12401410817981792, + "00000923_cdcebf32644443b9812097ba_step_007": 0.14241699148970582, + "00000927_cdcebf32644443b9812097ba_step_011": 0.1313261189891303, + "00000928_cdcebf32644443b9812097ba_step_012": 0.1538292523152045, + "00000955_a741c7966d2c40a8a511263e_step_001": 0.07181432325235683, + "00000984_0dd1386597104265ab67e6cf_step_007": 0.10558178533436346, + "00000986_0dd1386597104265ab67e6cf_step_009": 0.08690933886807856, + "00000987_0dd1386597104265ab67e6cf_step_010": 0.15043230024249143, + "00000988_0dd1386597104265ab67e6cf_step_011": 0.15490127456439406, + "00000996_6d71695ba3fd475c927a00a7_step_000": 0.1473608798446418, + "00001019_ca4cf9693d6d4a98b515ba99_step_000": 0.04806313652724216, + "00001020_ca4cf9693d6d4a98b515ba99_step_001": 0.1087400538319381, + "00001021_ca4cf9693d6d4a98b515ba99_step_002": 0.047830857393725314, + "00001026_0f133d8e7fd448708a554e28_step_000": 0.041359967707447196, + "00001027_0f133d8e7fd448708a554e28_step_001": 0.31211983900021895, + "00001040_74c42207fa764831b03f7658_step_000": 0.052893689863997964, + "00001052_2fd6155fa34f418083d0bd0f_step_001": 0.04065864961780492, + "00001053_2fd6155fa34f418083d0bd0f_step_002": 0.05407272129857645, + "00001097_256ee7ec3a4b4972a8b87c8d_step_001": 0.1605008064330961, + "00001099_256ee7ec3a4b4972a8b87c8d_step_003": 0.13480483317197903, + "00001102_256ee7ec3a4b4972a8b87c8d_step_006": 0.1604612417937521, + "00001107_78566795c3144719804f8d34_step_000": 0.08206973507138492, + "00001167_107f61d0402f4c9a9d9d1342_step_000": 0.15098881950153378, + "00001168_107f61d0402f4c9a9d9d1342_step_001": 0.061267105819128566, + "00001217_e517c61f1c49433bb81cf980_step_000": 0.15074544681325494, + "00001238_96503b6a26c64074ad598927_step_000": 0.07838898455339834, + "00001239_96503b6a26c64074ad598927_step_001": 0.09081730021802198, + "00001240_96503b6a26c64074ad598927_step_002": 0.06000187987183002, + "00001241_96503b6a26c64074ad598927_step_003": 0.2132383816296462, + "00001242_96503b6a26c64074ad598927_step_004": 0.03190889193566521, + "00001243_96503b6a26c64074ad598927_step_005": 0.03749073013370614, + "00001244_96503b6a26c64074ad598927_step_006": 0.07746417105718707, + "00001246_96503b6a26c64074ad598927_step_008": 0.1673811760108979, + "00001247_96503b6a26c64074ad598927_step_009": 0.09810865898584362, + "00001249_96503b6a26c64074ad598927_step_011": 0.17879500853144886, + "00001252_96503b6a26c64074ad598927_step_014": 0.07365205286236294, + "00001253_96503b6a26c64074ad598927_step_015": 0.13693958901292222, + "00001314_a9ed5e56a6a8414f91b39bdb_step_000": 0.07904392342403124, + "00001333_beab79941e884455a62af40c_step_001": 0.15638275101293253, + "00001334_beab79941e884455a62af40c_step_002": 0.20706939922580525, + "00001335_beab79941e884455a62af40c_step_003": 0.29567729965028205, + "00001336_beab79941e884455a62af40c_step_004": 0.2577203085123177, + "00001338_beab79941e884455a62af40c_step_006": 0.2031776655156273, + "00001339_beab79941e884455a62af40c_step_007": 0.12940160203223647, + "00001340_beab79941e884455a62af40c_step_008": 0.16728377453322937, + "00001352_4e1d56bc2cef49ea84626b11_step_000": 0.0741311102399176, + "00001353_4e1d56bc2cef49ea84626b11_step_001": 0.09498265095533552, + "00001354_4e1d56bc2cef49ea84626b11_step_002": 0.08043764503807907, + "00001379_5b1d6e2599ea47d498a04256_step_000": 0.08594059060342593, + "00001387_60cf3ad636dd42e0bcec6946_step_000": 0.12039848732051094, + "00001511_5cbe58077e4b4885a5c213b3_step_004": 0.09918785598004878, + "00001512_5cbe58077e4b4885a5c213b3_step_005": 0.10671436839797326, + "00001525_e0d35a03b46745259cf8878f_step_000": 0.18317951254580667, + "00001532_9f0a4a641e8f4cc588eb4315_step_000": 0.1035452933149037, + "00001552_b0a9c4d836294846891615fe_step_000": 0.13775679338902058, + "00001553_adfa1c33ae4d4d0aa441ddc3_step_000": 0.09657311693713902 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/best.pt b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/best.pt new file mode 100644 index 000000000..4bff5b663 Binary files /dev/null and b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/best.pt differ diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-000-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-000-amp.json new file mode 100644 index 000000000..0d8e3bf8a --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-000-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 2, + "successful_updates": 380, + "skipped_fraction": 0.005235602094240838, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-001-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-001-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-001-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-002-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-002-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-002-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-003-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-003-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-003-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-004-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-004-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-004-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-005-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-005-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-005-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-006-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-006-amp.json new file mode 100644 index 000000000..f66f9a3d7 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-006-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-007-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-007-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-007-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-008-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-008-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-008-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-009-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-009-amp.json new file mode 100644 index 000000000..0d8e3bf8a --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-009-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 2, + "successful_updates": 380, + "skipped_fraction": 0.005235602094240838, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-010-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-010-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-010-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-011-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-011-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-011-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-012-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-012-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-012-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-013-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-013-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-013-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-014-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-014-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-014-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-015-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-015-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-015-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-016-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-016-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-016-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-017-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-017-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-017-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-018-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-018-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-018-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-019-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-019-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-019-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-020-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-020-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-020-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-021-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-021-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-021-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-022-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-022-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-022-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-023-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-023-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-023-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-024-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-024-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-024-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-025-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-025-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-025-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-026-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-026-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-026-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-027-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-027-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-027-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-028-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-028-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-028-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-029-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-029-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-029-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-030-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-030-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-030-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-031-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-031-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-031-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-032-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-032-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-032-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-033-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-033-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-033-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-034-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-034-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-034-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-035-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-035-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-035-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-036-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-036-amp.json new file mode 100644 index 000000000..f66f9a3d7 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-036-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-037-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-037-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-037-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-038-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-038-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-038-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-039-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-039-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-039-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-040-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-040-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-040-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-041-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-041-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-041-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/history.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/history.json new file mode 100644 index 000000000..38a36120b --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/history.json @@ -0,0 +1,716 @@ +[ + { + "epoch": 0, + "amp": { + "attempted_steps": 382, + "skipped_steps": 2, + "successful_updates": 380, + "skipped_fraction": 0.005235602094240838, + "consecutive_skipped": 0 + }, + "attempted_updates": 382, + "training_global_weighted_mse": 0.0376763111023819, + "validation_component_equal_relative_l2": 0.18041538781651764, + "updates": 380, + "elapsed_seconds": 34.171638627012726, + "peak_allocated_bytes": 4704977408, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 1, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 764, + "training_global_weighted_mse": 0.013574996691685373, + "validation_component_equal_relative_l2": 0.15493626748365733, + "updates": 762, + "elapsed_seconds": 61.482827570987865, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 2, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 1146, + "training_global_weighted_mse": 0.013291919839288664, + "validation_component_equal_relative_l2": 0.1940242081799989, + "updates": 1144, + "elapsed_seconds": 88.85465831303736, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 3, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 1528, + "training_global_weighted_mse": 0.014076809640803156, + "validation_component_equal_relative_l2": 0.1708517999428941, + "updates": 1526, + "elapsed_seconds": 115.2649692420382, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 4, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 1910, + "training_global_weighted_mse": 0.010244692359512784, + "validation_component_equal_relative_l2": 0.15454044801141548, + "updates": 1908, + "elapsed_seconds": 141.01606989401625, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 5, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 2292, + "training_global_weighted_mse": 0.010016330175515616, + "validation_component_equal_relative_l2": 0.1377861135351885, + "updates": 2290, + "elapsed_seconds": 168.80598288500914, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 6, + "amp": { + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 + }, + "attempted_updates": 2674, + "training_global_weighted_mse": 0.011456984507376484, + "validation_component_equal_relative_l2": 0.12138804202529904, + "updates": 2671, + "elapsed_seconds": 195.41296529903775, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 7, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 3056, + "training_global_weighted_mse": 0.009360101058971292, + "validation_component_equal_relative_l2": 0.1360153005260273, + "updates": 3053, + "elapsed_seconds": 221.9400074610021, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 8, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 3438, + "training_global_weighted_mse": 0.009015654394901674, + "validation_component_equal_relative_l2": 0.13570373305049202, + "updates": 3435, + "elapsed_seconds": 249.30439730500802, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 9, + "amp": { + "attempted_steps": 382, + "skipped_steps": 2, + "successful_updates": 380, + "skipped_fraction": 0.005235602094240838, + "consecutive_skipped": 0 + }, + "attempted_updates": 3820, + "training_global_weighted_mse": 0.03311672873733853, + "validation_component_equal_relative_l2": 0.1221200065586085, + "updates": 3815, + "elapsed_seconds": 276.22303390403977, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 10, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 4202, + "training_global_weighted_mse": 0.00801496203818968, + "validation_component_equal_relative_l2": 0.12521708969429135, + "updates": 4197, + "elapsed_seconds": 304.6835392920184, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 11, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 4584, + "training_global_weighted_mse": 0.008661952212077132, + "validation_component_equal_relative_l2": 0.14101939395160956, + "updates": 4579, + "elapsed_seconds": 334.20663234003587, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 12, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 4966, + "training_global_weighted_mse": 0.007396083116754302, + "validation_component_equal_relative_l2": 0.13952052142293714, + "updates": 4961, + "elapsed_seconds": 361.0935193920159, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 13, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 5348, + "training_global_weighted_mse": 0.007140785819577804, + "validation_component_equal_relative_l2": 0.11647707701728477, + "updates": 5343, + "elapsed_seconds": 388.0761553220218, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 14, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 5730, + "training_global_weighted_mse": 0.007229909640164705, + "validation_component_equal_relative_l2": 0.11198374301310728, + "updates": 5725, + "elapsed_seconds": 414.78899292700225, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 15, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 6112, + "training_global_weighted_mse": 0.006851368930432469, + "validation_component_equal_relative_l2": 0.12048943284963896, + "updates": 6107, + "elapsed_seconds": 439.9251486940193, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 16, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 6494, + "training_global_weighted_mse": 0.00686519764697775, + "validation_component_equal_relative_l2": 0.1116265332689199, + "updates": 6489, + "elapsed_seconds": 467.63780933001544, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 17, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 6876, + "training_global_weighted_mse": 0.006735978730406744, + "validation_component_equal_relative_l2": 0.13150426355242878, + "updates": 6871, + "elapsed_seconds": 493.21079543302767, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 18, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 7258, + "training_global_weighted_mse": 0.006527966911833965, + "validation_component_equal_relative_l2": 0.11053252959907517, + "updates": 7253, + "elapsed_seconds": 519.5444782010163, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 19, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 7640, + "training_global_weighted_mse": 0.006637135257900121, + "validation_component_equal_relative_l2": 0.13184045023118085, + "updates": 7635, + "elapsed_seconds": 547.4968308720272, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8243904512 + }, + { + "epoch": 20, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 8022, + "training_global_weighted_mse": 0.0064657076111618635, + "validation_component_equal_relative_l2": 0.13575660649206883, + "updates": 8017, + "elapsed_seconds": 574.2422935080249, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 21, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 8404, + "training_global_weighted_mse": 0.006806664202271711, + "validation_component_equal_relative_l2": 0.11228872658539735, + "updates": 8399, + "elapsed_seconds": 600.6447302990127, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 22, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 8786, + "training_global_weighted_mse": 0.006255876271640644, + "validation_component_equal_relative_l2": 0.11283803579359453, + "updates": 8781, + "elapsed_seconds": 625.7384886500076, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 23, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 9168, + "training_global_weighted_mse": 0.006155923257225093, + "validation_component_equal_relative_l2": 0.11738448558748187, + "updates": 9163, + "elapsed_seconds": 650.9155354429968, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 24, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 9550, + "training_global_weighted_mse": 0.006293353169446939, + "validation_component_equal_relative_l2": 0.11377664699794135, + "updates": 9545, + "elapsed_seconds": 676.0251263119862, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 25, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 9932, + "training_global_weighted_mse": 0.006010053819056853, + "validation_component_equal_relative_l2": 0.12613637400301086, + "updates": 9927, + "elapsed_seconds": 701.7957015210413, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 26, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 10314, + "training_global_weighted_mse": 0.00577524656157743, + "validation_component_equal_relative_l2": 0.10825904997600434, + "updates": 10309, + "elapsed_seconds": 728.8601587369922, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 27, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 10696, + "training_global_weighted_mse": 0.005821629934919064, + "validation_component_equal_relative_l2": 0.111946352590569, + "updates": 10691, + "elapsed_seconds": 755.6950904800324, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 28, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 11078, + "training_global_weighted_mse": 0.0056493395603442005, + "validation_component_equal_relative_l2": 0.10898317999919208, + "updates": 11073, + "elapsed_seconds": 781.9649670849903, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 29, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 11460, + "training_global_weighted_mse": 0.00557189901612781, + "validation_component_equal_relative_l2": 0.10827198186038772, + "updates": 11455, + "elapsed_seconds": 808.9486410579993, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 30, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 11842, + "training_global_weighted_mse": 0.005348194421124851, + "validation_component_equal_relative_l2": 0.11470852761524397, + "updates": 11837, + "elapsed_seconds": 834.6714666190092, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 31, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 12224, + "training_global_weighted_mse": 0.005487700959080956, + "validation_component_equal_relative_l2": 0.11481427543551495, + "updates": 12219, + "elapsed_seconds": 860.4288292980054, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 32, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 12606, + "training_global_weighted_mse": 0.0052110311188465356, + "validation_component_equal_relative_l2": 0.10655526016119332, + "updates": 12601, + "elapsed_seconds": 887.3232852260116, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 33, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 12988, + "training_global_weighted_mse": 0.005177555075643265, + "validation_component_equal_relative_l2": 0.10787302746281699, + "updates": 12983, + "elapsed_seconds": 913.9468120289966, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 34, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 13370, + "training_global_weighted_mse": 0.00505741697636126, + "validation_component_equal_relative_l2": 0.10958265189103132, + "updates": 13365, + "elapsed_seconds": 939.7434611140052, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 35, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 13752, + "training_global_weighted_mse": 0.00497264900234944, + "validation_component_equal_relative_l2": 0.10586608727592606, + "updates": 13747, + "elapsed_seconds": 965.6309566330165, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 36, + "amp": { + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 + }, + "attempted_updates": 14134, + "training_global_weighted_mse": 0.004920186778995301, + "validation_component_equal_relative_l2": 0.10409490688124762, + "updates": 14128, + "elapsed_seconds": 991.8713217549957, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 37, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 14516, + "training_global_weighted_mse": 0.004877662268420088, + "validation_component_equal_relative_l2": 0.10432341104908842, + "updates": 14510, + "elapsed_seconds": 1018.4420697060414, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 38, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 14898, + "training_global_weighted_mse": 0.004820686372389598, + "validation_component_equal_relative_l2": 0.10437913802054469, + "updates": 14892, + "elapsed_seconds": 1045.714647390996, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 39, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 15280, + "training_global_weighted_mse": 0.004785080393501068, + "validation_component_equal_relative_l2": 0.10332562533675191, + "updates": 15274, + "elapsed_seconds": 1072.3014936200343, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 40, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 15662, + "training_global_weighted_mse": 0.004766768432453233, + "validation_component_equal_relative_l2": 0.10346326874711213, + "updates": 15656, + "elapsed_seconds": 1098.345144032035, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + }, + { + "epoch": 41, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 16044, + "training_global_weighted_mse": 0.004746648263794883, + "validation_component_equal_relative_l2": 0.10341266280676208, + "updates": 16038, + "elapsed_seconds": 1125.9099378320388, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664 + } +] diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/progress.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/progress.json new file mode 100644 index 000000000..46daf010d --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/progress.json @@ -0,0 +1,19 @@ +{ + "stage": "epoch_complete", + "epoch": 41, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 16044, + "training_global_weighted_mse": 0.004746648263794883, + "validation_component_equal_relative_l2": 0.10341266280676208, + "updates": 16038, + "elapsed_seconds": 1125.9099378320388, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664, + "patience_count": 2 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/result.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/result.json new file mode 100644 index 000000000..aefdde976 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/result.json @@ -0,0 +1,210 @@ +{ + "status": "COMPLETE", + "run": { + "name": "N128-seed0", + "arm": "N", + "width": 128, + "seed": 0 + }, + "E_max": 42, + "cosine_T_max": 42, + "epochs": 42, + "attempted_updates": 16044, + "updates": 16038, + "best_epoch": 39, + "best_validation_score": 0.10332562533675191, + "learning_parts": 382, + "validation_parts": 111, + "learning_components": 137, + "validation_components": 35, + "failed_preparation_parts_dropped": [ + { + "case": "00000575_5a4cef68211d4706b1ec8586_step_004", + "role": "validation", + "nodes": 75046, + "component_id": "component-3a3e340f4dfac391", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-inputs-2026-11-15", + "181" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.766116, + "wall_seconds": 1800.400393687014, + "peak_tree_rss_bytes": 602218496, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.128913332009688, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false + }, + "failure": null + }, + { + "case": "00000614_c710d414bbbc4340b682c58a_step_000", + "role": "learning", + "nodes": 145860, + "component_id": "component-4bd089778094e746", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-inputs-2026-11-15", + "201" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.796751, + "wall_seconds": 1800.0567526929663, + "peak_tree_rss_bytes": 629305344, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.1576595069491304, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false + }, + "failure": null + }, + { + "case": "00000625_6af98f5458b347bb843f0562_step_000", + "role": "learning", + "nodes": 145707, + "component_id": "component-4bd089778094e746", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-inputs-2026-11-15", + "211" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.8864629999998, + "wall_seconds": 1800.119237862993, + "peak_tree_rss_bytes": 647880704, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.159920622012578, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false + }, + "failure": null + }, + { + "case": "00001466_a45f6c99e2154f3b94955175_step_050", + "role": "learning", + "nodes": 77138, + "component_id": "component-7fe5173e77c86f9d", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-recovery-inputs-2026-11-15", + "83" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.926194, + "wall_seconds": 1800.0714844280155, + "peak_tree_rss_bytes": 383070208, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.0109735240112059, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null + }, + "failure": null + }, + { + "case": "00001470_a45f6c99e2154f3b94955175_step_054", + "role": "learning", + "nodes": 124817, + "component_id": "component-7fe5173e77c86f9d", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-recovery-inputs-2026-11-15", + "87" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.664371, + "wall_seconds": 1800.0443400070071, + "peak_tree_rss_bytes": 578732032, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 1.007104828953743, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null + }, + "failure": null + }, + { + "case": "00001473_a45f6c99e2154f3b94955175_step_057", + "role": "learning", + "nodes": 69428, + "component_id": "component-7fe5173e77c86f9d", + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/s2_pilot_prepare.py", + "artifacts/s2-pilot-recovery-inputs-2026-11-15", + "90" + ], + "returncode": -9, + "stop_reason": "wall_limit", + "cpu_seconds": 1799.751432, + "wall_seconds": 1800.2498029720155, + "peak_tree_rss_bytes": 438853632, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 4096, + "max_poll_gap_seconds": 0.8929723129840568, + "poll_target_seconds": 0.25, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": false, + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null + }, + "failure": null + } + ], + "standardizer_fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03", + "wall_seconds": 1125.9121631510206, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664, + "test_labels_read": false, + "test_predictions": false, + "capacity_selected": false, + "checkpoint_sha256": "ca2eb128116e877e2ffed65f4c1c5864e090beab9b7c0615f117e613599ccd34" +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/standardizer.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/standardizer.json new file mode 100644 index 000000000..11716fd2e --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/standardizer.json @@ -0,0 +1,429 @@ +{ + "fold": "0", + "pool": "100", + "learning_part_ids": [ + "00000003_1ffb81a71e5b402e966b9341_step_002", + "00000004_1ffb81a71e5b402e966b9341_step_003", + "00000021_ad34a3f60c4a4caa99646600_step_001", + "00000023_ad34a3f60c4a4caa99646600_step_003", + "00000024_ad34a3f60c4a4caa99646600_step_004", + "00000030_ad34a3f60c4a4caa99646600_step_010", + "00000031_ad34a3f60c4a4caa99646600_step_011", + "00000033_ad34a3f60c4a4caa99646600_step_013", + "00000049_aa208c45e8734584a147f595_step_000", + "00000057_666139e3bff64d4e8a6ce183_step_006", + "00000058_666139e3bff64d4e8a6ce183_step_007", + "00000060_767e4372b5f94a88a7a17d90_step_001", + "00000061_767e4372b5f94a88a7a17d90_step_002", + "00000065_767e4372b5f94a88a7a17d90_step_006", + "00000066_767e4372b5f94a88a7a17d90_step_007", + "00000067_767e4372b5f94a88a7a17d90_step_008", + "00000068_767e4372b5f94a88a7a17d90_step_009", + "00000070_767e4372b5f94a88a7a17d90_step_011", + "00000072_767e4372b5f94a88a7a17d90_step_013", + "00000077_767e4372b5f94a88a7a17d90_step_018", + "00000078_767e4372b5f94a88a7a17d90_step_019", + "00000079_767e4372b5f94a88a7a17d90_step_020", + "00000087_b4f9335a60a14cecaa599159_step_002", + "00000088_169bec34317b46e98eb1a403_step_000", + "00000093_169bec34317b46e98eb1a403_step_005", + "00000098_72cf0f9ca5324822a4d41a80_step_000", + "00000103_52f7124261ae4c558f800775_step_000", + "00000104_52f7124261ae4c558f800775_step_001", + "00000105_52f7124261ae4c558f800775_step_002", + "00000106_52f7124261ae4c558f800775_step_003", + "00000125_e18c7feacfb54f3880659b43_step_000", + "00000128_e18c7feacfb54f3880659b43_step_003", + "00000129_e18c7feacfb54f3880659b43_step_004", + "00000130_e18c7feacfb54f3880659b43_step_005", + "00000140_2fc54fcd110d4f49969163c4_step_003", + "00000142_2fc54fcd110d4f49969163c4_step_005", + "00000144_2fc54fcd110d4f49969163c4_step_007", + "00000145_2fc54fcd110d4f49969163c4_step_008", + "00000152_a951550b8a1043a68423c541_step_002", + "00000153_a951550b8a1043a68423c541_step_003", + "00000154_a951550b8a1043a68423c541_step_004", + "00000157_8208fd8b44b64dce91322f84_step_000", + "00000161_a583c6c3cdc143bab3f6c856_step_000", + "00000162_4dcda9fab6dd46cf98fb194e_step_000", + "00000163_ff9b2ed255484db78bb0e6da_step_000", + "00000164_91adbf180e114b698160c9a4_step_000", + "00000165_4070a99d6ebe45f5bd5d71eb_step_000", + "00000166_2246cab58dbe4a2fb00fed50_step_000", + "00000170_b21aa794b6b64d87a57c245e_step_000", + "00000171_b21aa794b6b64d87a57c245e_step_001", + "00000173_b21aa794b6b64d87a57c245e_step_003", + "00000174_b21aa794b6b64d87a57c245e_step_004", + "00000175_b21aa794b6b64d87a57c245e_step_005", + "00000176_b21aa794b6b64d87a57c245e_step_006", + "00000181_b21aa794b6b64d87a57c245e_step_011", + "00000198_826494e894454504b0971fd1_step_001", + "00000223_97b354a907114b7183faa9c4_step_001", + "00000224_97b354a907114b7183faa9c4_step_002", + "00000229_682d529d5e8c4387bc59b3a1_step_000", + "00000250_641f03cbe6a145508c6ae055_step_000", + "00000272_648a7c23a21f4e88bf25ffeb_step_000", + "00000273_648a7c23a21f4e88bf25ffeb_step_001", + "00000274_648a7c23a21f4e88bf25ffeb_step_002", + "00000275_648a7c23a21f4e88bf25ffeb_step_003", + "00000276_648a7c23a21f4e88bf25ffeb_step_004", + "00000280_5a6ef3285b89452ea6b390e7_step_001", + "00000282_5a6ef3285b89452ea6b390e7_step_003", + "00000285_d215b7bc60ae468288e5f1ea_step_000", + "00000287_58f8ff46b7564e989f9695b7_step_000", + "00000290_58f8ff46b7564e989f9695b7_step_003", + "00000291_58f8ff46b7564e989f9695b7_step_004", + "00000293_58f8ff46b7564e989f9695b7_step_006", + "00000295_52a153d2417747b5926ff9df_step_000", + "00000296_52a153d2417747b5926ff9df_step_001", + "00000301_52a153d2417747b5926ff9df_step_006", + "00000302_39b08d62a677414f86fdae52_step_000", + "00000305_3062bccff48e47a2b9de05e3_step_000", + "00000306_3062bccff48e47a2b9de05e3_step_001", + "00000307_3062bccff48e47a2b9de05e3_step_002", + "00000308_3062bccff48e47a2b9de05e3_step_003", + "00000309_3062bccff48e47a2b9de05e3_step_004", + "00000310_3062bccff48e47a2b9de05e3_step_005", + "00000311_3062bccff48e47a2b9de05e3_step_006", + "00000312_3062bccff48e47a2b9de05e3_step_007", + "00000314_3062bccff48e47a2b9de05e3_step_009", + "00000315_3062bccff48e47a2b9de05e3_step_010", + "00000316_3062bccff48e47a2b9de05e3_step_011", + "00000317_3062bccff48e47a2b9de05e3_step_012", + "00000318_3062bccff48e47a2b9de05e3_step_013", + "00000319_3062bccff48e47a2b9de05e3_step_014", + "00000321_3062bccff48e47a2b9de05e3_step_016", + "00000322_3062bccff48e47a2b9de05e3_step_017", + "00000323_3062bccff48e47a2b9de05e3_step_018", + "00000324_3062bccff48e47a2b9de05e3_step_019", + "00000325_3062bccff48e47a2b9de05e3_step_020", + "00000329_12a935cd9b684f879c0505fe_step_000", + "00000330_12a935cd9b684f879c0505fe_step_001", + "00000331_12a935cd9b684f879c0505fe_step_002", + "00000336_1a821c3883f7459a9615fca4_step_000", + "00000337_153efbd85ca64c4fb6b7ed5f_step_000", + "00000346_4185972a944744d8a7a0f2b4_step_002", + "00000348_4185972a944744d8a7a0f2b4_step_004", + "00000352_4185972a944744d8a7a0f2b4_step_008", + "00000355_68d1a5c2319a4a419ff625c6_step_001", + "00000356_68d1a5c2319a4a419ff625c6_step_002", + "00000357_68d1a5c2319a4a419ff625c6_step_003", + "00000364_38d899e2706a4d09960705dd_step_001", + "00000367_23c957e8e7b2428282a13ae2_step_000", + "00000368_23c957e8e7b2428282a13ae2_step_001", + "00000369_23c957e8e7b2428282a13ae2_step_002", + "00000374_23c957e8e7b2428282a13ae2_step_007", + "00000375_23c957e8e7b2428282a13ae2_step_008", + "00000376_23c957e8e7b2428282a13ae2_step_009", + "00000377_23c957e8e7b2428282a13ae2_step_010", + "00000378_23c957e8e7b2428282a13ae2_step_011", + "00000379_239bbf0e92974a32bd214366_step_000", + "00000382_239bbf0e92974a32bd214366_step_003", + "00000383_239bbf0e92974a32bd214366_step_004", + "00000386_239bbf0e92974a32bd214366_step_007", + "00000387_239bbf0e92974a32bd214366_step_008", + "00000388_239bbf0e92974a32bd214366_step_009", + "00000418_ce5b31679ee845a69ba89d6f_step_000", + "00000422_8e8a96d437584c77b3cfd84f_step_000", + "00000423_8e8a96d437584c77b3cfd84f_step_001", + "00000424_8e8a96d437584c77b3cfd84f_step_002", + "00000459_516b1784884d4facbc4c0d74_step_007", + "00000462_516b1784884d4facbc4c0d74_step_010", + "00000466_11951e51b89249dba38ded3e_step_001", + "00000469_11951e51b89249dba38ded3e_step_004", + "00000470_11951e51b89249dba38ded3e_step_005", + "00000474_a5639f2b84c445aea5da18ad_step_000", + "00000477_0fc8cc17c6af48dabe36c8f2_step_002", + "00000478_0fc8cc17c6af48dabe36c8f2_step_003", + "00000479_0fc8cc17c6af48dabe36c8f2_step_004", + "00000485_a7d0c851b5f24a97a7cac950_step_002", + "00000488_a7d0c851b5f24a97a7cac950_step_005", + "00000489_a7d0c851b5f24a97a7cac950_step_006", + "00000492_a7d0c851b5f24a97a7cac950_step_009", + "00000513_f78bf7cf156f4231988af062_step_001", + "00000519_5b8fa6b193d14279a77a0e50_step_000", + "00000523_1f8d7f343e634c11a80d23ac_step_000", + "00000526_f9d6559308dc4c85aab9fa38_step_000", + "00000530_bc0d6937206b43bc96521413_step_002", + "00000531_bc0d6937206b43bc96521413_step_003", + "00000543_8cf6554d440d40c4823dad6c_step_000", + "00000559_a1621e156ade4317ab6a5bc5_step_000", + "00000565_5b5119548f7b421e8f597438_step_000", + "00000587_2417b410de78416ebe50d460_step_000", + "00000589_2417b410de78416ebe50d460_step_002", + "00000590_2417b410de78416ebe50d460_step_003", + "00000591_2417b410de78416ebe50d460_step_004", + "00000609_04e799936ae446b9a3d8d2b8_step_000", + "00000610_04e799936ae446b9a3d8d2b8_step_001", + "00000611_04e799936ae446b9a3d8d2b8_step_002", + "00000613_189d1310aa8c4a2a92a21020_step_001", + "00000615_c710d414bbbc4340b682c58a_step_001", + "00000616_c710d414bbbc4340b682c58a_step_002", + "00000617_c710d414bbbc4340b682c58a_step_003", + "00000618_c710d414bbbc4340b682c58a_step_004", + "00000619_c710d414bbbc4340b682c58a_step_005", + "00000620_c710d414bbbc4340b682c58a_step_006", + "00000621_c710d414bbbc4340b682c58a_step_007", + "00000622_c710d414bbbc4340b682c58a_step_008", + "00000624_1eacb269d52a47f5b265c1ab_step_000", + "00000626_f686126d229f47aeb125b5ba_step_000", + "00000627_53bfa7e2e04c4d9dbf4a3db2_step_000", + "00000632_4f322b34e87c43289916c298_step_001", + "00000644_761a451dfea94bfa93e23cfe_step_000", + "00000658_a40da330321b44f286a27361_step_000", + "00000666_6f67a59c16b949a4b66dcfe1_step_000", + "00000683_8863c3497d6d4e05b0f41ddc_step_000", + "00000685_24615f5ff44d4cd5809e77a5_step_001", + "00000686_24615f5ff44d4cd5809e77a5_step_002", + "00000688_b9f3563b403143dbb86ab2b8_step_001", + "00000691_b9f3563b403143dbb86ab2b8_step_004", + "00000693_b9f3563b403143dbb86ab2b8_step_006", + "00000694_45f6100bfa3948e69405bb21_step_000", + "00000696_32309198fce9481896bd0a99_step_000", + "00000698_6f0fc559e8f3422c908b6060_step_001", + "00000699_cae88f0aaaf34de597863810_step_000", + "00000708_c08c1c78b86f4ee3bcb7d9de_step_004", + "00000709_c08c1c78b86f4ee3bcb7d9de_step_005", + "00000710_c08c1c78b86f4ee3bcb7d9de_step_006", + "00000721_9a461e8ae7dc4d33a55883eb_step_000", + "00000723_8ed79b6bcb024d1a9f2aa5e7_step_000", + "00000737_989e0f807cde438face0cb00_step_002", + "00000738_989e0f807cde438face0cb00_step_003", + "00000751_6f0d90054d40400593050930_step_000", + "00000752_6f0d90054d40400593050930_step_001", + "00000764_9cab8335065e4d439bbff471_step_000", + "00000765_9cab8335065e4d439bbff471_step_001", + "00000766_9cab8335065e4d439bbff471_step_002", + "00000767_9cab8335065e4d439bbff471_step_003", + "00000775_a25fb37392a44447b9684f16_step_000", + "00000776_a25fb37392a44447b9684f16_step_001", + "00000777_a25fb37392a44447b9684f16_step_002", + "00000778_a25fb37392a44447b9684f16_step_003", + "00000779_a25fb37392a44447b9684f16_step_004", + "00000780_a25fb37392a44447b9684f16_step_005", + "00000781_a25fb37392a44447b9684f16_step_006", + "00000782_a25fb37392a44447b9684f16_step_007", + "00000783_a25fb37392a44447b9684f16_step_008", + "00000784_a25fb37392a44447b9684f16_step_009", + "00000785_a25fb37392a44447b9684f16_step_010", + "00000786_a25fb37392a44447b9684f16_step_011", + "00000787_a25fb37392a44447b9684f16_step_012", + "00000788_a25fb37392a44447b9684f16_step_013", + "00000789_a25fb37392a44447b9684f16_step_014", + "00000790_a25fb37392a44447b9684f16_step_015", + "00000791_a25fb37392a44447b9684f16_step_016", + "00000792_a25fb37392a44447b9684f16_step_017", + "00000793_a25fb37392a44447b9684f16_step_018", + "00000797_e7533c87b9a347cdafd0c390_step_000", + "00000799_ad6107d9fc994ccca38cc4e4_step_001", + "00000800_ad6107d9fc994ccca38cc4e4_step_002", + "00000801_ad6107d9fc994ccca38cc4e4_step_003", + "00000802_ad6107d9fc994ccca38cc4e4_step_004", + "00000818_6bbad2756d7845bd8d731d60_step_000", + "00000819_6bbad2756d7845bd8d731d60_step_001", + "00000821_6bbad2756d7845bd8d731d60_step_003", + "00000822_c98978894f544f89aa10633f_step_000", + "00000823_c98978894f544f89aa10633f_step_001", + "00000827_b0d209f7a8ad4b02910c317f_step_003", + "00000829_889a065d51a24bb8ad704c69_step_000", + "00000830_889a065d51a24bb8ad704c69_step_001", + "00000832_889a065d51a24bb8ad704c69_step_003", + "00000834_889a065d51a24bb8ad704c69_step_005", + "00000837_b5409201f9a5495fa169af85_step_000", + "00000889_e2d3a4dc5f7c438bb240e9d1_step_000", + "00000890_e2d3a4dc5f7c438bb240e9d1_step_001", + "00000900_4df6ab9924674556bb4a509d_step_000", + "00000904_4df6ab9924674556bb4a509d_step_004", + "00000905_4df6ab9924674556bb4a509d_step_005", + "00000906_4df6ab9924674556bb4a509d_step_006", + "00000942_f297a24e1a4446cf9d74efd9_step_001", + "00000948_f297a24e1a4446cf9d74efd9_step_007", + "00000949_f297a24e1a4446cf9d74efd9_step_008", + "00000953_ccc14f0fcff54b5985cc9113_step_000", + "00000958_04906edf324a4bce8fc3f417_step_000", + "00000961_a98d481268ea4524a7769731_step_000", + "00000962_a98d481268ea4524a7769731_step_001", + "00000963_a98d481268ea4524a7769731_step_002", + "00000973_209426c5054945fab8a4af1b_step_000", + "00000976_b85ec2ff588148c4abd8c8d7_step_000", + "00001024_21adcacb9707488ea6655fd2_step_000", + "00001025_21adcacb9707488ea6655fd2_step_001", + "00001029_cde47df6e28543d1b434f206_step_001", + "00001030_cde47df6e28543d1b434f206_step_002", + "00001031_cde47df6e28543d1b434f206_step_003", + "00001033_18151464e6434759b7c2ece7_step_000", + "00001035_be3ed2c9696e4e6a91b34f79_step_000", + "00001036_be3ed2c9696e4e6a91b34f79_step_001", + "00001041_6baf25f5fe2947e8927797d8_step_000", + "00001048_2b2638f82b3946b1867f2d11_step_004", + "00001049_2b2638f82b3946b1867f2d11_step_005", + "00001050_2b2638f82b3946b1867f2d11_step_006", + "00001061_284effca5a19488fa8a228e2_step_002", + "00001062_c04956ddff6c4324b5cad67c_step_000", + "00001106_ba88e7579ced4a0aa2de629c_step_000", + "00001108_412675e2ea594e70bbd8cf6c_step_000", + "00001114_5f5cf6feacbc41fb88b2d873_step_000", + "00001121_da4037e40839451dbbc2bd1f_step_000", + "00001130_c2db313c2ce749d784c610d2_step_002", + "00001139_0b0ed9b0ccbb455d82c60d69_step_000", + "00001140_0b0ed9b0ccbb455d82c60d69_step_001", + "00001141_0b0ed9b0ccbb455d82c60d69_step_002", + "00001146_274866da478e4a009bed8750_step_001", + "00001159_f112d6f3e0c644ae9de2a724_step_000", + "00001165_4b65bb9a02f6494aa897a53e_step_000", + "00001169_0889c458f3f74dc288d2d531_step_000", + "00001170_0889c458f3f74dc288d2d531_step_001", + "00001182_bc0b5846e3d84904ad02e547_step_001", + "00001185_507b60b57791425bbb73273e_step_000", + "00001186_507b60b57791425bbb73273e_step_001", + "00001187_507b60b57791425bbb73273e_step_002", + "00001190_507b60b57791425bbb73273e_step_005", + "00001191_507b60b57791425bbb73273e_step_006", + "00001193_46b17dd88ed24f55bca2eb69_step_000", + "00001200_e44a0f9afbe84b918f789f0f_step_000", + "00001206_e44a0f9afbe84b918f789f0f_step_006", + "00001210_4be610c137ba42ae9933e69b_step_000", + "00001211_4be610c137ba42ae9933e69b_step_001", + "00001212_4be610c137ba42ae9933e69b_step_002", + "00001213_4be610c137ba42ae9933e69b_step_003", + "00001214_9edb90c9f417497fb70bddf6_step_000", + "00001228_2fd836ae38b44c41bfbd9584_step_002", + "00001232_73b40f9a4cd3419ab3b42b73_step_001", + "00001233_73b40f9a4cd3419ab3b42b73_step_002", + "00001235_17ea801126da4654b34f6c52_step_000", + "00001254_c2da273d137e475abb1ab1f4_step_000", + "00001255_3df9015915ff4fa7b57f0a71_step_000", + "00001257_a4715d363f88400c8f0238d1_step_000", + "00001260_a4715d363f88400c8f0238d1_step_003", + "00001266_b53ece83d8964b44bbf1f8ed_step_002", + "00001273_b53ece83d8964b44bbf1f8ed_step_009", + "00001274_b53ece83d8964b44bbf1f8ed_step_010", + "00001276_b53ece83d8964b44bbf1f8ed_step_012", + "00001285_b53ece83d8964b44bbf1f8ed_step_021", + "00001294_b53ece83d8964b44bbf1f8ed_step_030", + "00001298_b53ece83d8964b44bbf1f8ed_step_034", + "00001300_b9951da825954791b59b979d_step_000", + "00001301_a54ee779467b42ea8709e1ca_step_000", + "00001311_e95b8713cc0b4436909c78e9_step_000", + "00001319_6a5a7dbad03b4a57b33f2c0d_step_002", + "00001320_6a5a7dbad03b4a57b33f2c0d_step_003", + "00001321_7cef9b79a6b04db894fd74e5_step_000", + "00001324_5bf4a74af99f405392406e40_step_000", + "00001344_d8d97699a52842539e08d117_step_002", + "00001345_d8d97699a52842539e08d117_step_003", + "00001346_d8d97699a52842539e08d117_step_004", + "00001348_318b81a8e412438183201be7_step_000", + "00001349_27327ddba7f64417ae1e69a3_step_000", + "00001356_aa0de7c5e9e84d73816bdbde_step_000", + "00001380_c49484307f434da78fa06b1f_step_000", + "00001381_c49484307f434da78fa06b1f_step_001", + "00001389_21b525da345e4e62aebd906b_step_000", + "00001391_406eb61c554a429ebd8a4d4f_step_000", + "00001392_406eb61c554a429ebd8a4d4f_step_001", + "00001393_406eb61c554a429ebd8a4d4f_step_002", + "00001394_406eb61c554a429ebd8a4d4f_step_003", + "00001414_b38f661707674b8eb983d992_step_000", + "00001417_a45f6c99e2154f3b94955175_step_001", + "00001422_a45f6c99e2154f3b94955175_step_006", + "00001427_a45f6c99e2154f3b94955175_step_011", + "00001428_a45f6c99e2154f3b94955175_step_012", + "00001429_a45f6c99e2154f3b94955175_step_013", + "00001430_a45f6c99e2154f3b94955175_step_014", + "00001432_a45f6c99e2154f3b94955175_step_016", + "00001433_a45f6c99e2154f3b94955175_step_017", + "00001434_a45f6c99e2154f3b94955175_step_018", + "00001438_a45f6c99e2154f3b94955175_step_022", + "00001445_a45f6c99e2154f3b94955175_step_029", + "00001447_a45f6c99e2154f3b94955175_step_031", + "00001449_a45f6c99e2154f3b94955175_step_033", + "00001450_a45f6c99e2154f3b94955175_step_034", + "00001453_a45f6c99e2154f3b94955175_step_037", + "00001455_a45f6c99e2154f3b94955175_step_039", + "00001456_a45f6c99e2154f3b94955175_step_040", + "00001459_a45f6c99e2154f3b94955175_step_043", + "00001461_a45f6c99e2154f3b94955175_step_045", + "00001467_a45f6c99e2154f3b94955175_step_051", + "00001468_a45f6c99e2154f3b94955175_step_052", + "00001469_a45f6c99e2154f3b94955175_step_053", + "00001471_a45f6c99e2154f3b94955175_step_055", + "00001472_a45f6c99e2154f3b94955175_step_056", + "00001474_a45f6c99e2154f3b94955175_step_058", + "00001480_a45f6c99e2154f3b94955175_step_064", + "00001481_a45f6c99e2154f3b94955175_step_065", + "00001482_a45f6c99e2154f3b94955175_step_066", + "00001483_a45f6c99e2154f3b94955175_step_067", + "00001484_a45f6c99e2154f3b94955175_step_068", + "00001487_a45f6c99e2154f3b94955175_step_071", + "00001490_6c8ef6d2f5564d139a5077f8_step_001", + "00001491_6c8ef6d2f5564d139a5077f8_step_002", + "00001495_6c8ef6d2f5564d139a5077f8_step_006", + "00001496_eb4e2b01aaca4131bfc64adb_step_000", + "00001497_eb4e2b01aaca4131bfc64adb_step_001", + "00001500_eb4e2b01aaca4131bfc64adb_step_004", + "00001501_6a0f221e85e04411afe23f51_step_000", + "00001503_6a0f221e85e04411afe23f51_step_002", + "00001505_6d02705716e04d9081f4980b_step_000", + "00001527_9214869844f14e83a9dce3ec_step_000", + "00001534_e1d62958d7ca4ad18fefd54d_step_001", + "00001535_e1d62958d7ca4ad18fefd54d_step_002", + "00001538_e1d62958d7ca4ad18fefd54d_step_005", + "00001543_e1d62958d7ca4ad18fefd54d_step_010", + "00001550_a94d978cf62948ce80b417a9_step_000", + "00001554_30de00089a3f4a339eddb68d_step_000", + "00001555_30de00089a3f4a339eddb68d_step_001", + "00001556_30de00089a3f4a339eddb68d_step_002", + "00001557_16c191aef5034779aaab6e3f_step_000", + "00001563_2e9dc7e9a1a54dd7ab901478_step_000", + "00001567_9280741c840d4aa1af32658e_step_002", + "00001569_6fc579f3224d4d53a1b60710_step_000", + "00001578_a3b89691665043a08931897f_step_001", + "00001579_a3b89691665043a08931897f_step_002", + "00001582_a3b89691665043a08931897f_step_005", + "00001586_763da6a559544098acbf4154_step_002", + "00001592_35f4abaacc9447e4ac0b964f_step_001", + "00001594_2128dbbd4bf34879913b3a99_step_000", + "00001596_2128dbbd4bf34879913b3a99_step_002", + "00001597_13bd23e0e4c740969b0c71ab_step_000" + ], + "node_count": 11300812, + "mean": [ + -0.0023328224084797375, + 0.002493796834147401, + -0.001809911687675766, + 0.25742899061541036, + 1.3342682062027138, + 1.8059588831891302, + 3.111110856814303, + 9.224285832960385, + 1.0265793073404939, + -1.221687502467248 + ], + "std": [ + 0.6417830773452925, + 0.5878867083697583, + 0.5840912225689718, + 0.22858594008765615, + 0.6758364464754756, + 2.124656857116319, + 1.0612042542158557, + 2.4681213620763502, + 0.7631566124335305, + 1.6411759487011053 + ], + "columns": [ + "x_star", + "y_star", + "z_star", + "d_over_t", + "gate_distance_over_Rg", + "geodesic_over_Rg", + "log_Rg", + "log_volume", + "log_t_part", + "log_t_set" + ], + "weighting": "equal_learning_nodes", + "ddof": 0, + "constant_column_scale": 1.0, + "fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03" +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0-guard-receipt.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0-guard-receipt.json new file mode 100644 index 000000000..e692f21de --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0-guard-receipt.json @@ -0,0 +1,34 @@ +{ + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "2" + ], + "returncode": 1, + "stop_reason": null, + "cpu_seconds": 97.12944, + "wall_seconds": 96.86777837795671, + "peak_tree_rss_bytes": 2553110528, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 45056, + "max_poll_gap_seconds": 0.4935557850403711, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null, + "device_monitor": { + "peak_device_memory_bytes": 9786359808, + "peak_whole_run_tree_rss_bytes": 2960007168, + "max_poll_gap_seconds": 0.9654930130345747, + "reason": null, + "samples": 165 + }, + "total_run_wall_seconds": 96.91506972297793, + "result_sha256": null +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/best-validation-parts.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/best-validation-parts.json new file mode 100644 index 000000000..2e4124564 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/best-validation-parts.json @@ -0,0 +1,113 @@ +{ + "00000111_ce24e0cbec8e45c89735d148_step_003": 0.21812808632847222, + "00000115_ce24e0cbec8e45c89735d148_step_007": 0.21488521304037625, + "00000116_ce24e0cbec8e45c89735d148_step_008": 0.17777962251397103, + "00000117_ce24e0cbec8e45c89735d148_step_009": 0.19100343642080697, + "00000119_ce24e0cbec8e45c89735d148_step_011": 0.16318215645557835, + "00000120_ce24e0cbec8e45c89735d148_step_012": 0.19328033278939868, + "00000131_945244874ed84e599987daa9_step_000": 0.14524804973282435, + "00000132_438507a38e404bfdbd51a16b_step_000": 0.20675211669069493, + "00000133_08b739e0511f4305b161cd05_step_000": 0.14366315774878777, + "00000190_185ecbc783f34408aef52119_step_000": 0.10053718399728359, + "00000196_c9b71ecdd0654716891b0db8_step_000": 0.15077556794173522, + "00000206_27417003e762409185cb2b69_step_000": 0.14169294882382227, + "00000238_4829bc0645d6414d9b5cec31_step_001": 0.07985537081158034, + "00000241_4829bc0645d6414d9b5cec31_step_004": 0.26754534392202195, + "00000243_4829bc0645d6414d9b5cec31_step_006": 0.17060676739686387, + "00000248_e87554b09cdc48c3add39db9_step_000": 0.21065610669669646, + "00000257_845fb06d7f2f4802a806a600_step_002": 0.16238166542594662, + "00000258_845fb06d7f2f4802a806a600_step_003": 0.15644036417312687, + "00000259_845fb06d7f2f4802a806a600_step_004": 0.1500171512886505, + "00000261_845fb06d7f2f4802a806a600_step_006": 0.17018792567244215, + "00000262_845fb06d7f2f4802a806a600_step_007": 0.12754472626041208, + "00000264_845fb06d7f2f4802a806a600_step_009": 0.24506310649213484, + "00000265_845fb06d7f2f4802a806a600_step_010": 0.16204013838933803, + "00000266_845fb06d7f2f4802a806a600_step_011": 0.20288866150508864, + "00000267_845fb06d7f2f4802a806a600_step_012": 0.1382978783725861, + "00000269_845fb06d7f2f4802a806a600_step_014": 0.18428759394928124, + "00000391_fd490b564a44413884fab719_step_000": 0.09749444822607317, + "00000405_f3759bbb4682423caf207e8e_step_002": 0.1677597743971636, + "00000406_f3759bbb4682423caf207e8e_step_003": 0.1793933995580956, + "00000444_4ed4c78d6d754aac90876fc2_step_003": 0.11000941912106625, + "00000445_4ed4c78d6d754aac90876fc2_step_004": 0.15189124227873596, + "00000501_5499914343bd4f549b38046d_step_002": 0.10594725062210601, + "00000502_5499914343bd4f549b38046d_step_003": 0.1314623093896786, + "00000574_5a4cef68211d4706b1ec8586_step_003": 0.2008506179171846, + "00000578_5a4cef68211d4706b1ec8586_step_007": 0.18766904753300034, + "00000579_5a4cef68211d4706b1ec8586_step_008": 0.25648415326442303, + "00000580_5a4cef68211d4706b1ec8586_step_009": 0.1974406816687746, + "00000582_5a4cef68211d4706b1ec8586_step_011": 0.17623933919537313, + "00000583_5a4cef68211d4706b1ec8586_step_012": 0.26263000839916456, + "00000597_7fe894812c4244e18241aa08_step_003": 0.2356893038876161, + "00000601_7fe894812c4244e18241aa08_step_007": 0.2504062576426059, + "00000602_7fe894812c4244e18241aa08_step_008": 0.15290128577778725, + "00000603_7fe894812c4244e18241aa08_step_009": 0.30358438737774235, + "00000605_7fe894812c4244e18241aa08_step_011": 0.17294695353522754, + "00000606_7fe894812c4244e18241aa08_step_012": 0.2429677048601784, + "00000654_d87d11d9c3ca4ee5a88797ae_step_002": 0.15997686148818385, + "00000758_8ab50540e9804365a8717ec0_step_005": 0.2382338012884456, + "00000759_8ab50540e9804365a8717ec0_step_006": 0.13507237845798767, + "00000760_8ab50540e9804365a8717ec0_step_007": 0.11588133845697132, + "00000763_8ab50540e9804365a8717ec0_step_010": 0.14453895929680835, + "00000816_36e6258346004cabb4c091b2_step_004": 0.14933302625733616, + "00000817_36e6258346004cabb4c091b2_step_005": 0.2644586043777147, + "00000892_be26dc83f78947738b2bfabd_step_001": 0.22566772923718406, + "00000894_be26dc83f78947738b2bfabd_step_003": 0.19538109932672446, + "00000897_be26dc83f78947738b2bfabd_step_006": 0.2030893539508139, + "00000919_cdcebf32644443b9812097ba_step_003": 0.25592018826729906, + "00000923_cdcebf32644443b9812097ba_step_007": 0.21094866127055206, + "00000927_cdcebf32644443b9812097ba_step_011": 0.1662172502947532, + "00000928_cdcebf32644443b9812097ba_step_012": 0.2053027133560218, + "00000955_a741c7966d2c40a8a511263e_step_001": 0.12200340530475978, + "00000984_0dd1386597104265ab67e6cf_step_007": 0.13120701214222596, + "00000986_0dd1386597104265ab67e6cf_step_009": 0.1948588646282406, + "00000987_0dd1386597104265ab67e6cf_step_010": 0.25489492249897944, + "00000988_0dd1386597104265ab67e6cf_step_011": 0.13449257749100815, + "00000996_6d71695ba3fd475c927a00a7_step_000": 0.13053766072371262, + "00001019_ca4cf9693d6d4a98b515ba99_step_000": 0.12308672201424056, + "00001020_ca4cf9693d6d4a98b515ba99_step_001": 0.1495272997136864, + "00001021_ca4cf9693d6d4a98b515ba99_step_002": 0.1288403276869719, + "00001026_0f133d8e7fd448708a554e28_step_000": 0.1000878566182172, + "00001027_0f133d8e7fd448708a554e28_step_001": 0.36149904502919084, + "00001040_74c42207fa764831b03f7658_step_000": 0.1926875561754305, + "00001052_2fd6155fa34f418083d0bd0f_step_001": 0.1292240399443674, + "00001053_2fd6155fa34f418083d0bd0f_step_002": 0.12849428798745785, + "00001097_256ee7ec3a4b4972a8b87c8d_step_001": 0.21977138576736718, + "00001099_256ee7ec3a4b4972a8b87c8d_step_003": 0.1816795026853074, + "00001102_256ee7ec3a4b4972a8b87c8d_step_006": 0.11005911272515699, + "00001107_78566795c3144719804f8d34_step_000": 0.1891754576869961, + "00001167_107f61d0402f4c9a9d9d1342_step_000": 0.1711592378983605, + "00001168_107f61d0402f4c9a9d9d1342_step_001": 0.17882064334059392, + "00001217_e517c61f1c49433bb81cf980_step_000": 0.2515738972525035, + "00001238_96503b6a26c64074ad598927_step_000": 0.13292517911704352, + "00001239_96503b6a26c64074ad598927_step_001": 0.132587531407208, + "00001240_96503b6a26c64074ad598927_step_002": 0.0813628344361759, + "00001241_96503b6a26c64074ad598927_step_003": 0.29285063745909695, + "00001242_96503b6a26c64074ad598927_step_004": 0.0925634226945209, + "00001243_96503b6a26c64074ad598927_step_005": 0.09509165761281989, + "00001244_96503b6a26c64074ad598927_step_006": 0.1230585602126056, + "00001246_96503b6a26c64074ad598927_step_008": 0.14539551492569036, + "00001247_96503b6a26c64074ad598927_step_009": 0.13874208094725327, + "00001249_96503b6a26c64074ad598927_step_011": 0.2629669559138016, + "00001252_96503b6a26c64074ad598927_step_014": 0.13505092638975305, + "00001253_96503b6a26c64074ad598927_step_015": 0.24404680008109297, + "00001314_a9ed5e56a6a8414f91b39bdb_step_000": 0.31519282188702236, + "00001333_beab79941e884455a62af40c_step_001": 0.22630092547626443, + "00001334_beab79941e884455a62af40c_step_002": 0.24975399518081012, + "00001335_beab79941e884455a62af40c_step_003": 0.3232102320909878, + "00001336_beab79941e884455a62af40c_step_004": 0.2503703092650181, + "00001338_beab79941e884455a62af40c_step_006": 0.3354064339073836, + "00001339_beab79941e884455a62af40c_step_007": 0.19392859368744472, + "00001340_beab79941e884455a62af40c_step_008": 0.1946445864916501, + "00001352_4e1d56bc2cef49ea84626b11_step_000": 0.13524274531754865, + "00001353_4e1d56bc2cef49ea84626b11_step_001": 0.1420727389817591, + "00001354_4e1d56bc2cef49ea84626b11_step_002": 0.144701506201183, + "00001379_5b1d6e2599ea47d498a04256_step_000": 0.16561272764961235, + "00001387_60cf3ad636dd42e0bcec6946_step_000": 0.1480953559754164, + "00001511_5cbe58077e4b4885a5c213b3_step_004": 0.17201384774359854, + "00001512_5cbe58077e4b4885a5c213b3_step_005": 0.19464725873827227, + "00001525_e0d35a03b46745259cf8878f_step_000": 0.22782073658429386, + "00001532_9f0a4a641e8f4cc588eb4315_step_000": 0.1454125834548101, + "00001552_b0a9c4d836294846891615fe_step_000": 0.20275123535956904, + "00001553_adfa1c33ae4d4d0aa441ddc3_step_000": 0.1663017021836491 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/best.pt b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/best.pt new file mode 100644 index 000000000..ba1fcaa3d Binary files /dev/null and b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/best.pt differ diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-000-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-000-amp.json new file mode 100644 index 000000000..f66f9a3d7 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-000-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-001-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-001-amp.json new file mode 100644 index 000000000..00507dd40 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-001-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-002-amp.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-002-amp.json new file mode 100644 index 000000000..42c06ca50 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-002-amp.json @@ -0,0 +1,7 @@ +{ + "attempted_steps": 382, + "skipped_steps": 4, + "successful_updates": 378, + "skipped_fraction": 0.010471204188481676, + "consecutive_skipped": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/failure.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/failure.json new file mode 100644 index 000000000..e0ffc4108 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/failure.json @@ -0,0 +1,15 @@ +{ + "status": "FAILED_NO_RETRY", + "error_type": "FloatingPointError", + "message": "AMP skipped steps exceed 1 percent after epoch 0", + "traceback": "Traceback (most recent call last):\n File \"/home/frank14f/optane/MeltMind/tools/run_s2_pilot_training_v2.py\", line 113, in run\n skip_policy.finish_epoch(epoch)\n File \"/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py\", line 52, in finish_epoch\n if epoch>0 and self.skipped>self.attempted*.01:raise FloatingPointError('AMP skipped steps exceed 1 percent after epoch 0')\nFloatingPointError: AMP skipped steps exceed 1 percent after epoch 0\n", + "wall_seconds": 93.35228163201828, + "epoch": 2, + "amp": { + "attempted_steps": 382, + "skipped_steps": 4, + "successful_updates": 378, + "skipped_fraction": 0.010471204188481676, + "consecutive_skipped": 0 + } +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/history.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/history.json new file mode 100644 index 000000000..e1537ed73 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/history.json @@ -0,0 +1,36 @@ +[ + { + "epoch": 0, + "amp": { + "attempted_steps": 382, + "skipped_steps": 1, + "successful_updates": 381, + "skipped_fraction": 0.002617801047120419, + "consecutive_skipped": 0 + }, + "attempted_updates": 382, + "training_global_weighted_mse": 0.03836522725622865, + "validation_component_equal_relative_l2": 0.1710376600828109, + "updates": 381, + "elapsed_seconds": 36.435720991983544, + "peak_allocated_bytes": 7525132800, + "peak_reserved_bytes": 16416505856 + }, + { + "epoch": 1, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 764, + "training_global_weighted_mse": 0.01606502592664501, + "validation_component_equal_relative_l2": 0.22933832083575456, + "updates": 763, + "elapsed_seconds": 67.04866537201451, + "peak_allocated_bytes": 7525132800, + "peak_reserved_bytes": 16416505856 + } +] diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/progress.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/progress.json new file mode 100644 index 000000000..d8259ecfc --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/progress.json @@ -0,0 +1,10 @@ +{ + "stage": "training", + "epoch": 2, + "parts_done": 369, + "parts_total": 382, + "updates": 1128, + "elapsed_seconds": 92.62161607301096, + "peak_allocated_bytes": 7525132800, + "peak_reserved_bytes": 16416505856 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/standardizer.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/standardizer.json new file mode 100644 index 000000000..11716fd2e --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/standardizer.json @@ -0,0 +1,429 @@ +{ + "fold": "0", + "pool": "100", + "learning_part_ids": [ + "00000003_1ffb81a71e5b402e966b9341_step_002", + "00000004_1ffb81a71e5b402e966b9341_step_003", + "00000021_ad34a3f60c4a4caa99646600_step_001", + "00000023_ad34a3f60c4a4caa99646600_step_003", + "00000024_ad34a3f60c4a4caa99646600_step_004", + "00000030_ad34a3f60c4a4caa99646600_step_010", + "00000031_ad34a3f60c4a4caa99646600_step_011", + "00000033_ad34a3f60c4a4caa99646600_step_013", + "00000049_aa208c45e8734584a147f595_step_000", + "00000057_666139e3bff64d4e8a6ce183_step_006", + "00000058_666139e3bff64d4e8a6ce183_step_007", + "00000060_767e4372b5f94a88a7a17d90_step_001", + "00000061_767e4372b5f94a88a7a17d90_step_002", + "00000065_767e4372b5f94a88a7a17d90_step_006", + "00000066_767e4372b5f94a88a7a17d90_step_007", + "00000067_767e4372b5f94a88a7a17d90_step_008", + "00000068_767e4372b5f94a88a7a17d90_step_009", + "00000070_767e4372b5f94a88a7a17d90_step_011", + "00000072_767e4372b5f94a88a7a17d90_step_013", + "00000077_767e4372b5f94a88a7a17d90_step_018", + "00000078_767e4372b5f94a88a7a17d90_step_019", + "00000079_767e4372b5f94a88a7a17d90_step_020", + "00000087_b4f9335a60a14cecaa599159_step_002", + "00000088_169bec34317b46e98eb1a403_step_000", + "00000093_169bec34317b46e98eb1a403_step_005", + "00000098_72cf0f9ca5324822a4d41a80_step_000", + "00000103_52f7124261ae4c558f800775_step_000", + "00000104_52f7124261ae4c558f800775_step_001", + "00000105_52f7124261ae4c558f800775_step_002", + "00000106_52f7124261ae4c558f800775_step_003", + "00000125_e18c7feacfb54f3880659b43_step_000", + "00000128_e18c7feacfb54f3880659b43_step_003", + "00000129_e18c7feacfb54f3880659b43_step_004", + "00000130_e18c7feacfb54f3880659b43_step_005", + "00000140_2fc54fcd110d4f49969163c4_step_003", + "00000142_2fc54fcd110d4f49969163c4_step_005", + "00000144_2fc54fcd110d4f49969163c4_step_007", + "00000145_2fc54fcd110d4f49969163c4_step_008", + "00000152_a951550b8a1043a68423c541_step_002", + "00000153_a951550b8a1043a68423c541_step_003", + "00000154_a951550b8a1043a68423c541_step_004", + "00000157_8208fd8b44b64dce91322f84_step_000", + "00000161_a583c6c3cdc143bab3f6c856_step_000", + "00000162_4dcda9fab6dd46cf98fb194e_step_000", + "00000163_ff9b2ed255484db78bb0e6da_step_000", + "00000164_91adbf180e114b698160c9a4_step_000", + "00000165_4070a99d6ebe45f5bd5d71eb_step_000", + "00000166_2246cab58dbe4a2fb00fed50_step_000", + "00000170_b21aa794b6b64d87a57c245e_step_000", + "00000171_b21aa794b6b64d87a57c245e_step_001", + "00000173_b21aa794b6b64d87a57c245e_step_003", + "00000174_b21aa794b6b64d87a57c245e_step_004", + "00000175_b21aa794b6b64d87a57c245e_step_005", + "00000176_b21aa794b6b64d87a57c245e_step_006", + "00000181_b21aa794b6b64d87a57c245e_step_011", + "00000198_826494e894454504b0971fd1_step_001", + "00000223_97b354a907114b7183faa9c4_step_001", + "00000224_97b354a907114b7183faa9c4_step_002", + "00000229_682d529d5e8c4387bc59b3a1_step_000", + "00000250_641f03cbe6a145508c6ae055_step_000", + "00000272_648a7c23a21f4e88bf25ffeb_step_000", + "00000273_648a7c23a21f4e88bf25ffeb_step_001", + "00000274_648a7c23a21f4e88bf25ffeb_step_002", + "00000275_648a7c23a21f4e88bf25ffeb_step_003", + "00000276_648a7c23a21f4e88bf25ffeb_step_004", + "00000280_5a6ef3285b89452ea6b390e7_step_001", + "00000282_5a6ef3285b89452ea6b390e7_step_003", + "00000285_d215b7bc60ae468288e5f1ea_step_000", + "00000287_58f8ff46b7564e989f9695b7_step_000", + "00000290_58f8ff46b7564e989f9695b7_step_003", + "00000291_58f8ff46b7564e989f9695b7_step_004", + "00000293_58f8ff46b7564e989f9695b7_step_006", + "00000295_52a153d2417747b5926ff9df_step_000", + "00000296_52a153d2417747b5926ff9df_step_001", + "00000301_52a153d2417747b5926ff9df_step_006", + "00000302_39b08d62a677414f86fdae52_step_000", + "00000305_3062bccff48e47a2b9de05e3_step_000", + "00000306_3062bccff48e47a2b9de05e3_step_001", + "00000307_3062bccff48e47a2b9de05e3_step_002", + "00000308_3062bccff48e47a2b9de05e3_step_003", + "00000309_3062bccff48e47a2b9de05e3_step_004", + "00000310_3062bccff48e47a2b9de05e3_step_005", + "00000311_3062bccff48e47a2b9de05e3_step_006", + "00000312_3062bccff48e47a2b9de05e3_step_007", + "00000314_3062bccff48e47a2b9de05e3_step_009", + "00000315_3062bccff48e47a2b9de05e3_step_010", + "00000316_3062bccff48e47a2b9de05e3_step_011", + "00000317_3062bccff48e47a2b9de05e3_step_012", + "00000318_3062bccff48e47a2b9de05e3_step_013", + "00000319_3062bccff48e47a2b9de05e3_step_014", + "00000321_3062bccff48e47a2b9de05e3_step_016", + "00000322_3062bccff48e47a2b9de05e3_step_017", + "00000323_3062bccff48e47a2b9de05e3_step_018", + "00000324_3062bccff48e47a2b9de05e3_step_019", + "00000325_3062bccff48e47a2b9de05e3_step_020", + "00000329_12a935cd9b684f879c0505fe_step_000", + "00000330_12a935cd9b684f879c0505fe_step_001", + "00000331_12a935cd9b684f879c0505fe_step_002", + "00000336_1a821c3883f7459a9615fca4_step_000", + "00000337_153efbd85ca64c4fb6b7ed5f_step_000", + "00000346_4185972a944744d8a7a0f2b4_step_002", + "00000348_4185972a944744d8a7a0f2b4_step_004", + "00000352_4185972a944744d8a7a0f2b4_step_008", + "00000355_68d1a5c2319a4a419ff625c6_step_001", + "00000356_68d1a5c2319a4a419ff625c6_step_002", + "00000357_68d1a5c2319a4a419ff625c6_step_003", + "00000364_38d899e2706a4d09960705dd_step_001", + "00000367_23c957e8e7b2428282a13ae2_step_000", + "00000368_23c957e8e7b2428282a13ae2_step_001", + "00000369_23c957e8e7b2428282a13ae2_step_002", + "00000374_23c957e8e7b2428282a13ae2_step_007", + "00000375_23c957e8e7b2428282a13ae2_step_008", + "00000376_23c957e8e7b2428282a13ae2_step_009", + "00000377_23c957e8e7b2428282a13ae2_step_010", + "00000378_23c957e8e7b2428282a13ae2_step_011", + "00000379_239bbf0e92974a32bd214366_step_000", + "00000382_239bbf0e92974a32bd214366_step_003", + "00000383_239bbf0e92974a32bd214366_step_004", + "00000386_239bbf0e92974a32bd214366_step_007", + "00000387_239bbf0e92974a32bd214366_step_008", + "00000388_239bbf0e92974a32bd214366_step_009", + "00000418_ce5b31679ee845a69ba89d6f_step_000", + "00000422_8e8a96d437584c77b3cfd84f_step_000", + "00000423_8e8a96d437584c77b3cfd84f_step_001", + "00000424_8e8a96d437584c77b3cfd84f_step_002", + "00000459_516b1784884d4facbc4c0d74_step_007", + "00000462_516b1784884d4facbc4c0d74_step_010", + "00000466_11951e51b89249dba38ded3e_step_001", + "00000469_11951e51b89249dba38ded3e_step_004", + "00000470_11951e51b89249dba38ded3e_step_005", + "00000474_a5639f2b84c445aea5da18ad_step_000", + "00000477_0fc8cc17c6af48dabe36c8f2_step_002", + "00000478_0fc8cc17c6af48dabe36c8f2_step_003", + "00000479_0fc8cc17c6af48dabe36c8f2_step_004", + "00000485_a7d0c851b5f24a97a7cac950_step_002", + "00000488_a7d0c851b5f24a97a7cac950_step_005", + "00000489_a7d0c851b5f24a97a7cac950_step_006", + "00000492_a7d0c851b5f24a97a7cac950_step_009", + "00000513_f78bf7cf156f4231988af062_step_001", + "00000519_5b8fa6b193d14279a77a0e50_step_000", + "00000523_1f8d7f343e634c11a80d23ac_step_000", + "00000526_f9d6559308dc4c85aab9fa38_step_000", + "00000530_bc0d6937206b43bc96521413_step_002", + "00000531_bc0d6937206b43bc96521413_step_003", + "00000543_8cf6554d440d40c4823dad6c_step_000", + "00000559_a1621e156ade4317ab6a5bc5_step_000", + "00000565_5b5119548f7b421e8f597438_step_000", + "00000587_2417b410de78416ebe50d460_step_000", + "00000589_2417b410de78416ebe50d460_step_002", + "00000590_2417b410de78416ebe50d460_step_003", + "00000591_2417b410de78416ebe50d460_step_004", + "00000609_04e799936ae446b9a3d8d2b8_step_000", + "00000610_04e799936ae446b9a3d8d2b8_step_001", + "00000611_04e799936ae446b9a3d8d2b8_step_002", + "00000613_189d1310aa8c4a2a92a21020_step_001", + "00000615_c710d414bbbc4340b682c58a_step_001", + "00000616_c710d414bbbc4340b682c58a_step_002", + "00000617_c710d414bbbc4340b682c58a_step_003", + "00000618_c710d414bbbc4340b682c58a_step_004", + "00000619_c710d414bbbc4340b682c58a_step_005", + "00000620_c710d414bbbc4340b682c58a_step_006", + "00000621_c710d414bbbc4340b682c58a_step_007", + "00000622_c710d414bbbc4340b682c58a_step_008", + "00000624_1eacb269d52a47f5b265c1ab_step_000", + "00000626_f686126d229f47aeb125b5ba_step_000", + "00000627_53bfa7e2e04c4d9dbf4a3db2_step_000", + "00000632_4f322b34e87c43289916c298_step_001", + "00000644_761a451dfea94bfa93e23cfe_step_000", + "00000658_a40da330321b44f286a27361_step_000", + "00000666_6f67a59c16b949a4b66dcfe1_step_000", + "00000683_8863c3497d6d4e05b0f41ddc_step_000", + "00000685_24615f5ff44d4cd5809e77a5_step_001", + "00000686_24615f5ff44d4cd5809e77a5_step_002", + "00000688_b9f3563b403143dbb86ab2b8_step_001", + "00000691_b9f3563b403143dbb86ab2b8_step_004", + "00000693_b9f3563b403143dbb86ab2b8_step_006", + "00000694_45f6100bfa3948e69405bb21_step_000", + "00000696_32309198fce9481896bd0a99_step_000", + "00000698_6f0fc559e8f3422c908b6060_step_001", + "00000699_cae88f0aaaf34de597863810_step_000", + "00000708_c08c1c78b86f4ee3bcb7d9de_step_004", + "00000709_c08c1c78b86f4ee3bcb7d9de_step_005", + "00000710_c08c1c78b86f4ee3bcb7d9de_step_006", + "00000721_9a461e8ae7dc4d33a55883eb_step_000", + "00000723_8ed79b6bcb024d1a9f2aa5e7_step_000", + "00000737_989e0f807cde438face0cb00_step_002", + "00000738_989e0f807cde438face0cb00_step_003", + "00000751_6f0d90054d40400593050930_step_000", + "00000752_6f0d90054d40400593050930_step_001", + "00000764_9cab8335065e4d439bbff471_step_000", + "00000765_9cab8335065e4d439bbff471_step_001", + "00000766_9cab8335065e4d439bbff471_step_002", + "00000767_9cab8335065e4d439bbff471_step_003", + "00000775_a25fb37392a44447b9684f16_step_000", + "00000776_a25fb37392a44447b9684f16_step_001", + "00000777_a25fb37392a44447b9684f16_step_002", + "00000778_a25fb37392a44447b9684f16_step_003", + "00000779_a25fb37392a44447b9684f16_step_004", + "00000780_a25fb37392a44447b9684f16_step_005", + "00000781_a25fb37392a44447b9684f16_step_006", + "00000782_a25fb37392a44447b9684f16_step_007", + "00000783_a25fb37392a44447b9684f16_step_008", + "00000784_a25fb37392a44447b9684f16_step_009", + "00000785_a25fb37392a44447b9684f16_step_010", + "00000786_a25fb37392a44447b9684f16_step_011", + "00000787_a25fb37392a44447b9684f16_step_012", + "00000788_a25fb37392a44447b9684f16_step_013", + "00000789_a25fb37392a44447b9684f16_step_014", + "00000790_a25fb37392a44447b9684f16_step_015", + "00000791_a25fb37392a44447b9684f16_step_016", + "00000792_a25fb37392a44447b9684f16_step_017", + "00000793_a25fb37392a44447b9684f16_step_018", + "00000797_e7533c87b9a347cdafd0c390_step_000", + "00000799_ad6107d9fc994ccca38cc4e4_step_001", + "00000800_ad6107d9fc994ccca38cc4e4_step_002", + "00000801_ad6107d9fc994ccca38cc4e4_step_003", + "00000802_ad6107d9fc994ccca38cc4e4_step_004", + "00000818_6bbad2756d7845bd8d731d60_step_000", + "00000819_6bbad2756d7845bd8d731d60_step_001", + "00000821_6bbad2756d7845bd8d731d60_step_003", + "00000822_c98978894f544f89aa10633f_step_000", + "00000823_c98978894f544f89aa10633f_step_001", + "00000827_b0d209f7a8ad4b02910c317f_step_003", + "00000829_889a065d51a24bb8ad704c69_step_000", + "00000830_889a065d51a24bb8ad704c69_step_001", + "00000832_889a065d51a24bb8ad704c69_step_003", + "00000834_889a065d51a24bb8ad704c69_step_005", + "00000837_b5409201f9a5495fa169af85_step_000", + "00000889_e2d3a4dc5f7c438bb240e9d1_step_000", + "00000890_e2d3a4dc5f7c438bb240e9d1_step_001", + "00000900_4df6ab9924674556bb4a509d_step_000", + "00000904_4df6ab9924674556bb4a509d_step_004", + "00000905_4df6ab9924674556bb4a509d_step_005", + "00000906_4df6ab9924674556bb4a509d_step_006", + "00000942_f297a24e1a4446cf9d74efd9_step_001", + "00000948_f297a24e1a4446cf9d74efd9_step_007", + "00000949_f297a24e1a4446cf9d74efd9_step_008", + "00000953_ccc14f0fcff54b5985cc9113_step_000", + "00000958_04906edf324a4bce8fc3f417_step_000", + "00000961_a98d481268ea4524a7769731_step_000", + "00000962_a98d481268ea4524a7769731_step_001", + "00000963_a98d481268ea4524a7769731_step_002", + "00000973_209426c5054945fab8a4af1b_step_000", + "00000976_b85ec2ff588148c4abd8c8d7_step_000", + "00001024_21adcacb9707488ea6655fd2_step_000", + "00001025_21adcacb9707488ea6655fd2_step_001", + "00001029_cde47df6e28543d1b434f206_step_001", + "00001030_cde47df6e28543d1b434f206_step_002", + "00001031_cde47df6e28543d1b434f206_step_003", + "00001033_18151464e6434759b7c2ece7_step_000", + "00001035_be3ed2c9696e4e6a91b34f79_step_000", + "00001036_be3ed2c9696e4e6a91b34f79_step_001", + "00001041_6baf25f5fe2947e8927797d8_step_000", + "00001048_2b2638f82b3946b1867f2d11_step_004", + "00001049_2b2638f82b3946b1867f2d11_step_005", + "00001050_2b2638f82b3946b1867f2d11_step_006", + "00001061_284effca5a19488fa8a228e2_step_002", + "00001062_c04956ddff6c4324b5cad67c_step_000", + "00001106_ba88e7579ced4a0aa2de629c_step_000", + "00001108_412675e2ea594e70bbd8cf6c_step_000", + "00001114_5f5cf6feacbc41fb88b2d873_step_000", + "00001121_da4037e40839451dbbc2bd1f_step_000", + "00001130_c2db313c2ce749d784c610d2_step_002", + "00001139_0b0ed9b0ccbb455d82c60d69_step_000", + "00001140_0b0ed9b0ccbb455d82c60d69_step_001", + "00001141_0b0ed9b0ccbb455d82c60d69_step_002", + "00001146_274866da478e4a009bed8750_step_001", + "00001159_f112d6f3e0c644ae9de2a724_step_000", + "00001165_4b65bb9a02f6494aa897a53e_step_000", + "00001169_0889c458f3f74dc288d2d531_step_000", + "00001170_0889c458f3f74dc288d2d531_step_001", + "00001182_bc0b5846e3d84904ad02e547_step_001", + "00001185_507b60b57791425bbb73273e_step_000", + "00001186_507b60b57791425bbb73273e_step_001", + "00001187_507b60b57791425bbb73273e_step_002", + "00001190_507b60b57791425bbb73273e_step_005", + "00001191_507b60b57791425bbb73273e_step_006", + "00001193_46b17dd88ed24f55bca2eb69_step_000", + "00001200_e44a0f9afbe84b918f789f0f_step_000", + "00001206_e44a0f9afbe84b918f789f0f_step_006", + "00001210_4be610c137ba42ae9933e69b_step_000", + "00001211_4be610c137ba42ae9933e69b_step_001", + "00001212_4be610c137ba42ae9933e69b_step_002", + "00001213_4be610c137ba42ae9933e69b_step_003", + "00001214_9edb90c9f417497fb70bddf6_step_000", + "00001228_2fd836ae38b44c41bfbd9584_step_002", + "00001232_73b40f9a4cd3419ab3b42b73_step_001", + "00001233_73b40f9a4cd3419ab3b42b73_step_002", + "00001235_17ea801126da4654b34f6c52_step_000", + "00001254_c2da273d137e475abb1ab1f4_step_000", + "00001255_3df9015915ff4fa7b57f0a71_step_000", + "00001257_a4715d363f88400c8f0238d1_step_000", + "00001260_a4715d363f88400c8f0238d1_step_003", + "00001266_b53ece83d8964b44bbf1f8ed_step_002", + "00001273_b53ece83d8964b44bbf1f8ed_step_009", + "00001274_b53ece83d8964b44bbf1f8ed_step_010", + "00001276_b53ece83d8964b44bbf1f8ed_step_012", + "00001285_b53ece83d8964b44bbf1f8ed_step_021", + "00001294_b53ece83d8964b44bbf1f8ed_step_030", + "00001298_b53ece83d8964b44bbf1f8ed_step_034", + "00001300_b9951da825954791b59b979d_step_000", + "00001301_a54ee779467b42ea8709e1ca_step_000", + "00001311_e95b8713cc0b4436909c78e9_step_000", + "00001319_6a5a7dbad03b4a57b33f2c0d_step_002", + "00001320_6a5a7dbad03b4a57b33f2c0d_step_003", + "00001321_7cef9b79a6b04db894fd74e5_step_000", + "00001324_5bf4a74af99f405392406e40_step_000", + "00001344_d8d97699a52842539e08d117_step_002", + "00001345_d8d97699a52842539e08d117_step_003", + "00001346_d8d97699a52842539e08d117_step_004", + "00001348_318b81a8e412438183201be7_step_000", + "00001349_27327ddba7f64417ae1e69a3_step_000", + "00001356_aa0de7c5e9e84d73816bdbde_step_000", + "00001380_c49484307f434da78fa06b1f_step_000", + "00001381_c49484307f434da78fa06b1f_step_001", + "00001389_21b525da345e4e62aebd906b_step_000", + "00001391_406eb61c554a429ebd8a4d4f_step_000", + "00001392_406eb61c554a429ebd8a4d4f_step_001", + "00001393_406eb61c554a429ebd8a4d4f_step_002", + "00001394_406eb61c554a429ebd8a4d4f_step_003", + "00001414_b38f661707674b8eb983d992_step_000", + "00001417_a45f6c99e2154f3b94955175_step_001", + "00001422_a45f6c99e2154f3b94955175_step_006", + "00001427_a45f6c99e2154f3b94955175_step_011", + "00001428_a45f6c99e2154f3b94955175_step_012", + "00001429_a45f6c99e2154f3b94955175_step_013", + "00001430_a45f6c99e2154f3b94955175_step_014", + "00001432_a45f6c99e2154f3b94955175_step_016", + "00001433_a45f6c99e2154f3b94955175_step_017", + "00001434_a45f6c99e2154f3b94955175_step_018", + "00001438_a45f6c99e2154f3b94955175_step_022", + "00001445_a45f6c99e2154f3b94955175_step_029", + "00001447_a45f6c99e2154f3b94955175_step_031", + "00001449_a45f6c99e2154f3b94955175_step_033", + "00001450_a45f6c99e2154f3b94955175_step_034", + "00001453_a45f6c99e2154f3b94955175_step_037", + "00001455_a45f6c99e2154f3b94955175_step_039", + "00001456_a45f6c99e2154f3b94955175_step_040", + "00001459_a45f6c99e2154f3b94955175_step_043", + "00001461_a45f6c99e2154f3b94955175_step_045", + "00001467_a45f6c99e2154f3b94955175_step_051", + "00001468_a45f6c99e2154f3b94955175_step_052", + "00001469_a45f6c99e2154f3b94955175_step_053", + "00001471_a45f6c99e2154f3b94955175_step_055", + "00001472_a45f6c99e2154f3b94955175_step_056", + "00001474_a45f6c99e2154f3b94955175_step_058", + "00001480_a45f6c99e2154f3b94955175_step_064", + "00001481_a45f6c99e2154f3b94955175_step_065", + "00001482_a45f6c99e2154f3b94955175_step_066", + "00001483_a45f6c99e2154f3b94955175_step_067", + "00001484_a45f6c99e2154f3b94955175_step_068", + "00001487_a45f6c99e2154f3b94955175_step_071", + "00001490_6c8ef6d2f5564d139a5077f8_step_001", + "00001491_6c8ef6d2f5564d139a5077f8_step_002", + "00001495_6c8ef6d2f5564d139a5077f8_step_006", + "00001496_eb4e2b01aaca4131bfc64adb_step_000", + "00001497_eb4e2b01aaca4131bfc64adb_step_001", + "00001500_eb4e2b01aaca4131bfc64adb_step_004", + "00001501_6a0f221e85e04411afe23f51_step_000", + "00001503_6a0f221e85e04411afe23f51_step_002", + "00001505_6d02705716e04d9081f4980b_step_000", + "00001527_9214869844f14e83a9dce3ec_step_000", + "00001534_e1d62958d7ca4ad18fefd54d_step_001", + "00001535_e1d62958d7ca4ad18fefd54d_step_002", + "00001538_e1d62958d7ca4ad18fefd54d_step_005", + "00001543_e1d62958d7ca4ad18fefd54d_step_010", + "00001550_a94d978cf62948ce80b417a9_step_000", + "00001554_30de00089a3f4a339eddb68d_step_000", + "00001555_30de00089a3f4a339eddb68d_step_001", + "00001556_30de00089a3f4a339eddb68d_step_002", + "00001557_16c191aef5034779aaab6e3f_step_000", + "00001563_2e9dc7e9a1a54dd7ab901478_step_000", + "00001567_9280741c840d4aa1af32658e_step_002", + "00001569_6fc579f3224d4d53a1b60710_step_000", + "00001578_a3b89691665043a08931897f_step_001", + "00001579_a3b89691665043a08931897f_step_002", + "00001582_a3b89691665043a08931897f_step_005", + "00001586_763da6a559544098acbf4154_step_002", + "00001592_35f4abaacc9447e4ac0b964f_step_001", + "00001594_2128dbbd4bf34879913b3a99_step_000", + "00001596_2128dbbd4bf34879913b3a99_step_002", + "00001597_13bd23e0e4c740969b0c71ab_step_000" + ], + "node_count": 11300812, + "mean": [ + -0.0023328224084797375, + 0.002493796834147401, + -0.001809911687675766, + 0.25742899061541036, + 1.3342682062027138, + 1.8059588831891302, + 3.111110856814303, + 9.224285832960385, + 1.0265793073404939, + -1.221687502467248 + ], + "std": [ + 0.6417830773452925, + 0.5878867083697583, + 0.5840912225689718, + 0.22858594008765615, + 0.6758364464754756, + 2.124656857116319, + 1.0612042542158557, + 2.4681213620763502, + 0.7631566124335305, + 1.6411759487011053 + ], + "columns": [ + "x_star", + "y_star", + "z_star", + "d_over_t", + "gate_distance_over_Rg", + "geodesic_over_Rg", + "log_Rg", + "log_volume", + "log_t_part", + "log_t_set" + ], + "weighting": "equal_learning_nodes", + "ddof": 0, + "constant_column_scale": 1.0, + "fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03" +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/started.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/started.json new file mode 100644 index 000000000..0a0835dc3 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/started.json @@ -0,0 +1,4 @@ +{ + "started_unix": 1794697985.3034842, + "contract_sha256": "f6eaa3cb2fd6918e5bd940826a2cc954ffcee0a9a2d1578b2da1e5e1c72abd5c" +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/complete.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/complete.json new file mode 100644 index 000000000..9ea528953 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/complete.json @@ -0,0 +1,10 @@ +{ + "status": "PILOT_INCOMPLETE", + "selected_global_width": null, + "cumulative_v2_wall_seconds": 2439.912972155027, + "prior_wall_seconds": 11.143491394002922, + "limit_seconds": 7200, + "probe_reused": true, + "probe_rerun": false, + "test_labels_read": false +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.log b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.log new file mode 100644 index 000000000..4c439afed --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.log @@ -0,0 +1 @@ +{"status": "COMPLETE", "run": {"name": "E128-seed0", "arm": "E", "width": 128, "seed": 0}, "E_max": 42, "cosine_T_max": 42, "epochs": 42, "attempted_updates": 16044, "updates": 16038, "best_epoch": 39, "best_validation_score": 0.10435484679758687, "learning_parts": 382, "validation_parts": 111, "learning_components": 137, "validation_components": 35, "failed_preparation_parts_dropped": [{"case": "00000575_5a4cef68211d4706b1ec8586_step_004", "role": "validation", "nodes": 75046, "component_id": "component-3a3e340f4dfac391", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-inputs-2026-11-15", "181"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.766116, "wall_seconds": 1800.400393687014, "peak_tree_rss_bytes": 602218496, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.128913332009688, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false}, "failure": null}, {"case": "00000614_c710d414bbbc4340b682c58a_step_000", "role": "learning", "nodes": 145860, "component_id": "component-4bd089778094e746", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-inputs-2026-11-15", "201"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.796751, "wall_seconds": 1800.0567526929663, "peak_tree_rss_bytes": 629305344, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.1576595069491304, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false}, "failure": null}, {"case": "00000625_6af98f5458b347bb843f0562_step_000", "role": "learning", "nodes": 145707, "component_id": "component-4bd089778094e746", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-inputs-2026-11-15", "211"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.8864629999998, "wall_seconds": 1800.119237862993, "peak_tree_rss_bytes": 647880704, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.159920622012578, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false}, "failure": null}, {"case": "00001466_a45f6c99e2154f3b94955175_step_050", "role": "learning", "nodes": 77138, "component_id": "component-7fe5173e77c86f9d", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-recovery-inputs-2026-11-15", "83"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.926194, "wall_seconds": 1800.0714844280155, "peak_tree_rss_bytes": 383070208, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.0109735240112059, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false, "monitor_errors": 0, "last_monitor_error": null}, "failure": null}, {"case": "00001470_a45f6c99e2154f3b94955175_step_054", "role": "learning", "nodes": 124817, "component_id": "component-7fe5173e77c86f9d", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-recovery-inputs-2026-11-15", "87"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.664371, "wall_seconds": 1800.0443400070071, "peak_tree_rss_bytes": 578732032, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.007104828953743, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false, "monitor_errors": 0, "last_monitor_error": null}, "failure": null}, {"case": "00001473_a45f6c99e2154f3b94955175_step_057", "role": "learning", "nodes": 69428, "component_id": "component-7fe5173e77c86f9d", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-recovery-inputs-2026-11-15", "90"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.751432, "wall_seconds": 1800.2498029720155, "peak_tree_rss_bytes": 438853632, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 0.8929723129840568, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false, "monitor_errors": 0, "last_monitor_error": null}, "failure": null}], "standardizer_fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03", "wall_seconds": 1117.921224004007, "peak_allocated_bytes": 4706630656, "peak_reserved_bytes": 8243904512, "test_labels_read": false, "test_predictions": false, "capacity_selected": false, "checkpoint_sha256": "910437bd645a965241786a90ad14325fe7af09092238bfd0be5ee5bddf8a9a24"} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.receipt.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.receipt.json new file mode 100644 index 000000000..d8922b88d --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.receipt.json @@ -0,0 +1,25 @@ +{ + "status": "PASS", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "1" + ], + "returncode": 0, + "stop_reason": null, + "cpu_seconds": 1121.705467, + "wall_seconds": 1121.303854590049, + "peak_tree_rss_bytes": 2574909440, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 32768, + "max_poll_gap_seconds": 0.7979419510229491, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.started.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.started.json new file mode 100644 index 000000000..2409da7fc --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.started.json @@ -0,0 +1,17 @@ +{ + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "1" + ], + "limits": { + "total_cpu_seconds": 7200, + "job_cpu_seconds": 1800, + "job_wall_seconds": 1797.4561341169756, + "tree_rss_bytes": 17179869184, + "disk_bytes": 5368709120, + "max_processes": 2 + }, + "started_unix": 1794696831.0915108 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.log b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.log new file mode 100644 index 000000000..7c95152e9 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.log @@ -0,0 +1 @@ +{"status": "COMPLETE", "run": {"name": "N128-seed0", "arm": "N", "width": 128, "seed": 0}, "E_max": 42, "cosine_T_max": 42, "epochs": 42, "attempted_updates": 16044, "updates": 16038, "best_epoch": 39, "best_validation_score": 0.10332562533675191, "learning_parts": 382, "validation_parts": 111, "learning_components": 137, "validation_components": 35, "failed_preparation_parts_dropped": [{"case": "00000575_5a4cef68211d4706b1ec8586_step_004", "role": "validation", "nodes": 75046, "component_id": "component-3a3e340f4dfac391", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-inputs-2026-11-15", "181"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.766116, "wall_seconds": 1800.400393687014, "peak_tree_rss_bytes": 602218496, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.128913332009688, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false}, "failure": null}, {"case": "00000614_c710d414bbbc4340b682c58a_step_000", "role": "learning", "nodes": 145860, "component_id": "component-4bd089778094e746", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-inputs-2026-11-15", "201"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.796751, "wall_seconds": 1800.0567526929663, "peak_tree_rss_bytes": 629305344, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.1576595069491304, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false}, "failure": null}, {"case": "00000625_6af98f5458b347bb843f0562_step_000", "role": "learning", "nodes": 145707, "component_id": "component-4bd089778094e746", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-inputs-2026-11-15", "211"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.8864629999998, "wall_seconds": 1800.119237862993, "peak_tree_rss_bytes": 647880704, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.159920622012578, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false}, "failure": null}, {"case": "00001466_a45f6c99e2154f3b94955175_step_050", "role": "learning", "nodes": 77138, "component_id": "component-7fe5173e77c86f9d", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-recovery-inputs-2026-11-15", "83"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.926194, "wall_seconds": 1800.0714844280155, "peak_tree_rss_bytes": 383070208, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.0109735240112059, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false, "monitor_errors": 0, "last_monitor_error": null}, "failure": null}, {"case": "00001470_a45f6c99e2154f3b94955175_step_054", "role": "learning", "nodes": 124817, "component_id": "component-7fe5173e77c86f9d", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-recovery-inputs-2026-11-15", "87"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.664371, "wall_seconds": 1800.0443400070071, "peak_tree_rss_bytes": 578732032, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 1.007104828953743, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false, "monitor_errors": 0, "last_monitor_error": null}, "failure": null}, {"case": "00001473_a45f6c99e2154f3b94955175_step_057", "role": "learning", "nodes": 69428, "component_id": "component-7fe5173e77c86f9d", "guard": {"status": "STOPPED_OR_FAILED", "command": ["/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", "tools/s2_pilot_prepare.py", "artifacts/s2-pilot-recovery-inputs-2026-11-15", "90"], "returncode": -9, "stop_reason": "wall_limit", "cpu_seconds": 1799.751432, "wall_seconds": 1800.2498029720155, "peak_tree_rss_bytes": 438853632, "peak_process_count": 1, "output_disk_bytes_before_receipt": 4096, "max_poll_gap_seconds": 0.8929723129840568, "poll_target_seconds": 0.25, "reaped_orphan_cpu_seconds": 0.0, "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", "gpu_visible": false, "retry": false, "monitor_errors": 0, "last_monitor_error": null}, "failure": null}], "standardizer_fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03", "wall_seconds": 1125.9121631510206, "peak_allocated_bytes": 4706622464, "peak_reserved_bytes": 8246001664, "test_labels_read": false, "test_predictions": false, "capacity_selected": false, "checkpoint_sha256": "ca2eb128116e877e2ffed65f4c1c5864e090beab9b7c0615f117e613599ccd34"} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.receipt.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.receipt.json new file mode 100644 index 000000000..9bb6a4c87 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.receipt.json @@ -0,0 +1,25 @@ +{ + "status": "PASS", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "0" + ], + "returncode": 0, + "stop_reason": null, + "cpu_seconds": 1129.584761, + "wall_seconds": 1129.2963322199648, + "peak_tree_rss_bytes": 2574286848, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 12288, + "max_poll_gap_seconds": 0.8972175170201808, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.log b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.log new file mode 100644 index 000000000..606db387c --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.log @@ -0,0 +1,8 @@ +Traceback (most recent call last): + File "/home/frank14f/optane/MeltMind/tools/run_s2_pilot_training_v2.py", line 150, in + p=argparse.ArgumentParser();p.add_argument('root',type=Path);p.add_argument('index',type=int);a=p.parse_args();raise SystemExit(run(a.root,a.index)) + File "/home/frank14f/optane/MeltMind/tools/run_s2_pilot_training_v2.py", line 113, in run + skip_policy.finish_epoch(epoch) + File "/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py", line 52, in finish_epoch + if epoch>0 and self.skipped>self.attempted*.01:raise FloatingPointError('AMP skipped steps exceed 1 percent after epoch 0') +FloatingPointError: AMP skipped steps exceed 1 percent after epoch 0 diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.receipt.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.receipt.json new file mode 100644 index 000000000..ded2a5677 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.receipt.json @@ -0,0 +1,25 @@ +{ + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "2" + ], + "returncode": 1, + "stop_reason": null, + "cpu_seconds": 97.12944, + "wall_seconds": 96.86777837795671, + "peak_tree_rss_bytes": 2553110528, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 45056, + "max_poll_gap_seconds": 0.4935557850403711, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.started.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.started.json new file mode 100644 index 000000000..0dd17d16c --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.started.json @@ -0,0 +1,17 @@ +{ + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "2" + ], + "limits": { + "total_cpu_seconds": 7200, + "job_cpu_seconds": 1800, + "job_wall_seconds": 1797.4566384089994, + "tree_rss_bytes": 17179869184, + "disk_bytes": 5368709120, + "max_processes": 2 + }, + "started_unix": 1794697982.8880196 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/ledger.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/ledger.json new file mode 100644 index 000000000..2b50172f1 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/guards/ledger.json @@ -0,0 +1,20 @@ +{ + "cpu_seconds_charged": 2348.419668, + "attempts": [ + { + "name": "N128-seed0", + "status": "PASS", + "cpu_seconds": 1129.584761 + }, + { + "name": "E128-seed0", + "status": "PASS", + "cpu_seconds": 1121.705467 + }, + { + "name": "N256-seed0", + "status": "STOPPED_OR_FAILED", + "cpu_seconds": 97.12944 + } + ] +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/E128-seed0.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/E128-seed0.json new file mode 100644 index 000000000..5e57ca2b1 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/E128-seed0.json @@ -0,0 +1,15 @@ +{ + "status": "IDLE_CONFIRMED", + "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "polls": 4, + "consecutive_idle": 3, + "wall_seconds": 30.379757194954436, + "last": { + "unix": 1794696831.0448213, + "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "memory_MiB": 5, + "utilization_percent": 0, + "occupants": [], + "idle": true + } +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/E128-seed0.jsonl b/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/E128-seed0.jsonl new file mode 100644 index 000000000..7ec08ce57 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/E128-seed0.jsonl @@ -0,0 +1,4 @@ +{"unix": 1794696800.751581, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 61, "occupants": [], "idle": false, "poll": 1, "consecutive_idle": 0, "elapsed_seconds": 0.0863514129887335} +{"unix": 1794696810.8494375, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 0, "occupants": [], "idle": true, "poll": 2, "consecutive_idle": 1, "elapsed_seconds": 10.184205520956311} +{"unix": 1794696820.9473677, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 0, "occupants": [], "idle": true, "poll": 3, "consecutive_idle": 2, "elapsed_seconds": 20.28213056595996} +{"unix": 1794696831.0448213, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 0, "occupants": [], "idle": true, "poll": 4, "consecutive_idle": 3, "elapsed_seconds": 30.379584342008457} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/N256-seed0.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/N256-seed0.json new file mode 100644 index 000000000..6cc6104a9 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/N256-seed0.json @@ -0,0 +1,15 @@ +{ + "status": "IDLE_CONFIRMED", + "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "polls": 4, + "consecutive_idle": 3, + "wall_seconds": 30.443976683949586, + "last": { + "unix": 1794697982.8422568, + "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "memory_MiB": 5, + "utilization_percent": 0, + "occupants": [], + "idle": true + } +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/N256-seed0.jsonl b/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/N256-seed0.jsonl new file mode 100644 index 000000000..5a4eec8e2 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/N256-seed0.jsonl @@ -0,0 +1,4 @@ +{"unix": 1794697952.4835281, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 44, "occupants": [], "idle": false, "poll": 1, "consecutive_idle": 0, "elapsed_seconds": 0.08512524096295238} +{"unix": 1794697962.6504905, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 0, "occupants": [], "idle": true, "poll": 2, "consecutive_idle": 1, "elapsed_seconds": 10.252083649975248} +{"unix": 1794697972.746587, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 0, "occupants": [], "idle": true, "poll": 3, "consecutive_idle": 2, "elapsed_seconds": 20.34818573697703} +{"unix": 1794697982.8422568, "uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", "memory_MiB": 5, "utilization_percent": 0, "occupants": [], "idle": true, "poll": 4, "consecutive_idle": 3, "elapsed_seconds": 30.443847530987114} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/pilot-complete.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/pilot-complete.json new file mode 100644 index 000000000..b5cb58049 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/pilot-complete.json @@ -0,0 +1,133 @@ +{ + "status": "PILOT_INCOMPLETE", + "reason": "PILOT_RUN_FAILED_NO_RETRY", + "runs": [ + { + "run": { + "name": "N128-seed0", + "arm": "N", + "width": 128, + "seed": 0 + }, + "status": "PASS", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "0" + ], + "returncode": 0, + "stop_reason": null, + "cpu_seconds": 1129.584761, + "wall_seconds": 1129.2963322199648, + "peak_tree_rss_bytes": 2574286848, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 12288, + "max_poll_gap_seconds": 0.8972175170201808, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null, + "device_monitor": { + "peak_device_memory_bytes": 8662286336, + "peak_whole_run_tree_rss_bytes": 2980745216, + "max_poll_gap_seconds": 1.473451861005742, + "reason": null, + "samples": 1896 + }, + "total_run_wall_seconds": 1129.3405424560187, + "result_sha256": "f75ded80d92c4e9835ca8bcda6b9772fd1fe56b131c9964097fdfea7ea8f1a44" + }, + { + "run": { + "name": "E128-seed0", + "arm": "E", + "width": 128, + "seed": 0 + }, + "status": "PASS", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "1" + ], + "returncode": 0, + "stop_reason": null, + "cpu_seconds": 1121.705467, + "wall_seconds": 1121.303854590049, + "peak_tree_rss_bytes": 2574909440, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 32768, + "max_poll_gap_seconds": 0.7979419510229491, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null, + "device_monitor": { + "peak_device_memory_bytes": 8660189184, + "peak_whole_run_tree_rss_bytes": 2981724160, + "max_poll_gap_seconds": 1.105505651968997, + "reason": null, + "samples": 1887 + }, + "total_run_wall_seconds": 1121.3519870990422, + "result_sha256": "2e1271dbcc00ba4ee4889aa2c9e192283c06a9bd3eabc10a973dabba2f7dad78" + }, + { + "run": { + "name": "N256-seed0", + "arm": "N", + "width": 256, + "seed": 0 + }, + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "2" + ], + "returncode": 1, + "stop_reason": null, + "cpu_seconds": 97.12944, + "wall_seconds": 96.86777837795671, + "peak_tree_rss_bytes": 2553110528, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 45056, + "max_poll_gap_seconds": 0.4935557850403711, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null, + "device_monitor": { + "peak_device_memory_bytes": 9786359808, + "peak_whole_run_tree_rss_bytes": 2960007168, + "max_poll_gap_seconds": 0.9654930130345747, + "reason": null, + "samples": 165 + }, + "total_run_wall_seconds": 96.91506972297793, + "result_sha256": null + } + ], + "selected_global_width": null, + "selection_rule": "mean of N/E validation error at each width, ties smaller; all four runs required", + "pilot_wall_seconds": 2439.9096992989653, + "probe_and_setup_wall_seconds": 31.47485014400445, + "test_labels_read": false, + "test_predictions": false, + "retries": 0 +} diff --git a/artifacts/s2-pilot-v2-continuation-2026-11-15/shared-standardizer-fingerprint.json b/artifacts/s2-pilot-v2-continuation-2026-11-15/shared-standardizer-fingerprint.json new file mode 100644 index 000000000..d1727f0f6 --- /dev/null +++ b/artifacts/s2-pilot-v2-continuation-2026-11-15/shared-standardizer-fingerprint.json @@ -0,0 +1,387 @@ +{ + "fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03", + "learning_parts": [ + "00000003_1ffb81a71e5b402e966b9341_step_002", + "00000004_1ffb81a71e5b402e966b9341_step_003", + "00000021_ad34a3f60c4a4caa99646600_step_001", + "00000023_ad34a3f60c4a4caa99646600_step_003", + "00000024_ad34a3f60c4a4caa99646600_step_004", + "00000030_ad34a3f60c4a4caa99646600_step_010", + "00000031_ad34a3f60c4a4caa99646600_step_011", + "00000033_ad34a3f60c4a4caa99646600_step_013", + "00000049_aa208c45e8734584a147f595_step_000", + "00000057_666139e3bff64d4e8a6ce183_step_006", + "00000058_666139e3bff64d4e8a6ce183_step_007", + "00000060_767e4372b5f94a88a7a17d90_step_001", + "00000061_767e4372b5f94a88a7a17d90_step_002", + "00000065_767e4372b5f94a88a7a17d90_step_006", + "00000066_767e4372b5f94a88a7a17d90_step_007", + "00000067_767e4372b5f94a88a7a17d90_step_008", + "00000068_767e4372b5f94a88a7a17d90_step_009", + "00000070_767e4372b5f94a88a7a17d90_step_011", + "00000072_767e4372b5f94a88a7a17d90_step_013", + "00000077_767e4372b5f94a88a7a17d90_step_018", + "00000078_767e4372b5f94a88a7a17d90_step_019", + "00000079_767e4372b5f94a88a7a17d90_step_020", + "00000087_b4f9335a60a14cecaa599159_step_002", + "00000088_169bec34317b46e98eb1a403_step_000", + "00000093_169bec34317b46e98eb1a403_step_005", + "00000098_72cf0f9ca5324822a4d41a80_step_000", + "00000103_52f7124261ae4c558f800775_step_000", + "00000104_52f7124261ae4c558f800775_step_001", + "00000105_52f7124261ae4c558f800775_step_002", + "00000106_52f7124261ae4c558f800775_step_003", + "00000125_e18c7feacfb54f3880659b43_step_000", + "00000128_e18c7feacfb54f3880659b43_step_003", + "00000129_e18c7feacfb54f3880659b43_step_004", + "00000130_e18c7feacfb54f3880659b43_step_005", + "00000140_2fc54fcd110d4f49969163c4_step_003", + "00000142_2fc54fcd110d4f49969163c4_step_005", + "00000144_2fc54fcd110d4f49969163c4_step_007", + "00000145_2fc54fcd110d4f49969163c4_step_008", + "00000152_a951550b8a1043a68423c541_step_002", + "00000153_a951550b8a1043a68423c541_step_003", + "00000154_a951550b8a1043a68423c541_step_004", + "00000157_8208fd8b44b64dce91322f84_step_000", + "00000161_a583c6c3cdc143bab3f6c856_step_000", + "00000162_4dcda9fab6dd46cf98fb194e_step_000", + "00000163_ff9b2ed255484db78bb0e6da_step_000", + "00000164_91adbf180e114b698160c9a4_step_000", + "00000165_4070a99d6ebe45f5bd5d71eb_step_000", + "00000166_2246cab58dbe4a2fb00fed50_step_000", + "00000170_b21aa794b6b64d87a57c245e_step_000", + "00000171_b21aa794b6b64d87a57c245e_step_001", + "00000173_b21aa794b6b64d87a57c245e_step_003", + "00000174_b21aa794b6b64d87a57c245e_step_004", + "00000175_b21aa794b6b64d87a57c245e_step_005", + "00000176_b21aa794b6b64d87a57c245e_step_006", + "00000181_b21aa794b6b64d87a57c245e_step_011", + "00000198_826494e894454504b0971fd1_step_001", + "00000223_97b354a907114b7183faa9c4_step_001", + "00000224_97b354a907114b7183faa9c4_step_002", + "00000229_682d529d5e8c4387bc59b3a1_step_000", + "00000250_641f03cbe6a145508c6ae055_step_000", + "00000272_648a7c23a21f4e88bf25ffeb_step_000", + "00000273_648a7c23a21f4e88bf25ffeb_step_001", + "00000274_648a7c23a21f4e88bf25ffeb_step_002", + "00000275_648a7c23a21f4e88bf25ffeb_step_003", + "00000276_648a7c23a21f4e88bf25ffeb_step_004", + "00000280_5a6ef3285b89452ea6b390e7_step_001", + "00000282_5a6ef3285b89452ea6b390e7_step_003", + "00000285_d215b7bc60ae468288e5f1ea_step_000", + "00000287_58f8ff46b7564e989f9695b7_step_000", + "00000290_58f8ff46b7564e989f9695b7_step_003", + "00000291_58f8ff46b7564e989f9695b7_step_004", + "00000293_58f8ff46b7564e989f9695b7_step_006", + "00000295_52a153d2417747b5926ff9df_step_000", + "00000296_52a153d2417747b5926ff9df_step_001", + "00000301_52a153d2417747b5926ff9df_step_006", + "00000302_39b08d62a677414f86fdae52_step_000", + "00000305_3062bccff48e47a2b9de05e3_step_000", + "00000306_3062bccff48e47a2b9de05e3_step_001", + "00000307_3062bccff48e47a2b9de05e3_step_002", + "00000308_3062bccff48e47a2b9de05e3_step_003", + "00000309_3062bccff48e47a2b9de05e3_step_004", + "00000310_3062bccff48e47a2b9de05e3_step_005", + "00000311_3062bccff48e47a2b9de05e3_step_006", + "00000312_3062bccff48e47a2b9de05e3_step_007", + "00000314_3062bccff48e47a2b9de05e3_step_009", + "00000315_3062bccff48e47a2b9de05e3_step_010", + "00000316_3062bccff48e47a2b9de05e3_step_011", + "00000317_3062bccff48e47a2b9de05e3_step_012", + "00000318_3062bccff48e47a2b9de05e3_step_013", + "00000319_3062bccff48e47a2b9de05e3_step_014", + "00000321_3062bccff48e47a2b9de05e3_step_016", + "00000322_3062bccff48e47a2b9de05e3_step_017", + "00000323_3062bccff48e47a2b9de05e3_step_018", + "00000324_3062bccff48e47a2b9de05e3_step_019", + "00000325_3062bccff48e47a2b9de05e3_step_020", + "00000329_12a935cd9b684f879c0505fe_step_000", + "00000330_12a935cd9b684f879c0505fe_step_001", + "00000331_12a935cd9b684f879c0505fe_step_002", + "00000336_1a821c3883f7459a9615fca4_step_000", + "00000337_153efbd85ca64c4fb6b7ed5f_step_000", + "00000346_4185972a944744d8a7a0f2b4_step_002", + "00000348_4185972a944744d8a7a0f2b4_step_004", + "00000352_4185972a944744d8a7a0f2b4_step_008", + "00000355_68d1a5c2319a4a419ff625c6_step_001", + "00000356_68d1a5c2319a4a419ff625c6_step_002", + "00000357_68d1a5c2319a4a419ff625c6_step_003", + "00000364_38d899e2706a4d09960705dd_step_001", + "00000367_23c957e8e7b2428282a13ae2_step_000", + "00000368_23c957e8e7b2428282a13ae2_step_001", + "00000369_23c957e8e7b2428282a13ae2_step_002", + "00000374_23c957e8e7b2428282a13ae2_step_007", + "00000375_23c957e8e7b2428282a13ae2_step_008", + "00000376_23c957e8e7b2428282a13ae2_step_009", + "00000377_23c957e8e7b2428282a13ae2_step_010", + "00000378_23c957e8e7b2428282a13ae2_step_011", + "00000379_239bbf0e92974a32bd214366_step_000", + "00000382_239bbf0e92974a32bd214366_step_003", + "00000383_239bbf0e92974a32bd214366_step_004", + "00000386_239bbf0e92974a32bd214366_step_007", + "00000387_239bbf0e92974a32bd214366_step_008", + "00000388_239bbf0e92974a32bd214366_step_009", + "00000418_ce5b31679ee845a69ba89d6f_step_000", + "00000422_8e8a96d437584c77b3cfd84f_step_000", + "00000423_8e8a96d437584c77b3cfd84f_step_001", + "00000424_8e8a96d437584c77b3cfd84f_step_002", + "00000459_516b1784884d4facbc4c0d74_step_007", + "00000462_516b1784884d4facbc4c0d74_step_010", + "00000466_11951e51b89249dba38ded3e_step_001", + "00000469_11951e51b89249dba38ded3e_step_004", + "00000470_11951e51b89249dba38ded3e_step_005", + "00000474_a5639f2b84c445aea5da18ad_step_000", + "00000477_0fc8cc17c6af48dabe36c8f2_step_002", + "00000478_0fc8cc17c6af48dabe36c8f2_step_003", + "00000479_0fc8cc17c6af48dabe36c8f2_step_004", + "00000485_a7d0c851b5f24a97a7cac950_step_002", + "00000488_a7d0c851b5f24a97a7cac950_step_005", + "00000489_a7d0c851b5f24a97a7cac950_step_006", + "00000492_a7d0c851b5f24a97a7cac950_step_009", + "00000513_f78bf7cf156f4231988af062_step_001", + "00000519_5b8fa6b193d14279a77a0e50_step_000", + "00000523_1f8d7f343e634c11a80d23ac_step_000", + "00000526_f9d6559308dc4c85aab9fa38_step_000", + "00000530_bc0d6937206b43bc96521413_step_002", + "00000531_bc0d6937206b43bc96521413_step_003", + "00000543_8cf6554d440d40c4823dad6c_step_000", + "00000559_a1621e156ade4317ab6a5bc5_step_000", + "00000565_5b5119548f7b421e8f597438_step_000", + "00000587_2417b410de78416ebe50d460_step_000", + "00000589_2417b410de78416ebe50d460_step_002", + "00000590_2417b410de78416ebe50d460_step_003", + "00000591_2417b410de78416ebe50d460_step_004", + "00000609_04e799936ae446b9a3d8d2b8_step_000", + "00000610_04e799936ae446b9a3d8d2b8_step_001", + "00000611_04e799936ae446b9a3d8d2b8_step_002", + "00000613_189d1310aa8c4a2a92a21020_step_001", + "00000615_c710d414bbbc4340b682c58a_step_001", + "00000616_c710d414bbbc4340b682c58a_step_002", + "00000617_c710d414bbbc4340b682c58a_step_003", + "00000618_c710d414bbbc4340b682c58a_step_004", + "00000619_c710d414bbbc4340b682c58a_step_005", + "00000620_c710d414bbbc4340b682c58a_step_006", + "00000621_c710d414bbbc4340b682c58a_step_007", + "00000622_c710d414bbbc4340b682c58a_step_008", + "00000624_1eacb269d52a47f5b265c1ab_step_000", + "00000626_f686126d229f47aeb125b5ba_step_000", + "00000627_53bfa7e2e04c4d9dbf4a3db2_step_000", + "00000632_4f322b34e87c43289916c298_step_001", + "00000644_761a451dfea94bfa93e23cfe_step_000", + "00000658_a40da330321b44f286a27361_step_000", + "00000666_6f67a59c16b949a4b66dcfe1_step_000", + "00000683_8863c3497d6d4e05b0f41ddc_step_000", + "00000685_24615f5ff44d4cd5809e77a5_step_001", + "00000686_24615f5ff44d4cd5809e77a5_step_002", + "00000688_b9f3563b403143dbb86ab2b8_step_001", + "00000691_b9f3563b403143dbb86ab2b8_step_004", + "00000693_b9f3563b403143dbb86ab2b8_step_006", + "00000694_45f6100bfa3948e69405bb21_step_000", + "00000696_32309198fce9481896bd0a99_step_000", + "00000698_6f0fc559e8f3422c908b6060_step_001", + "00000699_cae88f0aaaf34de597863810_step_000", + "00000708_c08c1c78b86f4ee3bcb7d9de_step_004", + "00000709_c08c1c78b86f4ee3bcb7d9de_step_005", + "00000710_c08c1c78b86f4ee3bcb7d9de_step_006", + "00000721_9a461e8ae7dc4d33a55883eb_step_000", + "00000723_8ed79b6bcb024d1a9f2aa5e7_step_000", + "00000737_989e0f807cde438face0cb00_step_002", + "00000738_989e0f807cde438face0cb00_step_003", + "00000751_6f0d90054d40400593050930_step_000", + "00000752_6f0d90054d40400593050930_step_001", + "00000764_9cab8335065e4d439bbff471_step_000", + "00000765_9cab8335065e4d439bbff471_step_001", + "00000766_9cab8335065e4d439bbff471_step_002", + "00000767_9cab8335065e4d439bbff471_step_003", + "00000775_a25fb37392a44447b9684f16_step_000", + "00000776_a25fb37392a44447b9684f16_step_001", + "00000777_a25fb37392a44447b9684f16_step_002", + "00000778_a25fb37392a44447b9684f16_step_003", + "00000779_a25fb37392a44447b9684f16_step_004", + "00000780_a25fb37392a44447b9684f16_step_005", + "00000781_a25fb37392a44447b9684f16_step_006", + "00000782_a25fb37392a44447b9684f16_step_007", + "00000783_a25fb37392a44447b9684f16_step_008", + "00000784_a25fb37392a44447b9684f16_step_009", + "00000785_a25fb37392a44447b9684f16_step_010", + "00000786_a25fb37392a44447b9684f16_step_011", + "00000787_a25fb37392a44447b9684f16_step_012", + "00000788_a25fb37392a44447b9684f16_step_013", + "00000789_a25fb37392a44447b9684f16_step_014", + "00000790_a25fb37392a44447b9684f16_step_015", + "00000791_a25fb37392a44447b9684f16_step_016", + "00000792_a25fb37392a44447b9684f16_step_017", + "00000793_a25fb37392a44447b9684f16_step_018", + "00000797_e7533c87b9a347cdafd0c390_step_000", + "00000799_ad6107d9fc994ccca38cc4e4_step_001", + "00000800_ad6107d9fc994ccca38cc4e4_step_002", + "00000801_ad6107d9fc994ccca38cc4e4_step_003", + "00000802_ad6107d9fc994ccca38cc4e4_step_004", + "00000818_6bbad2756d7845bd8d731d60_step_000", + "00000819_6bbad2756d7845bd8d731d60_step_001", + "00000821_6bbad2756d7845bd8d731d60_step_003", + "00000822_c98978894f544f89aa10633f_step_000", + "00000823_c98978894f544f89aa10633f_step_001", + "00000827_b0d209f7a8ad4b02910c317f_step_003", + "00000829_889a065d51a24bb8ad704c69_step_000", + "00000830_889a065d51a24bb8ad704c69_step_001", + "00000832_889a065d51a24bb8ad704c69_step_003", + "00000834_889a065d51a24bb8ad704c69_step_005", + "00000837_b5409201f9a5495fa169af85_step_000", + "00000889_e2d3a4dc5f7c438bb240e9d1_step_000", + "00000890_e2d3a4dc5f7c438bb240e9d1_step_001", + "00000900_4df6ab9924674556bb4a509d_step_000", + "00000904_4df6ab9924674556bb4a509d_step_004", + "00000905_4df6ab9924674556bb4a509d_step_005", + "00000906_4df6ab9924674556bb4a509d_step_006", + "00000942_f297a24e1a4446cf9d74efd9_step_001", + "00000948_f297a24e1a4446cf9d74efd9_step_007", + "00000949_f297a24e1a4446cf9d74efd9_step_008", + "00000953_ccc14f0fcff54b5985cc9113_step_000", + "00000958_04906edf324a4bce8fc3f417_step_000", + "00000961_a98d481268ea4524a7769731_step_000", + "00000962_a98d481268ea4524a7769731_step_001", + "00000963_a98d481268ea4524a7769731_step_002", + "00000973_209426c5054945fab8a4af1b_step_000", + "00000976_b85ec2ff588148c4abd8c8d7_step_000", + "00001024_21adcacb9707488ea6655fd2_step_000", + "00001025_21adcacb9707488ea6655fd2_step_001", + "00001029_cde47df6e28543d1b434f206_step_001", + "00001030_cde47df6e28543d1b434f206_step_002", + "00001031_cde47df6e28543d1b434f206_step_003", + "00001033_18151464e6434759b7c2ece7_step_000", + "00001035_be3ed2c9696e4e6a91b34f79_step_000", + "00001036_be3ed2c9696e4e6a91b34f79_step_001", + "00001041_6baf25f5fe2947e8927797d8_step_000", + "00001048_2b2638f82b3946b1867f2d11_step_004", + "00001049_2b2638f82b3946b1867f2d11_step_005", + "00001050_2b2638f82b3946b1867f2d11_step_006", + "00001061_284effca5a19488fa8a228e2_step_002", + "00001062_c04956ddff6c4324b5cad67c_step_000", + "00001106_ba88e7579ced4a0aa2de629c_step_000", + "00001108_412675e2ea594e70bbd8cf6c_step_000", + "00001114_5f5cf6feacbc41fb88b2d873_step_000", + "00001121_da4037e40839451dbbc2bd1f_step_000", + "00001130_c2db313c2ce749d784c610d2_step_002", + "00001139_0b0ed9b0ccbb455d82c60d69_step_000", + "00001140_0b0ed9b0ccbb455d82c60d69_step_001", + "00001141_0b0ed9b0ccbb455d82c60d69_step_002", + "00001146_274866da478e4a009bed8750_step_001", + "00001159_f112d6f3e0c644ae9de2a724_step_000", + "00001165_4b65bb9a02f6494aa897a53e_step_000", + "00001169_0889c458f3f74dc288d2d531_step_000", + "00001170_0889c458f3f74dc288d2d531_step_001", + "00001182_bc0b5846e3d84904ad02e547_step_001", + "00001185_507b60b57791425bbb73273e_step_000", + "00001186_507b60b57791425bbb73273e_step_001", + "00001187_507b60b57791425bbb73273e_step_002", + "00001190_507b60b57791425bbb73273e_step_005", + "00001191_507b60b57791425bbb73273e_step_006", + "00001193_46b17dd88ed24f55bca2eb69_step_000", + "00001200_e44a0f9afbe84b918f789f0f_step_000", + "00001206_e44a0f9afbe84b918f789f0f_step_006", + "00001210_4be610c137ba42ae9933e69b_step_000", + "00001211_4be610c137ba42ae9933e69b_step_001", + "00001212_4be610c137ba42ae9933e69b_step_002", + "00001213_4be610c137ba42ae9933e69b_step_003", + "00001214_9edb90c9f417497fb70bddf6_step_000", + "00001228_2fd836ae38b44c41bfbd9584_step_002", + "00001232_73b40f9a4cd3419ab3b42b73_step_001", + "00001233_73b40f9a4cd3419ab3b42b73_step_002", + "00001235_17ea801126da4654b34f6c52_step_000", + "00001254_c2da273d137e475abb1ab1f4_step_000", + "00001255_3df9015915ff4fa7b57f0a71_step_000", + "00001257_a4715d363f88400c8f0238d1_step_000", + "00001260_a4715d363f88400c8f0238d1_step_003", + "00001266_b53ece83d8964b44bbf1f8ed_step_002", + "00001273_b53ece83d8964b44bbf1f8ed_step_009", + "00001274_b53ece83d8964b44bbf1f8ed_step_010", + "00001276_b53ece83d8964b44bbf1f8ed_step_012", + "00001285_b53ece83d8964b44bbf1f8ed_step_021", + "00001294_b53ece83d8964b44bbf1f8ed_step_030", + "00001298_b53ece83d8964b44bbf1f8ed_step_034", + "00001300_b9951da825954791b59b979d_step_000", + "00001301_a54ee779467b42ea8709e1ca_step_000", + "00001311_e95b8713cc0b4436909c78e9_step_000", + "00001319_6a5a7dbad03b4a57b33f2c0d_step_002", + "00001320_6a5a7dbad03b4a57b33f2c0d_step_003", + "00001321_7cef9b79a6b04db894fd74e5_step_000", + "00001324_5bf4a74af99f405392406e40_step_000", + "00001344_d8d97699a52842539e08d117_step_002", + "00001345_d8d97699a52842539e08d117_step_003", + "00001346_d8d97699a52842539e08d117_step_004", + "00001348_318b81a8e412438183201be7_step_000", + "00001349_27327ddba7f64417ae1e69a3_step_000", + "00001356_aa0de7c5e9e84d73816bdbde_step_000", + "00001380_c49484307f434da78fa06b1f_step_000", + "00001381_c49484307f434da78fa06b1f_step_001", + "00001389_21b525da345e4e62aebd906b_step_000", + "00001391_406eb61c554a429ebd8a4d4f_step_000", + "00001392_406eb61c554a429ebd8a4d4f_step_001", + "00001393_406eb61c554a429ebd8a4d4f_step_002", + "00001394_406eb61c554a429ebd8a4d4f_step_003", + "00001414_b38f661707674b8eb983d992_step_000", + "00001417_a45f6c99e2154f3b94955175_step_001", + "00001422_a45f6c99e2154f3b94955175_step_006", + "00001427_a45f6c99e2154f3b94955175_step_011", + "00001428_a45f6c99e2154f3b94955175_step_012", + "00001429_a45f6c99e2154f3b94955175_step_013", + "00001430_a45f6c99e2154f3b94955175_step_014", + "00001432_a45f6c99e2154f3b94955175_step_016", + "00001433_a45f6c99e2154f3b94955175_step_017", + "00001434_a45f6c99e2154f3b94955175_step_018", + "00001438_a45f6c99e2154f3b94955175_step_022", + "00001445_a45f6c99e2154f3b94955175_step_029", + "00001447_a45f6c99e2154f3b94955175_step_031", + "00001449_a45f6c99e2154f3b94955175_step_033", + "00001450_a45f6c99e2154f3b94955175_step_034", + "00001453_a45f6c99e2154f3b94955175_step_037", + "00001455_a45f6c99e2154f3b94955175_step_039", + "00001456_a45f6c99e2154f3b94955175_step_040", + "00001459_a45f6c99e2154f3b94955175_step_043", + "00001461_a45f6c99e2154f3b94955175_step_045", + "00001467_a45f6c99e2154f3b94955175_step_051", + "00001468_a45f6c99e2154f3b94955175_step_052", + "00001469_a45f6c99e2154f3b94955175_step_053", + "00001471_a45f6c99e2154f3b94955175_step_055", + "00001472_a45f6c99e2154f3b94955175_step_056", + "00001474_a45f6c99e2154f3b94955175_step_058", + "00001480_a45f6c99e2154f3b94955175_step_064", + "00001481_a45f6c99e2154f3b94955175_step_065", + "00001482_a45f6c99e2154f3b94955175_step_066", + "00001483_a45f6c99e2154f3b94955175_step_067", + "00001484_a45f6c99e2154f3b94955175_step_068", + "00001487_a45f6c99e2154f3b94955175_step_071", + "00001490_6c8ef6d2f5564d139a5077f8_step_001", + "00001491_6c8ef6d2f5564d139a5077f8_step_002", + "00001495_6c8ef6d2f5564d139a5077f8_step_006", + "00001496_eb4e2b01aaca4131bfc64adb_step_000", + "00001497_eb4e2b01aaca4131bfc64adb_step_001", + "00001500_eb4e2b01aaca4131bfc64adb_step_004", + "00001501_6a0f221e85e04411afe23f51_step_000", + "00001503_6a0f221e85e04411afe23f51_step_002", + "00001505_6d02705716e04d9081f4980b_step_000", + "00001527_9214869844f14e83a9dce3ec_step_000", + "00001534_e1d62958d7ca4ad18fefd54d_step_001", + "00001535_e1d62958d7ca4ad18fefd54d_step_002", + "00001538_e1d62958d7ca4ad18fefd54d_step_005", + "00001543_e1d62958d7ca4ad18fefd54d_step_010", + "00001550_a94d978cf62948ce80b417a9_step_000", + "00001554_30de00089a3f4a339eddb68d_step_000", + "00001555_30de00089a3f4a339eddb68d_step_001", + "00001556_30de00089a3f4a339eddb68d_step_002", + "00001557_16c191aef5034779aaab6e3f_step_000", + "00001563_2e9dc7e9a1a54dd7ab901478_step_000", + "00001567_9280741c840d4aa1af32658e_step_002", + "00001569_6fc579f3224d4d53a1b60710_step_000", + "00001578_a3b89691665043a08931897f_step_001", + "00001579_a3b89691665043a08931897f_step_002", + "00001582_a3b89691665043a08931897f_step_005", + "00001586_763da6a559544098acbf4154_step_002", + "00001592_35f4abaacc9447e4ac0b964f_step_001", + "00001594_2128dbbd4bf34879913b3a99_step_000", + "00001596_2128dbbd4bf34879913b3a99_step_002", + "00001597_13bd23e0e4c740969b0c71ab_step_000" + ] +} diff --git a/artifacts/s2-v2-stage-stop-2026-11-15/stage-audit.json b/artifacts/s2-v2-stage-stop-2026-11-15/stage-audit.json new file mode 100644 index 000000000..fbf23917d --- /dev/null +++ b/artifacts/s2-v2-stage-stop-2026-11-15/stage-audit.json @@ -0,0 +1,410 @@ +{ + "status": "PILOT_INCOMPLETE_NO_WIDTH_SELECTION", + "observed_unix": 1794698189.8437521, + "session": 34632, + "process_check": { + "time_utc8": "2026-11-15 07:14:59", + "matching_processes": [] + }, + "code_pins_match": true, + "source_bindings_match": true, + "sealed_tests": 125, + "test_manifest_overlap": 0, + "test_labels_read": false, + "test_predictions": false, + "cumulative_v2_wall_seconds": 2439.912972155027, + "limit_seconds": 7200, + "prior_v2_wall_seconds": 11.143491394002922, + "probe_reused": true, + "retries": 0, + "rows": [ + { + "run": "N128-seed0", + "started": true, + "guard": { + "status": "PASS", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "0" + ], + "returncode": 0, + "stop_reason": null, + "cpu_seconds": 1129.584761, + "wall_seconds": 1129.2963322199648, + "peak_tree_rss_bytes": 2574286848, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 12288, + "max_poll_gap_seconds": 0.8972175170201808, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null, + "device_monitor": { + "peak_device_memory_bytes": 8662286336, + "peak_whole_run_tree_rss_bytes": 2980745216, + "max_poll_gap_seconds": 1.473451861005742, + "reason": null, + "samples": 1896 + }, + "total_run_wall_seconds": 1129.3405424560187, + "result_sha256": "f75ded80d92c4e9835ca8bcda6b9772fd1fe56b131c9964097fdfea7ea8f1a44" + }, + "result": { + "status": "COMPLETE", + "run": { + "name": "N128-seed0", + "arm": "N", + "width": 128, + "seed": 0 + }, + "E_max": 42, + "cosine_T_max": 42, + "epochs": 42, + "attempted_updates": 16044, + "updates": 16038, + "best_epoch": 39, + "best_validation_score": 0.10332562533675191, + "learning_parts": 382, + "validation_parts": 111, + "learning_components": 137, + "validation_components": 35, + "standardizer_fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03", + "wall_seconds": 1125.9121631510206, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664, + "test_labels_read": false, + "test_predictions": false, + "capacity_selected": false, + "checkpoint_sha256": "ca2eb128116e877e2ffed65f4c1c5864e090beab9b7c0615f117e613599ccd34" + }, + "last_progress": { + "stage": "epoch_complete", + "epoch": 41, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 16044, + "training_global_weighted_mse": 0.004746648263794883, + "validation_component_equal_relative_l2": 0.10341266280676208, + "updates": 16038, + "elapsed_seconds": 1125.9099378320388, + "peak_allocated_bytes": 4706622464, + "peak_reserved_bytes": 8246001664, + "patience_count": 2 + } + }, + { + "run": "E128-seed0", + "started": true, + "guard": { + "status": "PASS", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "1" + ], + "returncode": 0, + "stop_reason": null, + "cpu_seconds": 1121.705467, + "wall_seconds": 1121.303854590049, + "peak_tree_rss_bytes": 2574909440, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 32768, + "max_poll_gap_seconds": 0.7979419510229491, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null, + "device_monitor": { + "peak_device_memory_bytes": 8660189184, + "peak_whole_run_tree_rss_bytes": 2981724160, + "max_poll_gap_seconds": 1.105505651968997, + "reason": null, + "samples": 1887 + }, + "total_run_wall_seconds": 1121.3519870990422, + "result_sha256": "2e1271dbcc00ba4ee4889aa2c9e192283c06a9bd3eabc10a973dabba2f7dad78" + }, + "result": { + "status": "COMPLETE", + "run": { + "name": "E128-seed0", + "arm": "E", + "width": 128, + "seed": 0 + }, + "E_max": 42, + "cosine_T_max": 42, + "epochs": 42, + "attempted_updates": 16044, + "updates": 16038, + "best_epoch": 39, + "best_validation_score": 0.10435484679758687, + "learning_parts": 382, + "validation_parts": 111, + "learning_components": 137, + "validation_components": 35, + "standardizer_fingerprint": "93ca89996de50e130d241a674591ad443fdbf334f965d52fdfe9574d10939b03", + "wall_seconds": 1117.921224004007, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512, + "test_labels_read": false, + "test_predictions": false, + "capacity_selected": false, + "checkpoint_sha256": "910437bd645a965241786a90ad14325fe7af09092238bfd0be5ee5bddf8a9a24" + }, + "last_progress": { + "stage": "epoch_complete", + "epoch": 41, + "amp": { + "attempted_steps": 382, + "skipped_steps": 0, + "successful_updates": 382, + "skipped_fraction": 0.0, + "consecutive_skipped": 0 + }, + "attempted_updates": 16044, + "training_global_weighted_mse": 0.004690328458595227, + "validation_component_equal_relative_l2": 0.10452529978657271, + "updates": 16038, + "elapsed_seconds": 1117.918959765986, + "peak_allocated_bytes": 4706630656, + "peak_reserved_bytes": 8243904512, + "patience_count": 2 + } + }, + { + "run": "N256-seed0", + "started": true, + "guard": { + "status": "STOPPED_OR_FAILED", + "command": [ + "/home/frank14f/anaconda3/envs/pycuda_3_10/bin/python", + "tools/run_s2_pilot_training_v2.py", + "artifacts/s2-pilot-v2-continuation-2026-11-15", + "2" + ], + "returncode": 1, + "stop_reason": null, + "cpu_seconds": 97.12944, + "wall_seconds": 96.86777837795671, + "peak_tree_rss_bytes": 2553110528, + "peak_process_count": 1, + "output_disk_bytes_before_receipt": 45056, + "max_poll_gap_seconds": 0.4935557850403711, + "poll_target_seconds": 0.1, + "reaped_orphan_cpu_seconds": 0.0, + "supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure", + "gpu_visible": true, + "gpu_uuid": "GPU-e8853ae7-a302-6a17-bcfa-aa767e9ed20c", + "retry": false, + "monitor_errors": 0, + "last_monitor_error": null, + "device_monitor": { + "peak_device_memory_bytes": 9786359808, + "peak_whole_run_tree_rss_bytes": 2960007168, + "max_poll_gap_seconds": 0.9654930130345747, + "reason": null, + "samples": 165 + }, + "total_run_wall_seconds": 96.91506972297793, + "result_sha256": null + }, + "failure": { + "status": "FAILED_NO_RETRY", + "error_type": "FloatingPointError", + "message": "AMP skipped steps exceed 1 percent after epoch 0", + "traceback": "Traceback (most recent call last):\n File \"/home/frank14f/optane/MeltMind/tools/run_s2_pilot_training_v2.py\", line 113, in run\n skip_policy.finish_epoch(epoch)\n File \"/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py\", line 52, in finish_epoch\n if epoch>0 and self.skipped>self.attempted*.01:raise FloatingPointError('AMP skipped steps exceed 1 percent after epoch 0')\nFloatingPointError: AMP skipped steps exceed 1 percent after epoch 0\n", + "wall_seconds": 93.35228163201828, + "epoch": 2, + "amp": { + "attempted_steps": 382, + "skipped_steps": 4, + "successful_updates": 378, + "skipped_fraction": 0.010471204188481676, + "consecutive_skipped": 0 + } + }, + "last_progress": { + "stage": "training", + "epoch": 2, + "parts_done": 369, + "parts_total": 382, + "updates": 1128, + "elapsed_seconds": 92.62161607301096, + "peak_allocated_bytes": 7525132800, + "peak_reserved_bytes": 16416505856 + } + }, + { + "run": "E256-seed0", + "started": false + } + ], + "additional_resource_boundary": { + "recorded_reserved_bytes": 16416505856, + "cap_bytes": 16106127360, + "over_bytes": 310378496, + "recorded_reserved_GiB": 15.2890625, + "seen_since": "N256 epoch0 history and latest progress", + "sampled_device_peak_bytes": 9786359808, + "sampled_guard_reason": null, + "guard_gap": "recorded peak checked only when result.json exists; failed run had no result.json", + "OOM_observed": false, + "all_resource_limits_satisfied": false + }, + "artifact_sha256": { + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/best-validation-parts.json": "201bf93e5459fa9df51722f2f7fab77a1671001120f6be0430ef4b439c9acb4a", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/best.pt": "910437bd645a965241786a90ad14325fe7af09092238bfd0be5ee5bddf8a9a24", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-000-amp.json": "1844d07337a39d62399d5f01afde805d890bc8d022fe3711255a1c3a61cd9884", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-001-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-002-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-003-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-004-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-005-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-006-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-007-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-008-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-009-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-010-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-011-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-012-amp.json": "c5e1bf965b9d9963d48cc0595e34cd6fa6456728c90efb11431a63432d68a4a8", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-013-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-014-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-015-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-016-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-017-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-018-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-019-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-020-amp.json": "c5e1bf965b9d9963d48cc0595e34cd6fa6456728c90efb11431a63432d68a4a8", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-021-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-022-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-023-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-024-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-025-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-026-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-027-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-028-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-029-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-030-amp.json": "c5e1bf965b9d9963d48cc0595e34cd6fa6456728c90efb11431a63432d68a4a8", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-031-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-032-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-033-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-034-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-035-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-036-amp.json": "c5e1bf965b9d9963d48cc0595e34cd6fa6456728c90efb11431a63432d68a4a8", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-037-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-038-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-039-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-040-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/epoch-041-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/history.json": "d79dfa8f65d630585380317bb3b686fb7e6a8308ece3cca0f7dbfccb6ce54f86", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/progress.json": "7b785df94bf3188bc0b61b06572ff73fca9a50b1b89a3e4f4a65c8731ffccc0e", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/result.json": "2e1271dbcc00ba4ee4889aa2c9e192283c06a9bd3eabc10a973dabba2f7dad78", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/standardizer.json": "d12d9cb1ba4fc4b9cb00b289ea71e630b05e776a9976c7998351be4244dcdc1c", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0/started.json": "aa7c82e575494e856eb118831bebc41240f38cb72dc62c9f3a9cab5fc8f83ef7", + "artifacts/s2-pilot-v2-continuation-2026-11-15/E128-seed0-guard-receipt.json": "39d3c063ab4ac979add7150e374ad1531f6621b49c3fe5653d443e999e57624b", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/best-validation-parts.json": "86cb4806c1657144dcabf5b246955ed1ce0b989199f9ff2cccd7c675f506d389", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/best.pt": "ca2eb128116e877e2ffed65f4c1c5864e090beab9b7c0615f117e613599ccd34", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-000-amp.json": "1844d07337a39d62399d5f01afde805d890bc8d022fe3711255a1c3a61cd9884", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-001-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-002-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-003-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-004-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-005-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-006-amp.json": "c5e1bf965b9d9963d48cc0595e34cd6fa6456728c90efb11431a63432d68a4a8", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-007-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-008-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-009-amp.json": "1844d07337a39d62399d5f01afde805d890bc8d022fe3711255a1c3a61cd9884", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-010-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-011-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-012-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-013-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-014-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-015-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-016-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-017-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-018-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-019-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-020-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-021-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-022-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-023-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-024-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-025-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-026-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-027-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-028-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-029-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-030-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-031-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-032-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-033-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-034-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-035-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-036-amp.json": "c5e1bf965b9d9963d48cc0595e34cd6fa6456728c90efb11431a63432d68a4a8", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-037-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-038-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-039-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-040-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/epoch-041-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/history.json": "ce1227a727eb29d9a2b31294b822a38e16647af5cc78371f3c3ff0832d78cb8d", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/progress.json": "5ee8fcb2f430bf2526147eb4569787d2c48680735122f6a08debd0710f709495", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/result.json": "f75ded80d92c4e9835ca8bcda6b9772fd1fe56b131c9964097fdfea7ea8f1a44", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/standardizer.json": "d12d9cb1ba4fc4b9cb00b289ea71e630b05e776a9976c7998351be4244dcdc1c", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0/started.json": "faafcb0a870d5de0524f6377dc11cf8be5bc9158a33bcc23e7ed16a241aea221", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N128-seed0-guard-receipt.json": "ab7b44f144d5563cb8ffcc91c62f7b4452c6db645adf1f6068985c5a11ce98eb", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/best-validation-parts.json": "1546ca4b06041ae9c281a43ff146f5191b6fabec4c5122d22fd88880320f6949", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/best.pt": "524cca32d95fe0be1530e55f40859a6a9706d158ed8bf9230f6d202c39b2ee8c", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-000-amp.json": "c5e1bf965b9d9963d48cc0595e34cd6fa6456728c90efb11431a63432d68a4a8", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-001-amp.json": "11a8aeed69fafbe2ff8bf450b89be193e5ddf68a64ab5a93a8c70d5bfbbe3e16", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/epoch-002-amp.json": "d30c6ebd52c995fe2fc94e6b25db53abef886ce4006d652c0e379c396dcc9874", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/failure.json": "37ff4bb9ed7c7d8833e0f332bfaf9eff0ecb92cdda60b96db863239da77716af", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/history.json": "03a086fa5dd18cbae94984cfd1991d98a3e4011105c237fa32ab98c9d29004a4", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/progress.json": "a7bb4a85494c928d0daeacd90ef61a8043d6f4b2c82c362d768517fe4e30ac67", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/standardizer.json": "d12d9cb1ba4fc4b9cb00b289ea71e630b05e776a9976c7998351be4244dcdc1c", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0/started.json": "243b532e85c8bc5e0e41db8a8dfd7c6847714f58d0277c3433f64785992807f5", + "artifacts/s2-pilot-v2-continuation-2026-11-15/N256-seed0-guard-receipt.json": "a40658fabc3c56cbef9198ed09ad5f549fd6e36884a563daed31bccb7e88b389", + "artifacts/s2-pilot-v2-continuation-2026-11-15/complete.json": "ed39ab1396234854cf6b46d66c3f67413ff356fce94a5555a6b327acad6fd380", + "artifacts/s2-pilot-v2-continuation-2026-11-15/continuation-contract.json": "225e51f42d545b868964fc7aa793805d6c35ba821915fcf1bffb47d516eb3574", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.log": "ae9729f5d22a471191328972fb95f30152b83e2d76028f1715011126a69458f3", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.receipt.json": "db164b3b188811ff95ee3a16a02d8d61f0449ac9eadfcbbbc9f53448fd761ee6", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/E128-seed0.started.json": "1fc597e4e852f6509b87c8f5c671673de8c9f1217e8ddcb8d9cb31d29d25a029", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.log": "c103c6122082977f8a9ba3566f429d6f13ee03987550dbbb7e0c6fc88b8fa8c7", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.receipt.json": "183491ae2e6afe23dc846007755dbc8935d7713281c83ab7a02a85e166e43be3", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N128-seed0.started.json": "1bc75baded2ffc1dc8aa53934e1be28a1e138782be37bd4ebaf2af644ecf0e5b", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.log": "a5c31d42302dffcd78a089663513a3e3fd70f298d5f56595b7f00e5afeb83e86", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.receipt.json": "9b8dcb1be0e33dfbc70569311d6e2f01665e170ef0c8d70783b7a2a82f873839", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/N256-seed0.started.json": "7ed3cb6b1f4e5a9c9205c0de816714e091115c648ecf7fd0b7936611043086b2", + "artifacts/s2-pilot-v2-continuation-2026-11-15/guards/ledger.json": "6c417f13062cd3327143ba7c6fcb38b9c17e3a7081a220f2fb0d129b88627bd0", + "artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/E128-seed0.json": "1a50a48fdb218d79f97c874ded06c681d97bae53edd71232663da3d88087fd93", + "artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/E128-seed0.jsonl": "de79269f8bc2691c0df8e9e481ba389fe1ed3ce35c99cf40c49b5a69330b7f47", + "artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/N256-seed0.json": "2b5715cd0286628a36d4f357c3a4a56f87ff77f78518dd7bb2e84eb03310d774", + "artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/N256-seed0.jsonl": "69ad9cf0f10566e06c579181aec4cd20355cadc5d32b33b0431cf417739b3de5", + "artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/initial-selected.json": "0e7a31405a607b1913b9acacf49f1e79d3dd8004d183ef0bc077c5bfebd8cb50", + "artifacts/s2-pilot-v2-continuation-2026-11-15/idle-gates/initial-selected.jsonl": "2aafe2f73dba34188ac857533f4def13b14cdff2ad7263012fcabc957a6b805d", + "artifacts/s2-pilot-v2-continuation-2026-11-15/pilot-complete.json": "0a621f234f7972064bb249ea405eab1ed73a33511652fbbe0810f1be1571787c", + "artifacts/s2-pilot-v2-continuation-2026-11-15/pilot-started.json": "8787de8b0f95ea0d4aadb3f44e4d899cee7cfa3eb11a505c6b3e033661dcf0fd", + "artifacts/s2-pilot-v2-continuation-2026-11-15/public/N128-actual-start-progress.json": "626b2efcbf5d38d3d5fffa24433a81698a55e7f93319f1b6c4fff02ae9b655cc", + "artifacts/s2-pilot-v2-continuation-2026-11-15/public/actual-start-audit.json": "b4c550397b16e7792a5e08837583f3a8b2922c37353aa33315c0ab0aa8d948e6", + "artifacts/s2-pilot-v2-continuation-2026-11-15/shared-standardizer-fingerprint.json": "a731e7c8f3eec8ff096a8d39e1248f2201bfb5a4f61cc6b1fb6aed6ddb789f99", + "artifacts/s2-pilot-v2-continuation-2026-11-15/started.json": "147318ce2fa0bc76488f14205eb6dca87747ff5234e6dda62fc46b28c1c4f77c", + "artifacts/s2-pilot-v2-continuation-2026-11-15/synthetic-receipt.json": "fd53db3dbc259ba03390bf1736ef3f7fce4743c80182b167fbba03ec6c52141c", + "artifacts/s2-pilot-v2-continuation-2026-11-15/training-contract.json": "f6eaa3cb2fd6918e5bd940826a2cc954ffcee0a9a2d1578b2da1e5e1c72abd5c" + } +} diff --git a/docs/s2-pilot-live-handoff-2026-11-15.md b/docs/s2-pilot-live-handoff-2026-11-15.md index 77f633b44..60d6cbff0 100644 --- a/docs/s2-pilot-live-handoff-2026-11-15.md +++ b/docs/s2-pilot-live-handoff-2026-11-15.md @@ -1,5 +1,9 @@ # S2 当前执行接续记录 +## 最新终态:34632已停止,无活动pilot(07:14:59核查) + +N128/E128均42epoch COMPLETE+guard PASS;N256在epoch2因AMP跳步4/382超过1%而失败,E256未启动,PILOT_INCOMPLETE/无选宽。正式累计2439.912972s/7200s,全部相关进程退出。N256同时记录reserved峰值16416505856B超过15GiB,失败路径guard未检查该记录峰值;详见 `s2-v2-pilot-stop-results-2026-11-15.md`。原始回执/源码不变、125测试封存。勿从下方历史运行描述或旧progress重启;等待Claude对已交失败证据的裁量。 + ## 当前唯一运行:v2工程接续34632,N128 worker已实证启动 Claude1794695421.177269裁量后的新入口固定ea5ac05。session34632、入口PID209380、N128 worker PID209415实际存活;三次10秒间隔空闲检查共20.285703秒通过,同UUID未切换。根artifacts/s2-pilot-v2-continuation-2026-11-15,actual-start-audit及两级worker.started/进度证明首run实际启动。新probe不重跑,E_max42;预算累计原11.143491秒加全部新等待/setup/run仍≤7200秒。源码pin不改、125测试封存。未完成run不作完成或选宽声明;禁止重复启动。 @@ -54,3 +58,5 @@ Claude 1794684155.730259已接受接续入口,并要求合法单件FAIL保持 首50已全部PASS并越过屏障,正式外推31.112177worker CPU-h /6.078539GiB,原60h队列自主继续,无需续队列。详见s2-first50-checkpoint-2026-11-15.md。受控接续session13519已实际启动,根artifacts/s2-pilot-continuation-2026-11-15;训练目标根artifacts/s2-pilot-training-2026-11-15。后续按当前progress/terminal回执查状态,不重复启动任一入口。准备结束后的清单、计时合同、训练状态与失败均会原位落盘;原Slack结果发布须据届时真实回执接续。 夜间核查补充(Claude 1794684781.751169):首50已正式接受,原60h队列继续,不启用80h续队列。pilot准备仍守单件1800s;失败件报告必须补case/role/nodes/停止原因及原回执hash。探针VRAM结论仅覆盖其实际最大成功学习件;不得把20件probe推到被剔除或更大未测件。当前固定池最大学习件为666,187,345节点,但尚无本次GPU测量。已追加artifacts/s2-pilot-continuation-2026-11-15/probe-coverage-boundary.json;probe实际回执产生后,另建与其hash关联的覆盖附件,列实际最大节点与失败件节点数,保留原回执/运行源码pin不改。主矩阵之前,超时大件须按另行标记的更长单件上限准备,并在真实最大件核VRAM;这不是pilot静默重试,本轮未启动该工作。 + +最新主审裁量1794698179.161599:主矩阵宽128按可行性选取,宽度效应未检验;原pilot自动选宽仍null。主矩阵前guard记录峰值修复、allocator设置、预算/执行计划待办;本轮未启动。详细依据与128整run显存上界见s2-v2-pilot-stop-results-2026-11-15.md。 diff --git a/docs/s2-v2-pilot-stop-results-2026-11-15.md b/docs/s2-v2-pilot-stop-results-2026-11-15.md new file mode 100644 index 000000000..c6d286ae0 --- /dev/null +++ b/docs/s2-v2-pilot-stop-results-2026-11-15.md @@ -0,0 +1,30 @@ +# S2 v2 pilot终态:两run成功,N256失败,未选宽 + +07:14:59(Asia/Shanghai)只读核查:session34632及全部训练子进程已结束。正式complete记录PILOT_INCOMPLETE、selected_global_width=null;累计2439.912972155秒/7200秒,包含先前v2 probe/零run入口11.143491394秒及新门控等待、设置和训练。E256从未启动,不根据残缺矩阵选宽。 + +| run | 终态 | 完整epoch | 最佳验证分数(仅该run) | guard总wall秒 | +| --- | --- | --- | --- | --- | +| N128 seed0 | COMPLETE / PASS | 42 | 0.10332562533675191 | 1129.340542456 | +| E128 seed0 | COMPLETE / PASS | 42 | 0.10435484679758687 | 1121.351987099 | +| N256 seed0 | FAILED_NO_RETRY | 2;epoch2训练后失败 | 不作为有效pilot结果 | 96.915069723 | +| E256 seed0 | 未启动 | 0 | 无 | 无 | + +两项成功结果及best.pt均核对result/guard/checkpoint SHA。分数是既定组件等权验证相对L2,单fold、seed0的部分pilot记录;不能据此声称N/E泛化优势或完成容量选择。 + +N256实际异常为FloatingPointError: AMP skipped steps exceed 1 percent after epoch 0。epoch0跳1/382(按原政策该轮比例豁免),epoch1跳0/382,epoch2跳4/382=1.0471204%,成功更新378;原1%阈值触发停止,没有改阈值/重试/自动精度退回。现有逐轮文件未记录每次跳步对应case或梯度tensor,因此跳步的具体数值根因尚未定位;不把本次异常等同于已修正的forward slice溢出。 + +另有必须保留的资源边界异常:N256的epoch0 history及最新progress记录peak_reserved_bytes=16,416,505,856(15.2890625GiB),超过既定15GiB=16,106,127,360字节上限310,378,496字节;peak_allocated=7,525,132,800。guard设备抽样峰值9,786,359,808、monitor reason=null且无OOM证据。原guard仅在result.json存在时检查记录峰值,本失败run无result.json,所以未触发RECORDED_VRAM_LIMIT。保留两类峰值及检测缺口,不能声称N256满足全部VRAM上限,不能用抽样峰值抹掉记录超限;本轮未修改guard或运行源码。 + +来源、训练契约与全部源码pin均通过;125件测试与493件成功manifest无交集,测试标签/预测仍封存。原数值失败、独立诊断、v2零run停止和本接续各自证据均保持原样。没有重probe、重准备、再次训练或启动主矩阵。 + +实际文件在artifacts/s2-pilot-v2-continuation-2026-11-15:三run逐轮AMP/history/progress/结果或失败/best.pt、逐run guard与累计ledger、门控每次PID/user/memory观察、pilot-complete及complete。artifacts/s2-v2-stage-stop-2026-11-15/stage-audit.json列全部交付文件SHA及独立资源边界说明。已在原Slack串1794698133.209129交Claude/TRON主审;下一步由其据失败与覆盖边界裁量,本文件不新增诊断/调参许可。 + +## Claude后续可行性裁量(与原pilot自动选择记录分别保存) + +Claude1794698179.161599已接收两项缺陷,明确N256 reserved峰值为VRAM上限违反;要求未来guard检查全部progress/result的最大记录峰值,并加仅progress越限的合成测试。原AMP1%阈值不变、失败不重跑。 + +Claude决定主矩阵宽度128,依据是256未通过冻结精度/VRAM可行性约束;他明确表示未看N128/E128验证分数,分数不得用于事后解释该选择。记为“capacity chosen by feasibility; the width effect was not tested”。原pilot-complete的selected_global_width=null仍不改,人工可行性裁量另立来源;这是一项限制,非宽度效应结论。 + +128证据边界:两个成功run均完整42轮、每轮382学习件,固定成功manifest的最大学习件为case666/187345节点。N128整run峰值allocated=4706622464B(4.383383GiB)、reserved=8246001664B(7.679688GiB);E128 allocated=4706630656B(4.383390GiB)、reserved=8243904512B(7.677734GiB)。这些是覆盖包含最大件在内全部学习/验证执行的run-wide上界,均低于15GiB;现有记录没有逐件隔离峰值,不能冒称直接测得最大件专属峰值。不为补该字段重跑。 + +主矩阵之前的未完成事项:上述guard修复与合成测试;设置PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True(需核实际环境支持与记录);保留AMP1%和E_max42;依据已测128时长提出预算/执行计划。该裁量不是本次只读核查已经完成这些工作或启动主矩阵的证明。