Preserve Track A monitor timeout and collateral stop with raw evidence and completed checkpoints

This commit is contained in:
MeltAstra committed 2026-11-18 19:15:18 +08:00
1 parent 8137f79be6
commit 32bd61e112
896 files changed
+1213518

No files matched your search

@@ -681,3 +681,8 @@ Claude1794992262要求的独立后处理绑定已固定0d8eb092/88fab6e5,19CPU
Seed1 postfit实际独立审阅采用:Assist1794993449的7b4da089/51c2514c原件已核SHA、五字段全等并逐字固定,review freeze 3086504ab049517ed0872a0499845b0a1dd7f092;validate_review/config实际PASS。implementation仍0d8eb092,e49851ba仅共享提交。仅完成审阅门;真实五训练终态/模型哈希/冻结/退出及累计耗时仍待运行时核验,未预测评分/读test目标或新建waiter。原两训练队列与pins保持,不重复主审/检查或自动扩科学范围。
## Track A基础设施STOP待裁量(2026-11-18 19:05)
原Track A因fold4外层nvidia-smi状态查询2秒TimeoutExpired停,fold3全局停止连带中断;0/1/2原200epoch PASS与模型保留,全部原进程退出。实际47943.947195GPU秒/最后投影PASS,不是预算耗尽或已证数值失败。证据a0a32fbf/audit d0433cb5完整冻结,1795000441已实际@Claude报分类、余额及CPU记录缺口。原no_retry/两个不完整折/全五冻结缺失均保留;没有修复或恢复实现、postfit、科学新计算。三PASS复用及两中断折下一处理仅待Claude工程裁量,不能从剩余预算推导自动授权。seed1原队列不动。
+6
View File
@@ -1,3 +1,9 @@
## 最新 Track A真实STOP:监控查询超时/连带停止,禁止按旧进度重启(2026-11-18 19:05)
Track A原1262364服务19:05:16 exit1,全部queue/guard/worker独立核验退出。fold4外层nvidia-smi固定2秒TimeoutExpired→MONITOR_ERROR→JOB_4_FAILED全局停;worker监控int0/return−9。fold3随全局停+SIGTERM连带终止,已完整189epoch、第190轮17件/57851steps,无独立数值failure。fold4第1轮75件/0完整epoch;两折无最后checkpoint/result。fold0/1/2各200轮真实PASS、模型/learning-only标准化哈希核;新fold2guard b60b2575。原no_retry不变,无自动恢复或postfit。
累计47943.94719455298GPU-s,最后投影87429.69551990522PASS,非耗尽;总162000/训练154800,post7200未用。guardCPU含worker,STOP未序列化coordinatorCPU,缺口保留。928原文件含3model共199235784B冻结a0a32fbf/audit d0433cb5,报告docs/track-a-monitor-stop-results-2026-11-18.md;未生成全五models/terminal成功冻结。已实际@Claude/TRON 1795000441请求下一裁量,未实现修复/续跑。seed1 v2原队列51pins PASS,审计约88/67/5继续,勿干扰。下方Track A“运行中”均为历史快照。
## 最新 seed1 postfit实际独立PASS已采用,运行门仍待满足(2026-11-18)
Assist1794993449新审阅review7b4da089/evidence51c2514c已直接读原件、SHA核验并逐字复制;完整binding e94e2457与0d8eb092/88fab6e5/41972211/8b7569e2/55code map全等。implementation-review freeze `3086504ab049517ed0872a0499845b0a1dd7f092`,原validate_review及config实际PASS;receipt在root/review-adoption.json。没有重跑19项合成检查,没有改源、预算或训练。核时训练complete仍未出现;本次不采用模型、不创建waiter/新队列、不预测/评分或读test目标。五训练真实PASS、模型/terminal冻结、退出及无globalstop等运行门仍须执行时实核。原prepared/report中的“待审”现由此实际采用更新,不把代码审阅PASS称作训练/科学结果PASS。
@@ -0,0 +1,43 @@
# Track A 监控超时与连带停止:终态审计(2026-11-18)
19:05:13.579676 CST,fold4外层监控的一次`nvidia-smi`状态查询超过固定2秒时限,原guard记录MONITOR_ERROR并杀停worker。协调器记录JOB_4_FAILED并停止同队列fold3。服务19:05:16退出1,MainPID=0、Restart=no,独立进程核验无Track A queue/guard/worker。此为监控基础设施停止及连带中断;没有依据将它归为预算耗尽或模型科学数值失败。底层查询超时原因尚未查明。
## 原始结果与模型
| fold | 状态 | 完整epoch | 最后进度 | 计费GPU秒 |
|---|---|---:|---|---:|
| 0 | PASS | 200 | 最后200轮checkpoint已核 | 11847.188839586 |
| 1 | PASS | 200 | 最后200轮checkpoint已核 | 11701.193322009 |
| 2 | PASS | 200 | 最后200轮checkpoint已核 | 12059.386084717 |
| 3 | STOPPED_OR_FAILED | 189 | epoch190,17件,57851steps | 12241.932636257 |
| 4 | STOPPED_OR_FAILED | 0 | epoch1,75件,75steps | 94.246311984 |
fold0/1/2原checkpoint、结果和learning-only19列标准化哈希一致,保留原200epoch/全学习池/last-only身份;本审计没有加载模型值或重新计算。fold3无最终checkpoint,完整189轮后在190轮第17/306件中断;fold4在首轮75/310件中断,无完整epoch。两中断折均无worker failure.json、完整result或checkpoint。保留全部逐件/逐轮历史,已逐条核对记录的loss/gradient有限;这不等于证明未记录时刻的数值状态。
全五折未PASS,原models-terminal、models-git-freeze和terminal-git-freeze均未生成。本次专门的infrastructure-stop冻结保存失败事实和已有三模型,绝不替代全批成功模型冻结。Track A postfit无started/models-adopted/predictions-manifest/complete。
## 错误层级与停止链
fold4 wrapper monitor_errors含一条TimeoutExpired:
```text
Command '['nvidia-smi', '--id=GPU-b8c11c01-c91f-aa0b-abf6-dad35085cc76', '--query-gpu=memory.used,utilization.gpu', '--format=csv,noheader,nounits']' timed out after 2 seconds
```
fold4 worker receipt的monitor_errors严格为整数0,returncode=-9;wrapper reasons为MONITOR_ERROR/WORKER_NOT_PASS/NO_COMPLETE_TRAIN_RESULT。外层状态查询失败和内层进程监控成功是两件事,不能统称“监控全零”。
fold3 wrapper errors=[]、worker monitor_errors=int0,returncode=-9;reasons=TRACK_A_GLOBAL_STOP/COORDINATOR_SIGNAL_15/WORKER_NOT_PASS/NO_COMPLETE_TRAIN_RESULT。原代码先记录fold4失败触发全局停止,guard按固定stop_group终止worker;与回执一致,不称fold3独立科学失败。各receipt记录RSS/VRAM未越界;约2.2秒实际poll gap与0.2秒目标的差异原样保留,不称连续无间隙监控。
## 账本与限制
实际training GPU=47943.94719455298036秒(13.317763109598小时),等于五entry之和和terminal-ledger。最后投影87429.69551990522秒仍PASS;总帽162000、训练154800,剩余总额114056.05280544701964秒,其中postfit7200未用,训练余额106856.05280544701964秒。余额仅账本事实,不自动授权重试。
worker CPU合计46192.559663秒;guard charged合计52274.50033081201秒已包含worker,二者不能相加。STOP协调器路径未序列化coordinator CPU,原限制明确保留,不编造完整CPU总计。后处理GPU=0,无预测/评分/测试目标访问。
## 不可变证据与下一裁量
928原文件199235784B(含3个checkpoint)及固定源完整本地冻结`a0a32fbf2d438cc6d39715cb2742f74ee415ebfe`。审计文件`/home/frank14f/MeltMind-runs/track-a-h2-2026-11-18/infrastructure-stop-audit.json`,SHA256 `d0433cb52d2695fcc5a9bb885521ff7fda2aea17524e894e7218d23a75ebe1ea`。原contract/source/review保持,原no_retry不改。
已在原Slack串1795000441.434319实际@Claude/TRON提交分类与账本,请Claude裁量三PASS是否哈希复用、两基础设施中断折如何处理及监控工程边界。本次未实现修复/恢复入口,没有重启原队列、启动postfit或新增科学任务。
seed1 v2原51pins校验PASS,审计时原队列fold0/1/2约88/67/5、真实guard/worker仍在;空闲卡利用由原调度器自行完成,未干扰。以上为时点快照,后续状态以其原回执为准。
@@ -0,0 +1,45 @@
{
"status": "STOPPED",
"reason": "Track A training incomplete",
"entries": [
{
"id": 1,
"status": "PASS",
"returncode": 0,
"gpu_seconds_charged": 11701.193322008941,
"guard_complete_sha256": "9259261f44a5813451e9f6f7878fd69d65db6e509889f7c78adcb6f417e00b3d"
},
{
"id": 0,
"status": "PASS",
"returncode": 0,
"gpu_seconds_charged": 11847.188839585986,
"guard_complete_sha256": "024bb110576da7c351ad8e98227cace2f40f4247a8f349c01ea4ebfde954891c"
},
{
"id": 2,
"status": "PASS",
"returncode": 0,
"gpu_seconds_charged": 12059.386084717,
"guard_complete_sha256": "b60b2575b9efdbb28eabcc2682dc04ce3519c9e7b2f6954cc4dcea0e29b1ab2f"
},
{
"id": 4,
"status": "STOPPED_OR_FAILED",
"returncode": 1,
"gpu_seconds_charged": 94.24631198402494,
"guard_complete_sha256": "c0a5681605747343a03d8e0fc7829947a8c4a8e56f4a7acbc668ae3f23dae2cc"
},
{
"id": 3,
"status": "STOPPED_OR_FAILED",
"returncode": 1,
"gpu_seconds_charged": 12241.932636257028,
"guard_complete_sha256": "a5aabd667d40ae538ae7dce38ae436943887fa67ffb5c0ab63090b8a4ec4d091"
}
],
"never_attempted": [],
"training_gpu_seconds": 47943.94719455298,
"postfit_started": false,
"no_retry": true
}
@@ -0,0 +1,183 @@
{
"status": "FIRST_TWO_FIRST_EPOCH_GATES_PASS",
"unix": 1794976204.9741762,
"jobs": [
{
"job": 0,
"receipts": {
"first-epoch-projection.json": {
"sha256": "4c71e8ed5d2a661a27805f2a7bac1275125fcd01e78e4789808952bf681cb471",
"value": {
"status": "PASS",
"rows": [
{
"id": 0,
"projected_gpu_seconds": 15642.152183703787,
"within_cap": true
},
{
"id": 1,
"projected_gpu_seconds": 14993.993712468538,
"within_cap": true
},
{
"id": 2,
"projected_gpu_seconds": 30960,
"within_cap": true
},
{
"id": 3,
"projected_gpu_seconds": 30960,
"within_cap": true
},
{
"id": 4,
"projected_gpu_seconds": 30960,
"within_cap": true
}
],
"projected_gpu_seconds": 130716.14589617233,
"postfit_reserve": 7200,
"cell": {
"id": 0,
"projected_gpu_seconds": 15642.152183703787,
"within_cap": true
},
"contract_sha256": "be4ac1f6d5a5d6c0183aa50d2e046b22d7e5fa77c7714fef7eab6d616e68bee7",
"unix": 1794976169.9310362
}
},
"worker/history/001.json": {
"sha256": "cee50d369b82874ffa7f8488f242e6d37820aab9eefc0052fe7fce90ba173637",
"value": {
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 1,
"optimizer_steps": 311,
"wall_seconds": 66.29813926306088,
"cpu_seconds": 66.160619786,
"train_mean_weighted_loss": 0.005536371032827926,
"train_epoch_seconds": 61.848708607954904,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0001,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1516240896
}
},
"worker/progress.json": {
"sha256": "ba2c87fa12e798a02e2bf3fb679e9354117e67899a2a0634fc4518917cef8e7b",
"value": {
"fold": 0,
"stage": "TRAIN",
"epoch": 2,
"optimizer_steps": 498,
"wall_seconds": 101.74945049802773,
"cpu_seconds": 101.108015754,
"case": "00001020_ca4cf9693d6d4a98b515ba99_step_001",
"epoch_parts": 187,
"epoch_nodes": 5280849,
"lr": 0.00019999999999999998,
"loss": 0.003331501968204975,
"gradient_norm_before_clip": 0.10909295827150345,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1516240896
}
}
}
},
{
"job": 1,
"receipts": {
"first-epoch-projection.json": {
"sha256": "6e90df3d3463c78d041ce2b77cec1bed9793ab0c738b1b953ff6648dc1985d00",
"value": {
"status": "PASS",
"rows": [
{
"id": 0,
"projected_gpu_seconds": 30960,
"within_cap": true
},
{
"id": 1,
"projected_gpu_seconds": 14993.993712468538,
"within_cap": true
},
{
"id": 2,
"projected_gpu_seconds": 30960,
"within_cap": true
},
{
"id": 3,
"projected_gpu_seconds": 30960,
"within_cap": true
},
{
"id": 4,
"projected_gpu_seconds": 30960,
"within_cap": true
}
],
"projected_gpu_seconds": 146033.99371246854,
"postfit_reserve": 7200,
"cell": {
"id": 1,
"projected_gpu_seconds": 14993.993712468538,
"within_cap": true
},
"contract_sha256": "be4ac1f6d5a5d6c0183aa50d2e046b22d7e5fa77c7714fef7eab6d616e68bee7",
"unix": 1794976166.8627698
}
},
"worker/history/001.json": {
"sha256": "171159d214879aeed9049bf0ec211b728eb9aa7564b1355ccff92abc4e047777",
"value": {
"fold": 1,
"stage": "EPOCH_COMPLETE",
"epoch": 1,
"optimizer_steps": 304,
"wall_seconds": 63.57294307497796,
"cpu_seconds": 63.397418468,
"train_mean_weighted_loss": 0.006312397840079747,
"train_epoch_seconds": 59.255394292995334,
"epoch_parts": 304,
"epoch_nodes": 8548261,
"lr": 0.0001,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1516240896
}
},
"worker/progress.json": {
"sha256": "55e5cce489f0d3e9395ce341fdf2022f517ea5121290e0b871572188e1b9022e",
"value": {
"fold": 1,
"stage": "TRAIN",
"epoch": 2,
"optimizer_steps": 509,
"wall_seconds": 102.50715935300104,
"cpu_seconds": 101.3272712,
"case": "00000778_a25fb37392a44447b9684f16_step_003",
"epoch_parts": 205,
"epoch_nodes": 5880915,
"lr": 0.00019999999999999998,
"loss": 0.0022961196955293417,
"gradient_norm_before_clip": 0.0613572783768177,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1516240896
}
}
}
}
],
"pending": [
2,
3,
4
],
"pending_full_budget_reserved": true,
"all_five_costs_measured": false,
"no_source_changes": true,
"postfit_started": false
}
@@ -0,0 +1,5 @@
{
"reason": "JOB_4_FAILED",
"unix": 1794999913.5796757,
"no_retry": true
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,4 @@
{
"contract_sha256": "be4ac1f6d5a5d6c0183aa50d2e046b22d7e5fa77c7714fef7eab6d616e68bee7",
"proof_sha256": "72eec78b4a6f6c803a2f30a7fd9a7e0acbde94473c182ec9eb25ec22e8f2897a"
}
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
Whitespace-only changes.
@@ -0,0 +1,10 @@
{
"cpu_seconds_charged": 11773.243537,
"attempts": [
{
"name": "train",
"status": "PASS",
"cpu_seconds": 11773.243537
}
]
}
Whitespace-only changes.
@@ -0,0 +1,25 @@
{
"status": "PASS",
"command": [
"/home/frank14f/optane/MeltMind/artifacts/gre24-reproduction-2026-11-15/environment/venv/bin/python",
"tools/track_a_h2_worker.py",
"/home/frank14f/MeltMind-runs/track-a-h2-2026-11-18",
"0"
],
"returncode": 0,
"stop_reason": null,
"cpu_seconds": 11773.243537,
"wall_seconds": 11778.565742944018,
"peak_tree_rss_bytes": 1466642432,
"peak_process_count": 1,
"output_disk_bytes_before_receipt": 4096,
"max_poll_gap_seconds": 2.576041355030611,
"poll_target_seconds": 0.2,
"reaped_orphan_cpu_seconds": 0.0,
"supervision": "sampled process-tree monitoring; wait4 final CPU/HWM; overshoot retained as failure",
"gpu_visible": true,
"gpu_uuid": "GPU-b8c11c01-c91f-aa0b-abf6-dad35085cc76",
"retry": false,
"monitor_errors": 0,
"last_monitor_error": null
}
@@ -0,0 +1,39 @@
{
"status": "PASS",
"rows": [
{
"id": 0,
"projected_gpu_seconds": 11965.050868351944,
"within_cap": true
},
{
"id": 1,
"projected_gpu_seconds": 11701.193322008941,
"within_cap": true
},
{
"id": 2,
"projected_gpu_seconds": 30960,
"within_cap": true
},
{
"id": 3,
"projected_gpu_seconds": 15944.660040876304,
"within_cap": true
},
{
"id": 4,
"projected_gpu_seconds": 30960,
"within_cap": true
}
],
"projected_gpu_seconds": 108730.90423123719,
"postfit_reserve": 7200,
"cell": {
"id": 0,
"projected_gpu_seconds": 11965.050868351944,
"within_cap": true
},
"contract_sha256": "be4ac1f6d5a5d6c0183aa50d2e046b22d7e5fa77c7714fef7eab6d616e68bee7",
"unix": 1794987877.6960218
}
@@ -0,0 +1,14 @@
{
"stage": "TRACK_A_TRAINING",
"fold": 0,
"elapsed_gpu_seconds": 11842.330767216044,
"device_bytes": 1930428416,
"device_utilization": 0,
"tree_rss_bytes": 1762070528,
"recorded_peaks": {
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
},
"reasons": [],
"unix": 1794987877.6225314
}
@@ -0,0 +1,11 @@
{
"torch": "2.5.1+cu124",
"gpu_uuid": "GPU-b8c11c01-c91f-aa0b-abf6-dad35085cc76",
"allocator_settings": "{'PYTORCH_CUDA_ALLOC_CONF': 'expandable_segments:True', 'max_split_size': -1, 'garbage_collection_threshold': 0.0, 'expandable_segments': True, 'pinned_num_register_threads': 1, 'release_lock_on_cudamalloc': False, 'pinned_use_cuda_host_register': False, 'roundup_power2_divisions': {'1': 0, '2': 0, '4': 0, '8': 0, '16': 0, '32': 0, '64': 0, '128': 0, '256': 0, '512': 0, '1024': 0, '2048': 0, '4096': 0, '8192': 0, '16384': 0, '32768': 0}}",
"model_dtype": "FP32",
"autocast": false,
"grad_scaler": false,
"fresh_seed": 0,
"pilot_weights_reused": false,
"parameter_count": 195077
}
@@ -0,0 +1,624 @@
{
"actual_fold": 0,
"learning": [
"00000008_9b3d6a97e8de4aa193b81000_step_000",
"00000009_9b3d6a97e8de4aa193b81000_step_001",
"00000011_e909f412cda24521865fac0f_step_000",
"00000023_ad34a3f60c4a4caa99646600_step_003",
"00000031_ad34a3f60c4a4caa99646600_step_011",
"00000057_666139e3bff64d4e8a6ce183_step_006",
"00000060_767e4372b5f94a88a7a17d90_step_001",
"00000068_767e4372b5f94a88a7a17d90_step_009",
"00000072_767e4372b5f94a88a7a17d90_step_013",
"00000077_767e4372b5f94a88a7a17d90_step_018",
"00000093_169bec34317b46e98eb1a403_step_005",
"00000111_ce24e0cbec8e45c89735d148_step_003",
"00000115_ce24e0cbec8e45c89735d148_step_007",
"00000120_ce24e0cbec8e45c89735d148_step_012",
"00000133_08b739e0511f4305b161cd05_step_000",
"00000145_2fc54fcd110d4f49969163c4_step_008",
"00000153_a951550b8a1043a68423c541_step_003",
"00000154_a951550b8a1043a68423c541_step_004",
"00000157_8208fd8b44b64dce91322f84_step_000",
"00000161_a583c6c3cdc143bab3f6c856_step_000",
"00000163_ff9b2ed255484db78bb0e6da_step_000",
"00000164_91adbf180e114b698160c9a4_step_000",
"00000166_2246cab58dbe4a2fb00fed50_step_000",
"00000174_b21aa794b6b64d87a57c245e_step_004",
"00000175_b21aa794b6b64d87a57c245e_step_005",
"00000176_b21aa794b6b64d87a57c245e_step_006",
"00000181_b21aa794b6b64d87a57c245e_step_011",
"00000190_185ecbc783f34408aef52119_step_000",
"00000198_826494e894454504b0971fd1_step_001",
"00000221_5324c79b172f41e8a49afc6f_step_008",
"00000238_4829bc0645d6414d9b5cec31_step_001",
"00000241_4829bc0645d6414d9b5cec31_step_004",
"00000261_845fb06d7f2f4802a806a600_step_006",
"00000267_845fb06d7f2f4802a806a600_step_012",
"00000274_648a7c23a21f4e88bf25ffeb_step_002",
"00000275_648a7c23a21f4e88bf25ffeb_step_003",
"00000282_5a6ef3285b89452ea6b390e7_step_003",
"00000287_58f8ff46b7564e989f9695b7_step_000",
"00000290_58f8ff46b7564e989f9695b7_step_003",
"00000293_58f8ff46b7564e989f9695b7_step_006",
"00000302_39b08d62a677414f86fdae52_step_000",
"00000307_3062bccff48e47a2b9de05e3_step_002",
"00000308_3062bccff48e47a2b9de05e3_step_003",
"00000310_3062bccff48e47a2b9de05e3_step_005",
"00000311_3062bccff48e47a2b9de05e3_step_006",
"00000312_3062bccff48e47a2b9de05e3_step_007",
"00000316_3062bccff48e47a2b9de05e3_step_011",
"00000321_3062bccff48e47a2b9de05e3_step_016",
"00000322_3062bccff48e47a2b9de05e3_step_017",
"00000323_3062bccff48e47a2b9de05e3_step_018",
"00000325_3062bccff48e47a2b9de05e3_step_020",
"00000329_12a935cd9b684f879c0505fe_step_000",
"00000331_12a935cd9b684f879c0505fe_step_002",
"00000342_fe70be720f7e49fa90ded969_step_001",
"00000355_68d1a5c2319a4a419ff625c6_step_001",
"00000356_68d1a5c2319a4a419ff625c6_step_002",
"00000357_68d1a5c2319a4a419ff625c6_step_003",
"00000364_38d899e2706a4d09960705dd_step_001",
"00000374_23c957e8e7b2428282a13ae2_step_007",
"00000375_23c957e8e7b2428282a13ae2_step_008",
"00000376_23c957e8e7b2428282a13ae2_step_009",
"00000378_23c957e8e7b2428282a13ae2_step_011",
"00000387_239bbf0e92974a32bd214366_step_008",
"00000388_239bbf0e92974a32bd214366_step_009",
"00000391_fd490b564a44413884fab719_step_000",
"00000405_f3759bbb4682423caf207e8e_step_002",
"00000406_f3759bbb4682423caf207e8e_step_003",
"00000408_36bcb4030e36417b8dc74b87_step_000",
"00000410_30afd12eede44fa8a1def049_step_000",
"00000413_94f421b1b33d47c592c9fe1d_step_000",
"00000421_bd0972033f464f8f961a150a_step_002",
"00000444_4ed4c78d6d754aac90876fc2_step_003",
"00000459_516b1784884d4facbc4c0d74_step_007",
"00000462_516b1784884d4facbc4c0d74_step_010",
"00000466_11951e51b89249dba38ded3e_step_001",
"00000469_11951e51b89249dba38ded3e_step_004",
"00000470_11951e51b89249dba38ded3e_step_005",
"00000478_0fc8cc17c6af48dabe36c8f2_step_003",
"00000488_a7d0c851b5f24a97a7cac950_step_005",
"00000489_a7d0c851b5f24a97a7cac950_step_006",
"00000492_a7d0c851b5f24a97a7cac950_step_009",
"00000501_5499914343bd4f549b38046d_step_002",
"00000513_f78bf7cf156f4231988af062_step_001",
"00000519_5b8fa6b193d14279a77a0e50_step_000",
"00000522_a56dd4152b7e4ccf91212d7b_step_000",
"00000526_f9d6559308dc4c85aab9fa38_step_000",
"00000527_db75f3a38d9746a1bd66168d_step_000",
"00000531_bc0d6937206b43bc96521413_step_003",
"00000542_4b1756f6e72b45c29a8e4eb4_step_000",
"00000543_8cf6554d440d40c4823dad6c_step_000",
"00000562_1724088724bf4e53a36e363e_step_000",
"00000566_2f2b015ef4904c1caef56154_step_000",
"00000574_5a4cef68211d4706b1ec8586_step_003",
"00000579_5a4cef68211d4706b1ec8586_step_008",
"00000582_5a4cef68211d4706b1ec8586_step_011",
"00000587_2417b410de78416ebe50d460_step_000",
"00000591_2417b410de78416ebe50d460_step_004",
"00000597_7fe894812c4244e18241aa08_step_003",
"00000601_7fe894812c4244e18241aa08_step_007",
"00000602_7fe894812c4244e18241aa08_step_008",
"00000605_7fe894812c4244e18241aa08_step_011",
"00000606_7fe894812c4244e18241aa08_step_012",
"00000610_04e799936ae446b9a3d8d2b8_step_001",
"00000616_c710d414bbbc4340b682c58a_step_002",
"00000618_c710d414bbbc4340b682c58a_step_004",
"00000619_c710d414bbbc4340b682c58a_step_005",
"00000621_c710d414bbbc4340b682c58a_step_007",
"00000622_c710d414bbbc4340b682c58a_step_008",
"00000624_1eacb269d52a47f5b265c1ab_step_000",
"00000626_f686126d229f47aeb125b5ba_step_000",
"00000627_53bfa7e2e04c4d9dbf4a3db2_step_000",
"00000632_4f322b34e87c43289916c298_step_001",
"00000637_5ae45a997c76423aba2fcc11_step_002",
"00000638_5ae45a997c76423aba2fcc11_step_003",
"00000643_bb55eaa4e9e24c2ea9d18ee0_step_000",
"00000644_761a451dfea94bfa93e23cfe_step_000",
"00000658_a40da330321b44f286a27361_step_000",
"00000659_6fae5591b3034f949b7e8eae_step_000",
"00000683_8863c3497d6d4e05b0f41ddc_step_000",
"00000685_24615f5ff44d4cd5809e77a5_step_001",
"00000693_b9f3563b403143dbb86ab2b8_step_006",
"00000695_3f46763af0744ab2a90e6b48_step_000",
"00000696_32309198fce9481896bd0a99_step_000",
"00000698_6f0fc559e8f3422c908b6060_step_001",
"00000708_c08c1c78b86f4ee3bcb7d9de_step_004",
"00000709_c08c1c78b86f4ee3bcb7d9de_step_005",
"00000721_9a461e8ae7dc4d33a55883eb_step_000",
"00000737_989e0f807cde438face0cb00_step_002",
"00000738_989e0f807cde438face0cb00_step_003",
"00000759_8ab50540e9804365a8717ec0_step_006",
"00000760_8ab50540e9804365a8717ec0_step_007",
"00000766_9cab8335065e4d439bbff471_step_002",
"00000767_9cab8335065e4d439bbff471_step_003",
"00000778_a25fb37392a44447b9684f16_step_003",
"00000779_a25fb37392a44447b9684f16_step_004",
"00000782_a25fb37392a44447b9684f16_step_007",
"00000783_a25fb37392a44447b9684f16_step_008",
"00000786_a25fb37392a44447b9684f16_step_011",
"00000787_a25fb37392a44447b9684f16_step_012",
"00000788_a25fb37392a44447b9684f16_step_013",
"00000789_a25fb37392a44447b9684f16_step_014",
"00000790_a25fb37392a44447b9684f16_step_015",
"00000791_a25fb37392a44447b9684f16_step_016",
"00000792_a25fb37392a44447b9684f16_step_017",
"00000793_a25fb37392a44447b9684f16_step_018",
"00000797_e7533c87b9a347cdafd0c390_step_000",
"00000799_ad6107d9fc994ccca38cc4e4_step_001",
"00000801_ad6107d9fc994ccca38cc4e4_step_003",
"00000807_05cadeb104be4c8aa75ccc71_step_001",
"00000808_05cadeb104be4c8aa75ccc71_step_002",
"00000819_6bbad2756d7845bd8d731d60_step_001",
"00000829_889a065d51a24bb8ad704c69_step_000",
"00000830_889a065d51a24bb8ad704c69_step_001",
"00000837_b5409201f9a5495fa169af85_step_000",
"00000842_2c1efcc32d094e9fb9e1a88e_step_002",
"00000843_2c1efcc32d094e9fb9e1a88e_step_003",
"00000844_2c1efcc32d094e9fb9e1a88e_step_004",
"00000850_2c1efcc32d094e9fb9e1a88e_step_010",
"00000855_2c1efcc32d094e9fb9e1a88e_step_015",
"00000856_2c1efcc32d094e9fb9e1a88e_step_016",
"00000859_58ac6a347bbe4cf5ae07a565_step_000",
"00000863_029c2575dd474650ba1b8d21_step_003",
"00000864_029c2575dd474650ba1b8d21_step_004",
"00000866_029c2575dd474650ba1b8d21_step_006",
"00000868_029c2575dd474650ba1b8d21_step_008",
"00000873_029c2575dd474650ba1b8d21_step_013",
"00000877_029c2575dd474650ba1b8d21_step_017",
"00000880_029c2575dd474650ba1b8d21_step_020",
"00000882_029c2575dd474650ba1b8d21_step_022",
"00000883_029c2575dd474650ba1b8d21_step_023",
"00000885_029c2575dd474650ba1b8d21_step_025",
"00000886_029c2575dd474650ba1b8d21_step_026",
"00000887_029c2575dd474650ba1b8d21_step_027",
"00000889_e2d3a4dc5f7c438bb240e9d1_step_000",
"00000890_e2d3a4dc5f7c438bb240e9d1_step_001",
"00000892_be26dc83f78947738b2bfabd_step_001",
"00000894_be26dc83f78947738b2bfabd_step_003",
"00000897_be26dc83f78947738b2bfabd_step_006",
"00000905_4df6ab9924674556bb4a509d_step_005",
"00000928_cdcebf32644443b9812097ba_step_012",
"00000942_f297a24e1a4446cf9d74efd9_step_001",
"00000948_f297a24e1a4446cf9d74efd9_step_007",
"00000953_ccc14f0fcff54b5985cc9113_step_000",
"00000955_a741c7966d2c40a8a511263e_step_001",
"00000961_a98d481268ea4524a7769731_step_000",
"00000973_209426c5054945fab8a4af1b_step_000",
"00000976_b85ec2ff588148c4abd8c8d7_step_000",
"00000986_0dd1386597104265ab67e6cf_step_009",
"00000987_0dd1386597104265ab67e6cf_step_010",
"00000988_0dd1386597104265ab67e6cf_step_011",
"00000993_ac4ba032d4b5431ca04f542c_step_000",
"00000994_ac4ba032d4b5431ca04f542c_step_001",
"00000995_ac4ba032d4b5431ca04f542c_step_002",
"00000996_6d71695ba3fd475c927a00a7_step_000",
"00000998_6962bc8aad2c4e27b046d5b4_step_000",
"00001004_6962bc8aad2c4e27b046d5b4_step_006",
"00001006_6962bc8aad2c4e27b046d5b4_step_008",
"00001011_6962bc8aad2c4e27b046d5b4_step_013",
"00001014_6962bc8aad2c4e27b046d5b4_step_016",
"00001019_ca4cf9693d6d4a98b515ba99_step_000",
"00001020_ca4cf9693d6d4a98b515ba99_step_001",
"00001021_ca4cf9693d6d4a98b515ba99_step_002",
"00001024_21adcacb9707488ea6655fd2_step_000",
"00001025_21adcacb9707488ea6655fd2_step_001",
"00001029_cde47df6e28543d1b434f206_step_001",
"00001033_18151464e6434759b7c2ece7_step_000",
"00001034_085828cd621d404a90ccd883_step_000",
"00001049_2b2638f82b3946b1867f2d11_step_005",
"00001053_2fd6155fa34f418083d0bd0f_step_002",
"00001062_c04956ddff6c4324b5cad67c_step_000",
"00001064_c6fda10dc58448d1951c5cc3_step_001",
"00001065_c6fda10dc58448d1951c5cc3_step_002",
"00001069_8a2edcf36bcd435092f4c8ae_step_003",
"00001097_256ee7ec3a4b4972a8b87c8d_step_001",
"00001099_256ee7ec3a4b4972a8b87c8d_step_003",
"00001106_ba88e7579ced4a0aa2de629c_step_000",
"00001108_412675e2ea594e70bbd8cf6c_step_000",
"00001112_3bd4a008a3a0483d86ccab3f_step_000",
"00001121_da4037e40839451dbbc2bd1f_step_000",
"00001140_0b0ed9b0ccbb455d82c60d69_step_001",
"00001141_0b0ed9b0ccbb455d82c60d69_step_002",
"00001159_f112d6f3e0c644ae9de2a724_step_000",
"00001167_107f61d0402f4c9a9d9d1342_step_000",
"00001168_107f61d0402f4c9a9d9d1342_step_001",
"00001170_0889c458f3f74dc288d2d531_step_001",
"00001177_9f1ce06d5e334fe88e1581aa_step_001",
"00001182_bc0b5846e3d84904ad02e547_step_001",
"00001185_507b60b57791425bbb73273e_step_000",
"00001190_507b60b57791425bbb73273e_step_005",
"00001193_46b17dd88ed24f55bca2eb69_step_000",
"00001200_e44a0f9afbe84b918f789f0f_step_000",
"00001210_4be610c137ba42ae9933e69b_step_000",
"00001211_4be610c137ba42ae9933e69b_step_001",
"00001212_4be610c137ba42ae9933e69b_step_002",
"00001214_9edb90c9f417497fb70bddf6_step_000",
"00001232_73b40f9a4cd3419ab3b42b73_step_001",
"00001235_17ea801126da4654b34f6c52_step_000",
"00001238_96503b6a26c64074ad598927_step_000",
"00001239_96503b6a26c64074ad598927_step_001",
"00001240_96503b6a26c64074ad598927_step_002",
"00001249_96503b6a26c64074ad598927_step_011",
"00001253_96503b6a26c64074ad598927_step_015",
"00001260_a4715d363f88400c8f0238d1_step_003",
"00001263_157e5348bf6545e282472856_step_001",
"00001266_b53ece83d8964b44bbf1f8ed_step_002",
"00001273_b53ece83d8964b44bbf1f8ed_step_009",
"00001274_b53ece83d8964b44bbf1f8ed_step_010",
"00001285_b53ece83d8964b44bbf1f8ed_step_021",
"00001294_b53ece83d8964b44bbf1f8ed_step_030",
"00001298_b53ece83d8964b44bbf1f8ed_step_034",
"00001300_b9951da825954791b59b979d_step_000",
"00001314_a9ed5e56a6a8414f91b39bdb_step_000",
"00001319_6a5a7dbad03b4a57b33f2c0d_step_002",
"00001333_beab79941e884455a62af40c_step_001",
"00001335_beab79941e884455a62af40c_step_003",
"00001336_beab79941e884455a62af40c_step_004",
"00001338_beab79941e884455a62af40c_step_006",
"00001339_beab79941e884455a62af40c_step_007",
"00001344_d8d97699a52842539e08d117_step_002",
"00001348_318b81a8e412438183201be7_step_000",
"00001349_27327ddba7f64417ae1e69a3_step_000",
"00001352_4e1d56bc2cef49ea84626b11_step_000",
"00001354_4e1d56bc2cef49ea84626b11_step_002",
"00001360_2e314c6e080b40fcb6a6db96_step_003",
"00001375_2e314c6e080b40fcb6a6db96_step_018",
"00001377_2e314c6e080b40fcb6a6db96_step_020",
"00001380_c49484307f434da78fa06b1f_step_000",
"00001387_60cf3ad636dd42e0bcec6946_step_000",
"00001391_406eb61c554a429ebd8a4d4f_step_000",
"00001393_406eb61c554a429ebd8a4d4f_step_002",
"00001394_406eb61c554a429ebd8a4d4f_step_003",
"00001398_49414db2dd594dc6a3907cf3_step_001",
"00001399_49414db2dd594dc6a3907cf3_step_002",
"00001401_49414db2dd594dc6a3907cf3_step_004",
"00001408_49414db2dd594dc6a3907cf3_step_011",
"00001409_49414db2dd594dc6a3907cf3_step_012",
"00001411_49414db2dd594dc6a3907cf3_step_014",
"00001413_4642a707b7894949ba816427_step_000",
"00001414_b38f661707674b8eb983d992_step_000",
"00001422_a45f6c99e2154f3b94955175_step_006",
"00001430_a45f6c99e2154f3b94955175_step_014",
"00001433_a45f6c99e2154f3b94955175_step_017",
"00001434_a45f6c99e2154f3b94955175_step_018",
"00001438_a45f6c99e2154f3b94955175_step_022",
"00001450_a45f6c99e2154f3b94955175_step_034",
"00001455_a45f6c99e2154f3b94955175_step_039",
"00001456_a45f6c99e2154f3b94955175_step_040",
"00001459_a45f6c99e2154f3b94955175_step_043",
"00001471_a45f6c99e2154f3b94955175_step_055",
"00001473_a45f6c99e2154f3b94955175_step_057",
"00001481_a45f6c99e2154f3b94955175_step_065",
"00001482_a45f6c99e2154f3b94955175_step_066",
"00001483_a45f6c99e2154f3b94955175_step_067",
"00001495_6c8ef6d2f5564d139a5077f8_step_006",
"00001496_eb4e2b01aaca4131bfc64adb_step_000",
"00001500_eb4e2b01aaca4131bfc64adb_step_004",
"00001503_6a0f221e85e04411afe23f51_step_002",
"00001511_5cbe58077e4b4885a5c213b3_step_004",
"00001522_7079a898746d41ba8ae13d69_step_003",
"00001527_9214869844f14e83a9dce3ec_step_000",
"00001532_9f0a4a641e8f4cc588eb4315_step_000",
"00001538_e1d62958d7ca4ad18fefd54d_step_005",
"00001543_e1d62958d7ca4ad18fefd54d_step_010",
"00001550_a94d978cf62948ce80b417a9_step_000",
"00001552_b0a9c4d836294846891615fe_step_000",
"00001554_30de00089a3f4a339eddb68d_step_000",
"00001556_30de00089a3f4a339eddb68d_step_002",
"00001563_2e9dc7e9a1a54dd7ab901478_step_000",
"00001569_6fc579f3224d4d53a1b60710_step_000",
"00001579_a3b89691665043a08931897f_step_002",
"00001596_2128dbbd4bf34879913b3a99_step_002"
],
"validation": [],
"test": [
"00000003_1ffb81a71e5b402e966b9341_step_002",
"00000004_1ffb81a71e5b402e966b9341_step_003",
"00000006_d4fe04f0f5f84b52bd4f10e4_step_001",
"00000021_ad34a3f60c4a4caa99646600_step_001",
"00000024_ad34a3f60c4a4caa99646600_step_004",
"00000030_ad34a3f60c4a4caa99646600_step_010",
"00000049_aa208c45e8734584a147f595_step_000",
"00000058_666139e3bff64d4e8a6ce183_step_007",
"00000061_767e4372b5f94a88a7a17d90_step_002",
"00000065_767e4372b5f94a88a7a17d90_step_006",
"00000066_767e4372b5f94a88a7a17d90_step_007",
"00000067_767e4372b5f94a88a7a17d90_step_008",
"00000070_767e4372b5f94a88a7a17d90_step_011",
"00000078_767e4372b5f94a88a7a17d90_step_019",
"00000079_767e4372b5f94a88a7a17d90_step_020",
"00000087_b4f9335a60a14cecaa599159_step_002",
"00000088_169bec34317b46e98eb1a403_step_000",
"00000098_72cf0f9ca5324822a4d41a80_step_000",
"00000102_5ed74bccca6f4e89829bcb5e_step_002",
"00000103_52f7124261ae4c558f800775_step_000",
"00000104_52f7124261ae4c558f800775_step_001",
"00000105_52f7124261ae4c558f800775_step_002",
"00000106_52f7124261ae4c558f800775_step_003",
"00000116_ce24e0cbec8e45c89735d148_step_008",
"00000117_ce24e0cbec8e45c89735d148_step_009",
"00000119_ce24e0cbec8e45c89735d148_step_011",
"00000125_e18c7feacfb54f3880659b43_step_000",
"00000128_e18c7feacfb54f3880659b43_step_003",
"00000129_e18c7feacfb54f3880659b43_step_004",
"00000130_e18c7feacfb54f3880659b43_step_005",
"00000131_945244874ed84e599987daa9_step_000",
"00000132_438507a38e404bfdbd51a16b_step_000",
"00000140_2fc54fcd110d4f49969163c4_step_003",
"00000142_2fc54fcd110d4f49969163c4_step_005",
"00000144_2fc54fcd110d4f49969163c4_step_007",
"00000152_a951550b8a1043a68423c541_step_002",
"00000162_4dcda9fab6dd46cf98fb194e_step_000",
"00000165_4070a99d6ebe45f5bd5d71eb_step_000",
"00000170_b21aa794b6b64d87a57c245e_step_000",
"00000171_b21aa794b6b64d87a57c245e_step_001",
"00000173_b21aa794b6b64d87a57c245e_step_003",
"00000183_6c4172e8a17140f89d3b1c0a_step_000",
"00000196_c9b71ecdd0654716891b0db8_step_000",
"00000206_27417003e762409185cb2b69_step_000",
"00000220_5324c79b172f41e8a49afc6f_step_007",
"00000223_97b354a907114b7183faa9c4_step_001",
"00000224_97b354a907114b7183faa9c4_step_002",
"00000229_682d529d5e8c4387bc59b3a1_step_000",
"00000243_4829bc0645d6414d9b5cec31_step_006",
"00000248_e87554b09cdc48c3add39db9_step_000",
"00000249_07a2da801a60492fb1e5eeb3_step_000",
"00000250_641f03cbe6a145508c6ae055_step_000",
"00000257_845fb06d7f2f4802a806a600_step_002",
"00000258_845fb06d7f2f4802a806a600_step_003",
"00000259_845fb06d7f2f4802a806a600_step_004",
"00000262_845fb06d7f2f4802a806a600_step_007",
"00000264_845fb06d7f2f4802a806a600_step_009",
"00000265_845fb06d7f2f4802a806a600_step_010",
"00000266_845fb06d7f2f4802a806a600_step_011",
"00000269_845fb06d7f2f4802a806a600_step_014",
"00000272_648a7c23a21f4e88bf25ffeb_step_000",
"00000273_648a7c23a21f4e88bf25ffeb_step_001",
"00000276_648a7c23a21f4e88bf25ffeb_step_004",
"00000280_5a6ef3285b89452ea6b390e7_step_001",
"00000285_d215b7bc60ae468288e5f1ea_step_000",
"00000291_58f8ff46b7564e989f9695b7_step_004",
"00000295_52a153d2417747b5926ff9df_step_000",
"00000296_52a153d2417747b5926ff9df_step_001",
"00000301_52a153d2417747b5926ff9df_step_006",
"00000305_3062bccff48e47a2b9de05e3_step_000",
"00000306_3062bccff48e47a2b9de05e3_step_001",
"00000309_3062bccff48e47a2b9de05e3_step_004",
"00000314_3062bccff48e47a2b9de05e3_step_009",
"00000315_3062bccff48e47a2b9de05e3_step_010",
"00000317_3062bccff48e47a2b9de05e3_step_012",
"00000318_3062bccff48e47a2b9de05e3_step_013",
"00000319_3062bccff48e47a2b9de05e3_step_014",
"00000324_3062bccff48e47a2b9de05e3_step_019",
"00000330_12a935cd9b684f879c0505fe_step_001",
"00000336_1a821c3883f7459a9615fca4_step_000",
"00000337_153efbd85ca64c4fb6b7ed5f_step_000",
"00000346_4185972a944744d8a7a0f2b4_step_002",
"00000348_4185972a944744d8a7a0f2b4_step_004",
"00000352_4185972a944744d8a7a0f2b4_step_008",
"00000359_9ee540a56d79451e8026e1e6_step_001",
"00000367_23c957e8e7b2428282a13ae2_step_000",
"00000368_23c957e8e7b2428282a13ae2_step_001",
"00000369_23c957e8e7b2428282a13ae2_step_002",
"00000377_23c957e8e7b2428282a13ae2_step_010",
"00000379_239bbf0e92974a32bd214366_step_000",
"00000382_239bbf0e92974a32bd214366_step_003",
"00000383_239bbf0e92974a32bd214366_step_004",
"00000386_239bbf0e92974a32bd214366_step_007",
"00000411_305e45c9479f43979b0d7b53_step_000",
"00000418_ce5b31679ee845a69ba89d6f_step_000",
"00000422_8e8a96d437584c77b3cfd84f_step_000",
"00000423_8e8a96d437584c77b3cfd84f_step_001",
"00000424_8e8a96d437584c77b3cfd84f_step_002",
"00000445_4ed4c78d6d754aac90876fc2_step_004",
"00000474_a5639f2b84c445aea5da18ad_step_000",
"00000477_0fc8cc17c6af48dabe36c8f2_step_002",
"00000479_0fc8cc17c6af48dabe36c8f2_step_004",
"00000485_a7d0c851b5f24a97a7cac950_step_002",
"00000502_5499914343bd4f549b38046d_step_003",
"00000508_d1c0a44331fd41faa91d4c2e_step_000",
"00000509_d1c0a44331fd41faa91d4c2e_step_001",
"00000510_d1c0a44331fd41faa91d4c2e_step_002",
"00000523_1f8d7f343e634c11a80d23ac_step_000",
"00000530_bc0d6937206b43bc96521413_step_002",
"00000549_41a24c0b74a34217832f5af0_step_000",
"00000551_41a24c0b74a34217832f5af0_step_002",
"00000559_a1621e156ade4317ab6a5bc5_step_000",
"00000565_5b5119548f7b421e8f597438_step_000",
"00000578_5a4cef68211d4706b1ec8586_step_007",
"00000580_5a4cef68211d4706b1ec8586_step_009",
"00000583_5a4cef68211d4706b1ec8586_step_012",
"00000590_2417b410de78416ebe50d460_step_003",
"00000603_7fe894812c4244e18241aa08_step_009",
"00000609_04e799936ae446b9a3d8d2b8_step_000",
"00000611_04e799936ae446b9a3d8d2b8_step_002",
"00000613_189d1310aa8c4a2a92a21020_step_001",
"00000615_c710d414bbbc4340b682c58a_step_001",
"00000617_c710d414bbbc4340b682c58a_step_003",
"00000620_c710d414bbbc4340b682c58a_step_006",
"00000654_d87d11d9c3ca4ee5a88797ae_step_002",
"00000686_24615f5ff44d4cd5809e77a5_step_002",
"00000688_b9f3563b403143dbb86ab2b8_step_001",
"00000691_b9f3563b403143dbb86ab2b8_step_004",
"00000694_45f6100bfa3948e69405bb21_step_000",
"00000699_cae88f0aaaf34de597863810_step_000",
"00000710_c08c1c78b86f4ee3bcb7d9de_step_006",
"00000723_8ed79b6bcb024d1a9f2aa5e7_step_000",
"00000746_6cb3ddbb49974d7db0e97135_step_001",
"00000751_6f0d90054d40400593050930_step_000",
"00000752_6f0d90054d40400593050930_step_001",
"00000758_8ab50540e9804365a8717ec0_step_005",
"00000763_8ab50540e9804365a8717ec0_step_010",
"00000764_9cab8335065e4d439bbff471_step_000",
"00000765_9cab8335065e4d439bbff471_step_001",
"00000775_a25fb37392a44447b9684f16_step_000",
"00000776_a25fb37392a44447b9684f16_step_001",
"00000777_a25fb37392a44447b9684f16_step_002",
"00000780_a25fb37392a44447b9684f16_step_005",
"00000781_a25fb37392a44447b9684f16_step_006",
"00000784_a25fb37392a44447b9684f16_step_009",
"00000785_a25fb37392a44447b9684f16_step_010",
"00000796_dec80701adc84239b485449d_step_002",
"00000800_ad6107d9fc994ccca38cc4e4_step_002",
"00000802_ad6107d9fc994ccca38cc4e4_step_004",
"00000806_05cadeb104be4c8aa75ccc71_step_000",
"00000809_05cadeb104be4c8aa75ccc71_step_003",
"00000816_36e6258346004cabb4c091b2_step_004",
"00000817_36e6258346004cabb4c091b2_step_005",
"00000818_6bbad2756d7845bd8d731d60_step_000",
"00000821_6bbad2756d7845bd8d731d60_step_003",
"00000822_c98978894f544f89aa10633f_step_000",
"00000823_c98978894f544f89aa10633f_step_001",
"00000827_b0d209f7a8ad4b02910c317f_step_003",
"00000832_889a065d51a24bb8ad704c69_step_003",
"00000834_889a065d51a24bb8ad704c69_step_005",
"00000840_2c1efcc32d094e9fb9e1a88e_step_000",
"00000849_2c1efcc32d094e9fb9e1a88e_step_009",
"00000851_2c1efcc32d094e9fb9e1a88e_step_011",
"00000867_029c2575dd474650ba1b8d21_step_007",
"00000870_029c2575dd474650ba1b8d21_step_010",
"00000871_029c2575dd474650ba1b8d21_step_011",
"00000872_029c2575dd474650ba1b8d21_step_012",
"00000874_029c2575dd474650ba1b8d21_step_014",
"00000879_029c2575dd474650ba1b8d21_step_019",
"00000884_029c2575dd474650ba1b8d21_step_024",
"00000900_4df6ab9924674556bb4a509d_step_000",
"00000904_4df6ab9924674556bb4a509d_step_004",
"00000906_4df6ab9924674556bb4a509d_step_006",
"00000919_cdcebf32644443b9812097ba_step_003",
"00000923_cdcebf32644443b9812097ba_step_007",
"00000927_cdcebf32644443b9812097ba_step_011",
"00000938_4f351b53bfd84bb4a2f7809c_step_003",
"00000949_f297a24e1a4446cf9d74efd9_step_008",
"00000958_04906edf324a4bce8fc3f417_step_000",
"00000962_a98d481268ea4524a7769731_step_001",
"00000963_a98d481268ea4524a7769731_step_002",
"00000970_edacb215ec6949d8af6c8c68_step_000",
"00000984_0dd1386597104265ab67e6cf_step_007",
"00001000_6962bc8aad2c4e27b046d5b4_step_002",
"00001002_6962bc8aad2c4e27b046d5b4_step_004",
"00001003_6962bc8aad2c4e27b046d5b4_step_005",
"00001005_6962bc8aad2c4e27b046d5b4_step_007",
"00001009_6962bc8aad2c4e27b046d5b4_step_011",
"00001010_6962bc8aad2c4e27b046d5b4_step_012",
"00001013_6962bc8aad2c4e27b046d5b4_step_015",
"00001015_6962bc8aad2c4e27b046d5b4_step_017",
"00001026_0f133d8e7fd448708a554e28_step_000",
"00001027_0f133d8e7fd448708a554e28_step_001",
"00001030_cde47df6e28543d1b434f206_step_002",
"00001031_cde47df6e28543d1b434f206_step_003",
"00001035_be3ed2c9696e4e6a91b34f79_step_000",
"00001036_be3ed2c9696e4e6a91b34f79_step_001",
"00001040_74c42207fa764831b03f7658_step_000",
"00001041_6baf25f5fe2947e8927797d8_step_000",
"00001048_2b2638f82b3946b1867f2d11_step_004",
"00001050_2b2638f82b3946b1867f2d11_step_006",
"00001052_2fd6155fa34f418083d0bd0f_step_001",
"00001061_284effca5a19488fa8a228e2_step_002",
"00001063_c6fda10dc58448d1951c5cc3_step_000",
"00001067_8a2edcf36bcd435092f4c8ae_step_001",
"00001071_8a2edcf36bcd435092f4c8ae_step_005",
"00001102_256ee7ec3a4b4972a8b87c8d_step_006",
"00001107_78566795c3144719804f8d34_step_000",
"00001114_5f5cf6feacbc41fb88b2d873_step_000",
"00001130_c2db313c2ce749d784c610d2_step_002",
"00001139_0b0ed9b0ccbb455d82c60d69_step_000",
"00001146_274866da478e4a009bed8750_step_001",
"00001165_4b65bb9a02f6494aa897a53e_step_000",
"00001169_0889c458f3f74dc288d2d531_step_000",
"00001178_515d7e1aeb364a2d88083cf3_step_000",
"00001186_507b60b57791425bbb73273e_step_001",
"00001187_507b60b57791425bbb73273e_step_002",
"00001191_507b60b57791425bbb73273e_step_006",
"00001206_e44a0f9afbe84b918f789f0f_step_006",
"00001213_4be610c137ba42ae9933e69b_step_003",
"00001217_e517c61f1c49433bb81cf980_step_000",
"00001228_2fd836ae38b44c41bfbd9584_step_002",
"00001233_73b40f9a4cd3419ab3b42b73_step_002",
"00001241_96503b6a26c64074ad598927_step_003",
"00001242_96503b6a26c64074ad598927_step_004",
"00001243_96503b6a26c64074ad598927_step_005",
"00001244_96503b6a26c64074ad598927_step_006",
"00001246_96503b6a26c64074ad598927_step_008",
"00001247_96503b6a26c64074ad598927_step_009",
"00001252_96503b6a26c64074ad598927_step_014",
"00001254_c2da273d137e475abb1ab1f4_step_000",
"00001255_3df9015915ff4fa7b57f0a71_step_000",
"00001257_a4715d363f88400c8f0238d1_step_000",
"00001262_157e5348bf6545e282472856_step_000",
"00001276_b53ece83d8964b44bbf1f8ed_step_012",
"00001301_a54ee779467b42ea8709e1ca_step_000",
"00001305_3f3a389e9d5a4370b952bdce_step_001",
"00001310_3f3a389e9d5a4370b952bdce_step_006",
"00001311_e95b8713cc0b4436909c78e9_step_000",
"00001320_6a5a7dbad03b4a57b33f2c0d_step_003",
"00001321_7cef9b79a6b04db894fd74e5_step_000",
"00001324_5bf4a74af99f405392406e40_step_000",
"00001334_beab79941e884455a62af40c_step_002",
"00001340_beab79941e884455a62af40c_step_008",
"00001345_d8d97699a52842539e08d117_step_003",
"00001346_d8d97699a52842539e08d117_step_004",
"00001353_4e1d56bc2cef49ea84626b11_step_001",
"00001356_aa0de7c5e9e84d73816bdbde_step_000",
"00001358_2e314c6e080b40fcb6a6db96_step_001",
"00001368_2e314c6e080b40fcb6a6db96_step_011",
"00001369_2e314c6e080b40fcb6a6db96_step_012",
"00001371_2e314c6e080b40fcb6a6db96_step_014",
"00001373_2e314c6e080b40fcb6a6db96_step_016",
"00001379_5b1d6e2599ea47d498a04256_step_000",
"00001381_c49484307f434da78fa06b1f_step_001",
"00001389_21b525da345e4e62aebd906b_step_000",
"00001392_406eb61c554a429ebd8a4d4f_step_001",
"00001400_49414db2dd594dc6a3907cf3_step_003",
"00001402_49414db2dd594dc6a3907cf3_step_005",
"00001403_49414db2dd594dc6a3907cf3_step_006",
"00001404_49414db2dd594dc6a3907cf3_step_007",
"00001405_49414db2dd594dc6a3907cf3_step_008",
"00001406_49414db2dd594dc6a3907cf3_step_009",
"00001407_49414db2dd594dc6a3907cf3_step_010",
"00001410_49414db2dd594dc6a3907cf3_step_013",
"00001412_49414db2dd594dc6a3907cf3_step_015",
"00001417_a45f6c99e2154f3b94955175_step_001",
"00001427_a45f6c99e2154f3b94955175_step_011",
"00001428_a45f6c99e2154f3b94955175_step_012",
"00001429_a45f6c99e2154f3b94955175_step_013",
"00001432_a45f6c99e2154f3b94955175_step_016",
"00001445_a45f6c99e2154f3b94955175_step_029",
"00001447_a45f6c99e2154f3b94955175_step_031",
"00001449_a45f6c99e2154f3b94955175_step_033",
"00001453_a45f6c99e2154f3b94955175_step_037",
"00001461_a45f6c99e2154f3b94955175_step_045",
"00001467_a45f6c99e2154f3b94955175_step_051",
"00001468_a45f6c99e2154f3b94955175_step_052",
"00001469_a45f6c99e2154f3b94955175_step_053",
"00001472_a45f6c99e2154f3b94955175_step_056",
"00001474_a45f6c99e2154f3b94955175_step_058",
"00001480_a45f6c99e2154f3b94955175_step_064",
"00001484_a45f6c99e2154f3b94955175_step_068",
"00001487_a45f6c99e2154f3b94955175_step_071",
"00001490_6c8ef6d2f5564d139a5077f8_step_001",
"00001491_6c8ef6d2f5564d139a5077f8_step_002",
"00001497_eb4e2b01aaca4131bfc64adb_step_001",
"00001501_6a0f221e85e04411afe23f51_step_000",
"00001505_6d02705716e04d9081f4980b_step_000",
"00001512_5cbe58077e4b4885a5c213b3_step_005",
"00001525_e0d35a03b46745259cf8878f_step_000",
"00001534_e1d62958d7ca4ad18fefd54d_step_001",
"00001535_e1d62958d7ca4ad18fefd54d_step_002",
"00001553_adfa1c33ae4d4d0aa441ddc3_step_000",
"00001555_30de00089a3f4a339eddb68d_step_001",
"00001557_16c191aef5034779aaab6e3f_step_000",
"00001567_9280741c840d4aa1af32658e_step_002",
"00001578_a3b89691665043a08931897f_step_001",
"00001582_a3b89691665043a08931897f_step_005",
"00001586_763da6a559544098acbf4154_step_002",
"00001592_35f4abaacc9447e4ac0b964f_step_001",
"00001594_2128dbbd4bf34879913b3a99_step_000",
"00001597_13bd23e0e4c740969b0c71ab_step_000",
"00001598_c118664e9f6f4943855293fa_step_000"
],
"clock_sha256": "d523928bcad46b21ec2c6db3522d1937c5a3d601676f3b362ecb78daa6f84e8e"
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 1,
"optimizer_steps": 311,
"wall_seconds": 66.29813926306088,
"cpu_seconds": 66.160619786,
"train_mean_weighted_loss": 0.005536371032827926,
"train_epoch_seconds": 61.848708607954904,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0001,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1516240896
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 2,
"optimizer_steps": 622,
"wall_seconds": 124.15732400200795,
"cpu_seconds": 123.51344017599999,
"train_mean_weighted_loss": 0.004889392921523456,
"train_epoch_seconds": 57.35501777601894,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.00019999999999999998,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 3,
"optimizer_steps": 933,
"wall_seconds": 180.42945110902656,
"cpu_seconds": 179.779946259,
"train_mean_weighted_loss": 0.004691685294478424,
"train_epoch_seconds": 56.271566351060756,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0003,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 4,
"optimizer_steps": 1244,
"wall_seconds": 238.35086372005753,
"cpu_seconds": 237.696224134,
"train_mean_weighted_loss": 0.0045925562550882095,
"train_epoch_seconds": 57.920835393015295,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0004,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 5,
"optimizer_steps": 1555,
"wall_seconds": 293.10901099897455,
"cpu_seconds": 292.443968418,
"train_mean_weighted_loss": 0.0043069594514434945,
"train_epoch_seconds": 54.757579599972814,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0005,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 6,
"optimizer_steps": 1866,
"wall_seconds": 348.7527438560501,
"cpu_seconds": 348.081727519,
"train_mean_weighted_loss": 0.004123815799795622,
"train_epoch_seconds": 55.64316035900265,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0006000000000000001,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 7,
"optimizer_steps": 2177,
"wall_seconds": 404.79169990797527,
"cpu_seconds": 404.115206312,
"train_mean_weighted_loss": 0.004416301625067109,
"train_epoch_seconds": 56.03834787802771,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007000000000000001,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 8,
"optimizer_steps": 2488,
"wall_seconds": 460.32385085104033,
"cpu_seconds": 459.638692224,
"train_mean_weighted_loss": 0.004342801111022802,
"train_epoch_seconds": 55.53153447189834,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 9,
"optimizer_steps": 2799,
"wall_seconds": 516.6686804830097,
"cpu_seconds": 515.978301883,
"train_mean_weighted_loss": 0.004010721321936164,
"train_epoch_seconds": 56.34422177798115,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 10,
"optimizer_steps": 3110,
"wall_seconds": 574.8606034059776,
"cpu_seconds": 574.16552932,
"train_mean_weighted_loss": 0.004030773956658739,
"train_epoch_seconds": 58.19134562602267,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.001,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 11,
"optimizer_steps": 3421,
"wall_seconds": 632.514700409025,
"cpu_seconds": 631.812884143,
"train_mean_weighted_loss": 0.003915810133654082,
"train_epoch_seconds": 57.65350357606076,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000999932335971272,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 12,
"optimizer_steps": 3732,
"wall_seconds": 688.8068664050661,
"cpu_seconds": 688.10113247,
"train_mean_weighted_loss": 0.003703818894680638,
"train_epoch_seconds": 56.291574383038096,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009997293623837584,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 13,
"optimizer_steps": 4043,
"wall_seconds": 746.179747557966,
"cpu_seconds": 745.4700566180001,
"train_mean_weighted_loss": 0.0037753406233511765,
"train_epoch_seconds": 57.372288508107886,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009993911347284108,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 14,
"optimizer_steps": 4354,
"wall_seconds": 804.487771620974,
"cpu_seconds": 803.7735574440001,
"train_mean_weighted_loss": 0.0038099247030328946,
"train_epoch_seconds": 58.30739646195434,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009989177454732933,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 15,
"optimizer_steps": 4665,
"wall_seconds": 859.9711029310711,
"cpu_seconds": 859.252627752,
"train_mean_weighted_loss": 0.0036861179022476772,
"train_epoch_seconds": 55.48274022806436,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009983093240383016,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 16,
"optimizer_steps": 4976,
"wall_seconds": 915.1564861950465,
"cpu_seconds": 914.423128044,
"train_mean_weighted_loss": 0.003607429089669907,
"train_epoch_seconds": 55.18480871303473,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009975660367597806,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 17,
"optimizer_steps": 5287,
"wall_seconds": 972.3705514200265,
"cpu_seconds": 971.633255147,
"train_mean_weighted_loss": 0.003520099541601316,
"train_epoch_seconds": 57.21348866005428,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000996688086845051,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 18,
"optimizer_steps": 5598,
"wall_seconds": 1027.9399859589757,
"cpu_seconds": 1027.1990461189998,
"train_mean_weighted_loss": 0.0034840636436658193,
"train_epoch_seconds": 55.56884702399839,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009956757143168527,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 19,
"optimizer_steps": 5909,
"wall_seconds": 1085.7161443270743,
"cpu_seconds": 1084.9710528179999,
"train_mean_weighted_loss": 0.003489970083836387,
"train_epoch_seconds": 57.77557372790761,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009945291959477269,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 20,
"optimizer_steps": 6220,
"wall_seconds": 1140.9214427530533,
"cpu_seconds": 1140.172647621,
"train_mean_weighted_loss": 0.0033653746809761925,
"train_epoch_seconds": 55.20472492801491,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009932488451843476,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 21,
"optimizer_steps": 6531,
"wall_seconds": 1196.4509679459734,
"cpu_seconds": 1195.6981555739999,
"train_mean_weighted_loss": 0.003814164798237833,
"train_epoch_seconds": 55.528886202024296,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009918350120618293,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 22,
"optimizer_steps": 6842,
"wall_seconds": 1251.941644886043,
"cpu_seconds": 1251.18597777,
"train_mean_weighted_loss": 0.0034182753608366185,
"train_epoch_seconds": 55.49005533498712,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009902880831080306,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 23,
"optimizer_steps": 7153,
"wall_seconds": 1309.8889389360556,
"cpu_seconds": 1309.130678094,
"train_mean_weighted_loss": 0.0032480277788396357,
"train_epoch_seconds": 57.94669203099329,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009886084812378818,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 24,
"optimizer_steps": 7464,
"wall_seconds": 1366.6676641129889,
"cpu_seconds": 1365.906442499,
"train_mean_weighted_loss": 0.0032462371543474297,
"train_epoch_seconds": 56.77808104804717,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009867966656377629,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 25,
"optimizer_steps": 7775,
"wall_seconds": 1422.631648294977,
"cpu_seconds": 1421.8669710919999,
"train_mean_weighted_loss": 0.0033709138553510553,
"train_epoch_seconds": 55.963369537028484,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009848531316399684,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 26,
"optimizer_steps": 8086,
"wall_seconds": 1479.5850508200238,
"cpu_seconds": 1478.816583305,
"train_mean_weighted_loss": 0.003425664389130721,
"train_epoch_seconds": 56.95263769896701,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009827784105872876,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 27,
"optimizer_steps": 8397,
"wall_seconds": 1536.3580456100171,
"cpu_seconds": 1535.586948662,
"train_mean_weighted_loss": 0.0034285730198579238,
"train_epoch_seconds": 56.772428613039665,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.00098057306968774,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 28,
"optimizer_steps": 8708,
"wall_seconds": 1591.3689561419887,
"cpu_seconds": 1590.5950908789998,
"train_mean_weighted_loss": 0.003225591575612674,
"train_epoch_seconds": 55.01031015091576,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000978237711859507,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 29,
"optimizer_steps": 9019,
"wall_seconds": 1649.7191747210454,
"cpu_seconds": 1648.9422212549998,
"train_mean_weighted_loss": 0.003148333391217277,
"train_epoch_seconds": 58.349619408953004,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009757729755661011,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 30,
"optimizer_steps": 9330,
"wall_seconds": 1707.383819097071,
"cpu_seconds": 1706.602456547,
"train_mean_weighted_loss": 0.0033110023010329117,
"train_epoch_seconds": 57.6640742609743,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009731795346418141,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 31,
"optimizer_steps": 9641,
"wall_seconds": 1763.9269456190523,
"cpu_seconds": 1763.1412428219999,
"train_mean_weighted_loss": 0.0030950826939497755,
"train_epoch_seconds": 56.542514736996964,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009704580981074999,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 32,
"optimizer_steps": 9952,
"wall_seconds": 1819.7357107569696,
"cpu_seconds": 1818.9475850959998,
"train_mean_weighted_loss": 0.0029214901605473494,
"train_epoch_seconds": 55.808194641023874,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009676094099767331,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 33,
"optimizer_steps": 10263,
"wall_seconds": 1875.0576626360416,
"cpu_seconds": 1874.264765859,
"train_mean_weighted_loss": 0.003200667634244739,
"train_epoch_seconds": 55.32139663991984,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000964634249052405,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 34,
"optimizer_steps": 10574,
"wall_seconds": 1931.07434238703,
"cpu_seconds": 1930.279384017,
"train_mean_weighted_loss": 0.0028956474367879334,
"train_epoch_seconds": 56.01602576696314,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009615334287138054,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 35,
"optimizer_steps": 10885,
"wall_seconds": 1986.3395145570394,
"cpu_seconds": 1985.541640084,
"train_mean_weighted_loss": 0.002984283413291968,
"train_epoch_seconds": 55.26458435005043,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009583077966942535,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 36,
"optimizer_steps": 11196,
"wall_seconds": 2042.704589007073,
"cpu_seconds": 2041.9041333859998,
"train_mean_weighted_loss": 0.0030681371022051557,
"train_epoch_seconds": 56.36445028008893,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009549582348493355,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 37,
"optimizer_steps": 11507,
"wall_seconds": 2097.9550633929903,
"cpu_seconds": 2097.152730419,
"train_mean_weighted_loss": 0.0032654866109621057,
"train_epoch_seconds": 55.249902345007285,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009514856589158148,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 38,
"optimizer_steps": 11818,
"wall_seconds": 2153.106378669967,
"cpu_seconds": 2152.300883631,
"train_mean_weighted_loss": 0.0028836906286294987,
"train_epoch_seconds": 55.150725771905854,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000947891018261278,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 39,
"optimizer_steps": 12129,
"wall_seconds": 2213.321683358052,
"cpu_seconds": 2212.511827231,
"train_mean_weighted_loss": 0.0029532496343630515,
"train_epoch_seconds": 60.21474614099134,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009441752956245879,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 40,
"optimizer_steps": 12440,
"wall_seconds": 2270.131811041967,
"cpu_seconds": 2269.311236814,
"train_mean_weighted_loss": 0.003073738670995975,
"train_epoch_seconds": 56.809556663036346,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009403395068472121,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 41,
"optimizer_steps": 12751,
"wall_seconds": 2326.215443012072,
"cpu_seconds": 2325.386420011,
"train_mean_weighted_loss": 0.002957939762072023,
"train_epoch_seconds": 56.08304485992994,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009363847005955023,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 42,
"optimizer_steps": 13062,
"wall_seconds": 2381.640802599024,
"cpu_seconds": 2380.8017825700003,
"train_mean_weighted_loss": 0.002542357172212245,
"train_epoch_seconds": 55.4247885060031,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000932311958073999,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 43,
"optimizer_steps": 13373,
"wall_seconds": 2435.9552576650167,
"cpu_seconds": 2435.107109095,
"train_mean_weighted_loss": 0.002941823187111397,
"train_epoch_seconds": 54.313842957955785,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009281223927298418,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 44,
"optimizer_steps": 13684,
"wall_seconds": 2490.7514865139965,
"cpu_seconds": 2489.8925777590002,
"train_mean_weighted_loss": 0.002937818304885213,
"train_epoch_seconds": 54.79565145506058,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000923817149948363,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 45,
"optimizer_steps": 13995,
"wall_seconds": 2549.3259650130058,
"cpu_seconds": 2548.458334888,
"train_mean_weighted_loss": 0.0027216249298982506,
"train_epoch_seconds": 58.57390510302503,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009193974067399517,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 46,
"optimizer_steps": 14306,
"wall_seconds": 2604.6425427059876,
"cpu_seconds": 2603.5106188590003,
"train_mean_weighted_loss": 0.002646489219531908,
"train_epoch_seconds": 55.31599332799669,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.00091486437141827,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 47,
"optimizer_steps": 14617,
"wall_seconds": 2659.403775647981,
"cpu_seconds": 2658.2629387280003,
"train_mean_weighted_loss": 0.0031091505158957434,
"train_epoch_seconds": 54.76065339392517,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009102192832699127,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 48,
"optimizer_steps": 14928,
"wall_seconds": 2717.158662558999,
"cpu_seconds": 2716.008164151,
"train_mean_weighted_loss": 0.002688926805223667,
"train_epoch_seconds": 57.754319917061366,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000905463412215599,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 49,
"optimizer_steps": 15239,
"wall_seconds": 2774.462440910982,
"cpu_seconds": 2773.303658491,
"train_mean_weighted_loss": 0.0025576336151313197,
"train_epoch_seconds": 57.303227548953146,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0009005980584629878,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 50,
"optimizer_steps": 15550,
"wall_seconds": 2829.914921710966,
"cpu_seconds": 2828.742865967,
"train_mean_weighted_loss": 0.0026941941615489895,
"train_epoch_seconds": 55.45191173197236,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008956245521512149,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 51,
"optimizer_steps": 15861,
"wall_seconds": 2885.8797778730514,
"cpu_seconds": 2884.6954730760003,
"train_mean_weighted_loss": 0.002446141616811822,
"train_epoch_seconds": 55.964267106959596,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008905442529872457,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 52,
"optimizer_steps": 16172,
"wall_seconds": 2941.6013749360573,
"cpu_seconds": 2940.407165655,
"train_mean_weighted_loss": 0.002296942295767923,
"train_epoch_seconds": 55.721006604027934,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008853585498741453,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 53,
"optimizer_steps": 16483,
"wall_seconds": 2996.2565909039695,
"cpu_seconds": 2995.052324542,
"train_mean_weighted_loss": 0.002752046630995587,
"train_epoch_seconds": 54.654659413034096,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008800688605313658,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 54,
"optimizer_steps": 16794,
"wall_seconds": 3051.3589634660166,
"cpu_seconds": 3050.143576244,
"train_mean_weighted_loss": 0.002491599861632699,
"train_epoch_seconds": 55.10179502808023,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008746766311071562,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 55,
"optimizer_steps": 17105,
"wall_seconds": 3107.150162909995,
"cpu_seconds": 3105.923963713,
"train_mean_weighted_loss": 0.0025196614915973803,
"train_epoch_seconds": 55.79063613701146,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008691833357832002,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 56,
"optimizer_steps": 17416,
"wall_seconds": 3161.6160512419883,
"cpu_seconds": 3160.380469007,
"train_mean_weighted_loss": 0.0024181597309156267,
"train_epoch_seconds": 54.46533364104107,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008635904763715891,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 57,
"optimizer_steps": 17727,
"wall_seconds": 3217.404881088063,
"cpu_seconds": 3216.1594624970003,
"train_mean_weighted_loss": 0.0025932617516539625,
"train_epoch_seconds": 55.788239968940616,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008578995819042416,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 58,
"optimizer_steps": 18038,
"wall_seconds": 3272.0662638390204,
"cpu_seconds": 3270.8120796480002,
"train_mean_weighted_loss": 0.0026917719307377433,
"train_epoch_seconds": 54.660613626940176,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008521122082148818,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 59,
"optimizer_steps": 18349,
"wall_seconds": 3326.8987117230427,
"cpu_seconds": 3325.634215268,
"train_mean_weighted_loss": 0.0024195504198985967,
"train_epoch_seconds": 54.83190955303144,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.00084622993751369,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 60,
"optimizer_steps": 18660,
"wall_seconds": 3381.346791852964,
"cpu_seconds": 3380.072707755,
"train_mean_weighted_loss": 0.0021264059955668855,
"train_epoch_seconds": 54.447503386065364,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000840254377954742,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 61,
"optimizer_steps": 18971,
"wall_seconds": 3434.506817481015,
"cpu_seconds": 3433.222890626,
"train_mean_weighted_loss": 0.0022798852635954317,
"train_epoch_seconds": 53.15946241794154,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000834187163196355,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 62,
"optimizer_steps": 19282,
"wall_seconds": 3488.58105941501,
"cpu_seconds": 3487.281248653,
"train_mean_weighted_loss": 0.0023334470438165994,
"train_epoch_seconds": 54.073643819079734,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008280299519544635,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 63,
"optimizer_steps": 19593,
"wall_seconds": 3544.60534358304,
"cpu_seconds": 3543.293881986,
"train_mean_weighted_loss": 0.0023273801369892663,
"train_epoch_seconds": 56.02369819500018,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008217844275491417,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 64,
"optimizer_steps": 19904,
"wall_seconds": 3601.0461193870287,
"cpu_seconds": 3599.7238774820003,
"train_mean_weighted_loss": 0.0021615127269770766,
"train_epoch_seconds": 56.44021801301278,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008154522974444009,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 65,
"optimizer_steps": 20215,
"wall_seconds": 3656.2174888449954,
"cpu_seconds": 3654.8853316990003,
"train_mean_weighted_loss": 0.0021085195369740357,
"train_epoch_seconds": 55.17080914101098,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008090352927813857,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 66,
"optimizer_steps": 20526,
"wall_seconds": 3711.176876681042,
"cpu_seconds": 3709.834289313,
"train_mean_weighted_loss": 0.002064140007573667,
"train_epoch_seconds": 54.95882580708712,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0008025351679050974,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 67,
"optimizer_steps": 20837,
"wall_seconds": 3767.1843163210433,
"cpu_seconds": 3765.8257895740003,
"train_mean_weighted_loss": 0.0019556205242666266,
"train_epoch_seconds": 56.006865005008876,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007959536998847743,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 68,
"optimizer_steps": 21148,
"wall_seconds": 3823.257234784076,
"cpu_seconds": 3821.88510682,
"train_mean_weighted_loss": 0.0020952074103038386,
"train_epoch_seconds": 56.07235930801835,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007892926880280571,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 69,
"optimizer_steps": 21459,
"wall_seconds": 3880.226587282028,
"cpu_seconds": 3878.843411871,
"train_mean_weighted_loss": 0.0018901118632644208,
"train_epoch_seconds": 56.968785225995816,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.000782553953389078,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 70,
"optimizer_steps": 21770,
"wall_seconds": 3938.4310566959903,
"cpu_seconds": 3937.038522402,
"train_mean_weighted_loss": 0.00196020021143738,
"train_epoch_seconds": 58.20387857907917,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007757393382706014,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 71,
"optimizer_steps": 22081,
"wall_seconds": 3995.397004571045,
"cpu_seconds": 3993.993182224,
"train_mean_weighted_loss": 0.0020781453383667808,
"train_epoch_seconds": 56.96531532204244,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007688507057203569,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 72,
"optimizer_steps": 22392,
"wall_seconds": 4051.590204279986,
"cpu_seconds": 4050.17605082,
"train_mean_weighted_loss": 0.00199873863176257,
"train_epoch_seconds": 56.19260987499729,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007618899390217015,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 73,
"optimizer_steps": 22703,
"wall_seconds": 4107.806691508042,
"cpu_seconds": 4106.382927307,
"train_mean_weighted_loss": 0.001916189166087709,
"train_epoch_seconds": 56.215924013988115,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007548589411787476,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 74,
"optimizer_steps": 23014,
"wall_seconds": 4164.293628916028,
"cpu_seconds": 4162.862143984999,
"train_mean_weighted_loss": 0.0018808565372728189,
"train_epoch_seconds": 56.48636630899273,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007477596343961015,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 75,
"optimizer_steps": 23325,
"wall_seconds": 4221.030488559976,
"cpu_seconds": 4219.552892766999,
"train_mean_weighted_loss": 0.0020018476376921688,
"train_epoch_seconds": 56.7362714119954,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007405939595533515,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 76,
"optimizer_steps": 23636,
"wall_seconds": 4279.130492378026,
"cpu_seconds": 4277.640741345999,
"train_mean_weighted_loss": 0.0018909594715922234,
"train_epoch_seconds": 58.09944974002428,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007333638756744505,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 77,
"optimizer_steps": 23947,
"wall_seconds": 4339.256927896058,
"cpu_seconds": 4337.743972986,
"train_mean_weighted_loss": 0.002132709233489968,
"train_epoch_seconds": 60.125865020090714,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007260713593921393,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 78,
"optimizer_steps": 24258,
"wall_seconds": 4397.245503138984,
"cpu_seconds": 4395.7183990739995,
"train_mean_weighted_loss": 0.0019891621138714245,
"train_epoch_seconds": 57.987989084096625,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007187184044075539,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 79,
"optimizer_steps": 24569,
"wall_seconds": 4454.20473248302,
"cpu_seconds": 4452.671040028999,
"train_mean_weighted_loss": 0.0019338765913805695,
"train_epoch_seconds": 56.95861076598521,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0007113070209451688,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 80,
"optimizer_steps": 24880,
"wall_seconds": 4515.467565760016,
"cpu_seconds": 4513.9222878069995,
"train_mean_weighted_loss": 0.0018550923320831067,
"train_epoch_seconds": 61.26222390506882,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.00070383923520322,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
@@ -0,0 +1,15 @@
{
"fold": 0,
"stage": "EPOCH_COMPLETE",
"epoch": 81,
"optimizer_steps": 25191,
"wall_seconds": 4573.862099966034,
"cpu_seconds": 4572.310333506,
"train_mean_weighted_loss": 0.0018401045023550792,
"train_epoch_seconds": 58.39392313105054,
"epoch_parts": 311,
"epoch_nodes": 8768067,
"lr": 0.0006963170887997639,
"peak_allocated_bytes": 1458252288,
"peak_reserved_bytes": 1518338048
}
Loaded 100 of 896 files, more files were not shown because too many files have changed in this diff. Show more