Files

16 lines
1.8 KiB
JSON

{
"status": "FAILED_NO_RETRY",
"error_type": "ValueError",
"message": "nonfinite encoder output",
"traceback": "Traceback (most recent call last):\n File \"/home/frank14f/optane/MeltMind/tools/run_s2_pilot_training.py\", line 105, in run\n loss=train_part(model,optimizer,data['base'],data['ids'],data['gaps'],data['edge'],data['target'],weights,scaler=scaler,amp=True,step_info=step_info)\n File \"/home/frank14f/optane/MeltMind/tools/s2_pilot_train.py\", line 20, in train_part\n prediction=model(base,ids,gaps,edge if model.arm=='E' else None)\n File \"/home/frank14f/anaconda3/envs/pycuda_3_10/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1736, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n File \"/home/frank14f/anaconda3/envs/pycuda_3_10/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1747, in _call_impl\n return forward_call(*args, **kwargs)\n File \"/home/frank14f/optane/MeltMind/tools/s2_models/network.py\", line 94, in forward\n embedding = self.encoder(base)[query_ids]\n File \"/home/frank14f/anaconda3/envs/pycuda_3_10/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1736, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n File \"/home/frank14f/anaconda3/envs/pycuda_3_10/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1747, in _call_impl\n return forward_call(*args, **kwargs)\n File \"/home/frank14f/optane/MeltMind/tools/s2_models/network.py\", line 63, in forward\n raise ValueError(\"nonfinite encoder output\")\nValueError: nonfinite encoder output\n",
"wall_seconds": 282.7322630760027,
"epoch": 10,
"amp": {
"attempted_steps": 371,
"skipped_steps": 0,
"successful_updates": 371,
"skipped_fraction": 0.0,
"consecutive_skipped": 0
}
}