DynamisLab/src/drl_pinball/train/TRAIN_PIPELINE.md

Current DRL training pipeline

This document describes the executable modern training path in this directory. It is a workflow specification, not evidence that the current source has been fully retrained or validated after later fixes. The release includes compact best-policy bundles for retained active runs; retained result summaries remain historical writing inputs rather than post-fix retraining evidence.

1. Scope and cases

The active path uses CelerisLab on a 2000×600 D2Q9 grid, uniform regularized inlet, free-slip y walls, MRT collision, double buffering, zero action bias, Gymnasium environments, Stable-Baselines3 PPO, and observation-only VecNormalize.

Canonical active IDs omit _sc:

Family	Cases	Config	SI
Karman baseline	kar_re100	configs/config_lbm_karman_2000x600.json	800
Karman cross-Re	kar_re60, kar_re200, kar_re400	matching _re60, _re200, _re400 JSON	800, 500, 400
Karman variable disturbance radius	kar_d075, kar_d15, kar_d2	base JSON	800
Illusion	ill_075L, ill_1L, ill_15L, ill_2L	base JSON	1100, 1200, 1200, 1200

Code-level Re uses reference length 2L0=40, so code re100 corresponds to physical cylinder-diameter Re_D=50. _tr is reserved for archived transfer experiments. _sc survives only in historical paths/artifacts and must not be introduced into active prose or new case IDs.

2. End-to-end flow

calibrate.py
  ├─ record calibrated target signal
  ├─ warm training geometry and snapshot it
  ├─ measure Stage0 (zero rotation)
  ├─ measure Stage1 (reference open loop)
  └─ write calibration.json + target artifacts
          │
          ▼
train_karman.py / train_illusion.py
  ├─ load the matched calibration and target artifacts
  ├─ build physical env → symmetry wrapper → DummyVecEnv → VecNormalize
  ├─ repeat: PPO learn chunk → same-env deterministic evaluation → checkpoint
  └─ write policy, paired normalizer, log, TensorBoard, metadata
          │
          ▼
../eval/ (separate inference/evaluation workflow)

Training and evaluation intentionally share one wrapped environment. This preserves CFD data continuity and avoids constructing a second GPU simulation, but evaluation is not side-effect-free: resetting/stepping changes environment state and, while VecNormalize.training remains true, updates observation running statistics. The policy weights do not update during evaluation.

3. Phase 0: calibration

calibrate.py performs the expensive target and baseline measurements. For Karman it records the wake of one upstream disturbance cylinder at three sensors. For Illusion it records a standalone target cylinder at three sensors plus target drag/lift, then stores a five-harmonic reconstruction of all eight channels.

Artifacts are a contract:

• calibration.json: geometry metadata, SI, scales, DTW mapping, reward constants, action mapping.
• target.npy: calibrated six-channel sensor target, shape (150, 6).
• target_harmonics.json: Illusion only; calibrated sensor/force harmonic model.
• calibrate.log: provenance log when calibration is run locally.

Target artifacts must come from the same calibrated case, config, geometry, SI, and force convention as calibration.json. Training fallback target recording is convenience behavior, not a substitute for producing and preserving a calibrated artifact set.

Calibration warms the target simulation for 4*NX/U0 = 800000 lattice steps. It then warms the training geometry, snapshots it, and measures:

1. Stage0 — zero rotation.
2. Stage1 — reference open-loop rotation ([0, 0.004, -0.004] Karman; [0, 0.005, -0.005] Illusion).

Stage numbers describe acquisition order, not quality. The code sorts their mean DTW similarities into worst_sim and better_sim before constructing SIM_BP; Stage1 is not assumed better. Karman falls back to the generic [0, .30, .65, .79, .89, 1] mapping when the measured spread is below 0.10 or the better baseline is below 0.5. Illusion uses its scene-specific sorted mapping and SIM_VAL=[0, .1, .35, .7, .85, 1].

Reward force sensitivities are:

• Karman: K_CD=50, K_CL=100.
• Illusion: K_CD=12, K_CL=25.

4. Geometry, observations, and actions

Karman contains disturbance cylinder + three sensors + three controlled cylinders. Its 12-vector is six controlled-cylinder forces followed by six sensor velocities. Drag and lift rewards use the average across the three controlled cylinders and target zero force.

Illusion contains three sensors + three controlled cylinders; the target cylinder exists only during target acquisition. Its 14-vector appends reconstructed target drag and lift to the same 12 channels. Its force reward compares the sum of all three controlled-cylinder forces with the single target-cylinder force. This wider mismatch motivates the lower 12/25 Gaussian constants.

The historical field name target_diam is misleading. The implementation passes target_diam * L0 to CelerisLab as the circle radius. Therefore values 0.75, 1.0, 1.5, 2.0 are radius ratios relative to L0=20, giving physical lattice radii 15, 20, 30, 40 and diameters 30, 40, 60, 80. Preserve this interpretation when reading or regenerating artifacts.

Actions are three normalized rotations:

omega = -(action * ACTION_SCALE) * U0 / RADIUS
ACTION_SCALE=12, U0=0.01, RADIUS=10

A 0.1 EMA smooths commanded omega. There is no legacy action bias.

5. Three normalization layers

The observation path has three distinct layers; they must not be collapsed conceptually:

1. Solver observation averaging — CelerisLab read_sensor(..., normalize=True) returns a sensor-area and sampling-time average; force reads are sampling-time averages.
2. Calibration pre-scaling — force channels divide by FORCE_SCALE; sensor channels divide by SENS_SCALE. Illusion target-force channels divide by FORCE_SCALE before entering the 14-vector. Current drl-pinball-calibration-v2 artifacts store native CelerisLab sensor units.
3. Online VecNormalize — running mean/variance whiten the complete observation (norm_obs=True, norm_reward=False, clip_obs=10, gamma=0.99). This is learned online during training; it is not the Phase 0 calibration.

Legacy schema-less calibrations instead used SENSOR_CC=78 to convert native sensor values into the old policy/DTW unit convention before pre-scaling. normalization.py keeps that compatibility path for an existing legacy policy plus its matching VecNormalize; the active v2 schema does not apply 78.

Inference must load the model's matching VecNormalize and freeze it (training=False). A “best model” without the normalizer state saved at the same best-selection point is incomplete.

6. PPO loop and shared evaluation

Defaults: sinusoidal MLP [64,64], n_steps=2048, learn_timesteps=2048, batch 64, 10 epochs, learning rate 3e-4, PPO gamma 0.995.

One outer total_episodes iteration is:

1. model.learn(learn_timesteps, reset_num_timesteps=False).
2. Disable symmetry probability on the same wrapped env.
3. Reset and run up to 360 policy steps with model.predict(..., deterministic=True).
4. Score mean reward over the last 180 steps.
5. If improved, save best_model.zip and root best_vecnormalize.pkl at the same point.
6. Save per-iteration model and normalizer checkpoints; finally save final_model.zip.

The environments never terminate a Gym episode themselves. “Episode” in logs and CLI means this outer PPO chunk, not a Gym episode.

Because evaluation uses the training VecNormalize, its observations contribute to running statistics. This is intentional and must be reproduced when continuing this workflow.

7. Symmetry status

SymmetryAugmentWrapper implements the up/down G transform. The supported training default is off (--symmetry-prob 0); enable it only as an explicit experiment.

History matters: legacy runs were configured with probability 0.5, but repeated model.learn(...) and evaluation resets reset the wrapper counter at each chunk. The mirror coin flip occurred only at the chunk boundary, after the last transition, so those models were effectively trained without augmented transitions. They must not be described as symmetry-augmented evidence.

8. Checkpoints, resume, and seeds

Each iteration pairs epNNNN_model.zip with epNNNN_vecnormalize.pkl. Resume loads both, reconstructs a new CFD environment, parses historical best reward from train.log, and continues at N+1.

Resume is policy-and-normalizer continuation. It is not exact state restoration: optimizer/RNG details, CFD process/context, environment trajectory, wrapper state, TensorBoard state, and wall-clock scheduling are not guaranteed to replay identically. Likewise, --seed differentiates stochastic runs; it is not a deterministic replay guarantee for GPU CFD + PyTorch + PPO.

Transfer loading remains in the Python entry points for historical compatibility, but active launchers train from scratch. Historical transfer scripts, logs, calibrations, and metadata live below archive/; see the warnings in CROSSRE_ANALYSIS.md and VARDIST_ANALYSIS.md.

9. Output contract and release status

A run directory contains calibration.json, train.log, tb/, meta.json, models/, and normalizer files. best_model.zip must travel with best_vecnormalize.pkl; final_model.zip uses final_vecnormalize.pkl, and per-iteration policies use the matching epNNNN_vecnormalize.pkl. The legacy root vec_normalize.pkl remains a final-state compatibility alias.

The repository publishes the source and canonical targets needed to reproduce the workflow, plus compact retained best-policy bundles (best_model.zip with the same-selection best_vecnormalize.pkl and provenance files). It does not publish intermediate/final checkpoints, full training logs, or TensorBoard streams. Any retained results/ summaries and figures are historical writing basis only. Do not infer current post-fix validation, complete retraining, reproducibility across hardware, or release qualification from them.