# V5 Eval Pipeline — Inference & Results Documentation

> **Scope**: Inference infrastructure for V5-trained PPO models on the new CelerisLab solver.
> Legacy model reproduction is handled by @src/drl_pinball/reproduce/.
> Companion to `train/TRAIN_PIPELINE.md` for the training infrastructure.

---

## 1. Architecture Overview

```
scene_manifest.py (Single Source of Truth)
└── TRAIN_SCENES[20 entries]         → used by infer_train.py

infer_train.py (GPU 2)
├── Create V5 CFD env
├── Per-seed:
│   ├── VecNormalize.load(seed.pkl)
│   ├── Skeleton PPO + weight inject
│   └── 360-step deterministic roll
├── Output: signals.npz + fields.npz + PNGs
└── Best-seed metrics.json

run_all_with_fields.sh
├── Phase 1: infer_train.py (stagger 120s per config change)
├── Phase 2: collect_baselines.py
├── Phase 3: bridge_to_sr.py
└── Phase 4: data integrity check
```

---

## 2. Scene Manifest (`scene_manifest.py`)

The single source of truth for all scene configurations used by both pipelines.

### 2.1 TRAIN_SCENES (V5 PPO, 2000×600, uniform + free-slip)

| scene_id | Type | SI | num_steps | Seeds | Description |
|----------|------|:--:|:---------:|:-----:|-------------|
| `kar_re100_sc` | karman | 800 | 360 | 41-45 | Karman Re100, 5-seed study |
| `kar_re60_tr` | karman | 800 | 360 | 43 | Cross-Re transfer to Re60 |
| `kar_re200_tr` | karman | 500 | 360 | 43 | Cross-Re transfer to Re200 |
| `kar_re400_tr` | karman | 400 | 360 | 43 | Cross-Re transfer to Re400 |
| `kar_d075_sc` | karman | 800 | 360 | 44 | VarDist d=0.75L scratch |
| `kar_d15_sc` | karman | 800 | 360 | 45 | VarDist d=1.5L scratch |
| `kar_d2_sc` | karman | 800 | 360 | 45 | VarDist d=2.0L scratch |
| `ill_075L_sc` | illusion | 400 | 360 | 43 | Illusion 0.75L target |
| `ill_1L_sc` | illusion | 600 | 360 | 43 | Illusion 1.0L target |
| `ill_15L_sc` | illusion | 800 | 360 | 43 | Illusion 1.5L target |
| `ill_2L_sc` | illusion | 800 | 360 | 43 | Illusion 2.0L target (new) — training bug known |

> **Note on illusion scenes**: Training bug — all target diameters were accidentally set to 1L. Transfer results invalid.

### 2.2 Legacy REPRODUCE_SCENES

Handled by @src/drl_pinball/reproduce/ — not by eval.

---

## 3. Train Inference Pipeline (`infer_train.py`)

### 3.1 Per-Scene Workflow

```
For each scene in TRAIN_SCENES:
  1. Load calibration.json + target.npy (+ target_harmonics.json for illusion)
  2. Create KarmanCloakEnv / IllusionCloakEnv (fresh CFD init)
  3. For each seed (seed_label, model_dir):
     a. Create skeleton PPO(env=vec_env, Sin, [64,64])
     b. Extract policy weights from best_model.zip
     c. Load VecNormalize from seed's vec_normalize.pkl (frozen)
     d. 360-step deterministic rollout
     e. Record: sensors, forces, actions, rewards, per-component r_cd/r_cl/r_sim
  4. Pick best seed by tail-180 avg reward
  5. Re-create env, load best seed model, run → capture vorticity PNGs
  6. Generate target vorticity (dist_cyl only / target cyl only)
  7. Generate zero-action baseline vorticity
  8. Write: signals.npz, metrics.json, all_seeds.json, vorticity_*.png
```

### 3.2 Skeleton Injection Pattern

Due to `numpy._core.numeric` cloudpickle deserialization issues with SB3 models trained on certain Python versions, `PPO.load()` may fail. The fallback is **skeleton injection**:

```python
# Method: skeleton PPO + manual weight injection
skeleton = PPO(
    "MlpPolicy",
    policy_kwargs={"activation_fn": Sin, "net_arch": [64, 64]},
    env=vec_env, device=device,
    n_steps=2048, batch_size=64, n_epochs=10,
    learning_rate=3e-4, gamma=0.995, verbose=0,
)
# Extract weights from zip
with zipfile.ZipFile("best_model.zip") as zf:
    with zf.open("policy.pth") as f:
        state_dict = torch.load(io.BytesIO(f.read()), map_location="cpu")
skeleton.policy.load_state_dict(state_dict, strict=False)

# VecNormalize loaded separately
vec_env = VecNormalize.load("vec_normalize.pkl", vec_env)
vec_env.training = False  # Frozen statistics
vec_env.norm_reward = False
```

### 3.3 Output Files per Scene

```
eval/output/train/{scene_id}/
├── signals.npz              # sensors (360,6), forces (360,6), actions (360,3), rewards (360,)
├── metrics.json              # Best seed summary: DTW, reward, action stats
├── all_seeds.json            # Per-seed breakdown: reward, r_cd, r_cl, r_sim, sim_raw, dt_sec
├── vorticity_controlled.png  # Final frame after full DRL rollout
├── vorticity_target.png      # Target state (disturbance only / target cylinder only)
└── vorticity_zero.png        # Zero-action baseline (no control)
```

### 3.4 Config Switching Delay

When consecutive scenes use different LBM config files (different ν → different kernel compilation), a 120-second delay is inserted to allow the previous GPU context to fully release before the new config triggers PTX recompilation. Scenes sharing the same config file run back-to-back without delay.

---

## 4. Baseline Collection (`collect_baselines.py`)

Collects `q_in.npz` (background flow) and `q_blk.npz` (zero-action pinball) for each scene.
Baselines are needed by OID and CCD analysis pipelines.

```bash
python collect_baselines.py --device 2            # all scenes
python collect_baselines.py --scene kar_re100_sc  # single scene
```

## 5. SR Bridge (`bridge_to_sr.py`)

Converts V5 eval output to the format expected by `SR_analysis/`:
- `calibration.json` → `norm.json` (FORCE_SCALE, SENS_SCALE, sens_deviation=0)
- `signals.npz` → `controlled.npz` (sensors, forces, actions, rewards)
- `target.npy` → `target.npz`

## 6. GPU Scheduling (`run_all_with_fields.sh`)

Single GPU (2). Config switch waits 120s to avoid PTX compilation conflicts.

```
GPU2: [kar_re100_sc(5 seeds)] → [vardist_sc+tr] → [ill_*_sc]
      --120s--> [kar_re60_sc] --120s--> [kar_re200_sc] --120s--> [kar_re400_sc]
      --120s--> [cross-re transfers]
```

### Run Commands

```bash
# Full pipeline (infer + baselines + bridge):
bash run_all_with_fields.sh

# Single scene:
bash run_all_with_fields.sh --scene kar_re100_sc --gpu 2
python infer_train.py --scene kar_re100_sc --device-id 2
python viz_flow.py
```

---

## 6. Eval Results

### 6.1 Train Pipeline — Completed Cases

| Scene | Best Seed | DTW sim_raw | Reward | r_cd | r_cl | r_sim |
|-------|:---------:|:-----------:|:------:|:----:|:----:|:-----:|
| kar_re100_sc | 45 | **0.926** | 0.941 | 0.984 | 0.990 | 0.872 |
| kar_d075_sc | 44 | **0.911** | 0.949 | 0.966 | 0.974 | 0.919 |
| kar_re60_tr | 43 | 0.187 | 0.312 | 0.659 | 0.334 | 0.091 |
| kar_re200_tr | 43 | 0.506 | 0.367 | 0.669 | 0.212 | 0.278 |
| kar_re400_tr | 43 | 0.428 | 0.399 | 0.737 | 0.400 | 0.176 |

### 6.2 Train Pipeline — Re100 5-Seed Breakdown

| Seed | DTW sim_raw | r_cd | r_cl | r_sim |
|:----:|:-----------:|:----:|:----:|:-----:|
| 41 | 0.634 | 0.307 | 0.241 | 0.388 |
| 42 | 0.893 | 0.959 | 0.762 | 0.788 |
| 43 | 0.850 | 0.908 | 0.469 | 0.629 |
| 44 | 0.523 | 0.282 | 0.177 | 0.322 |
| 45 | **0.926** | 0.984 | 0.990 | 0.872 |

Mean DTW across seeds: 0.765 ± 0.172. Best-worst spread: 0.403.

### 6.3 DTW Formula Note

**Important**: V5 uses a different DTW formula than Legacy pipelines. See `recompute_unified_dtw.py` for fair comparison using the same formula.

---

## 7. Visualization & Reporting Tools

### 7.1 `viz_signals.py`

Per-scene diagnostic plots (4×2 subplot grid): sensors, forces, actions, reward, FFT, phase portrait.

### 7.2 `viz_flow.py`

Cross-scene vorticity comparison panels with unified [-0.003, 0.003] colormap.

### 7.3 `generate_report.py`

Master comparison report with DTW bar chart.

---

## 8. Known Limitations

- **Illusion training bug**: All target diameters were accidentally set to 1L. Transfer results invalid.
- **VecNormalize compat**: `PPO.load()` may fail with cloudpickle on Python 3.10+. Skeleton injection fallback handles this.
- **GPU cleanup**: Between scenes, `pkill -f pycuda; sleep 180` may be needed.

---

## 9. Companion Documents

| File | Content |
|------|---------|
| `eval/README.md` | Quick start + self-review checklist |
| `eval/scene_manifest.py` | TRAIN_SCENES single source of truth |
| `eval/infer_train.py` | V5 train model inference |
| `eval/recompute_unified_dtw.py` | Fair DTW comparison (unified formula) |
| `train/TRAIN_PIPELINE.md` | Training pipeline documentation |
| `reproduce/REPRODUCE_KNOWLEDGE.md` | Reproduce module (separate) |
| `../SR_analysis/README.md` | SR analysis pipeline overview |

---

*Document last updated: 2026-07-13.*