# Server deployment

This guide uses only scripts and configs currently present in the repository. The release reproduces the training workflow and distributes compact best-policy bundles for retained active runs.

## Install

```bash
git clone --recurse-submodules <repo-url> DynamisLab
cd DynamisLab
conda create -n pycuda_3_10 python=3.10 -y
conda activate pycuda_3_10
pip install pycuda
pip install -e CelerisLab
pip install -e .
python -c "from CelerisLab import Simulation; print('CelerisLab import OK')"
```

Requirements: Linux, NVIDIA GPU/driver and a compatible CUDA toolkit. Run commands from `src/drl_pinball/train` unless stated otherwise.

## Canonical cases and configs

Active IDs have no `_sc` suffix:

- `kar_re100`, `kar_d075`, `kar_d15`, `kar_d2`, and all `ill_*` use `../../../configs/config_lbm_karman_2000x600.json`.
- `kar_re60`, `kar_re200`, `kar_re400` use the matching `..._re60.json`, `..._re200.json`, `..._re400.json`.
- `_tr` appears only in archived transfer experiments.

## Calibrate a new artifact set

Generate a matched calibrated target set on the training server before training. To create the Karman baseline set used by the active launcher:

```bash
conda run --no-capture-output -n pycuda_3_10 python -u calibrate.py \
  --case kar_re100 --device-id 0 --si 800 \
  --config ../../../configs/config_lbm_karman_2000x600.json \
  --out-dir calibrations/kar_re100
```

For Illusion, remember that the historical `--target-diam` option is passed as a radius ratio: `1.0` produces radius `1.0*L0=20` lattice units, hence physical diameter 40.

```bash
conda run --no-capture-output -n pycuda_3_10 python -u calibrate.py \
  --case ill_1L --scene illusion --target-diam 1.0 \
  --device-id 0 --si 1200 \
  --config ../../../configs/config_lbm_karman_2000x600.json \
  --out-dir calibrations/ill_1L
```

Keep each generated `calibration.json` with its `target.npy` and, for Illusion, `target_harmonics.json`. Targets are calibrated artifacts, not interchangeable input samples.

## Train

Real launchers in `scripts/` run serially on one GPU:

```bash
bash scripts/train_baseline.sh --gpu 0 --seeds 41,42,43 --episodes 500
bash scripts/crossre_scratch.sh --gpu 1 --only re200 --episodes 500
bash scripts/vardist_scratch.sh --gpu 2 --only d15 --episodes 500
bash scripts/train_illusion.sh --gpu 3 --only 1L --episodes 500
```

There is no active `launch_multi.sh`. For multiple GPUs, start one existing launcher per GPU and give each a disjoint case/seed selection. Stagger heavy CelerisLab initialization if the shared server or CUDA compilation cache requires it; no universal seven-minute delay is guaranteed by this repository.

Explicit single-run example:

```bash
conda run --no-capture-output -n pycuda_3_10 python -u train_karman.py \
  --case-name kar_re100 --device-id 0 --seed 42 \
  --config ../../../configs/config_lbm_karman_2000x600.json \
  --calibration calibrations/kar_re100/calibration.json \
  --total-episodes 500 --symmetry-prob 0
```

Resume with both checkpoint members present:

```bash
bash scripts/resume.sh --case kar_re200 --seed 43 --resume 460 --episodes 500 --gpu 0
```

Resume continues policy and `VecNormalize`; it is not exact deterministic replay. Seeds distinguish runs but do not guarantee replay.

## Monitor and retain

```bash
tail -f output/kar_re100_seed42/train.log
tensorboard --logdir output/kar_re100_seed42/tb --port 6006 --bind_all
nvidia-smi
```

Expected run layout:

```text
output/kar_re100_seed42/
├── calibration.json
├── meta.json
├── train.log
├── tb/
├── best_vecnormalize.pkl
├── final_vecnormalize.pkl
├── vec_normalize.pkl        # legacy final-state alias
└── models/
    ├── best_model.zip
    ├── final_model.zip
    ├── epNNNN_model.zip
    └── epNNNN_vecnormalize.pkl
```

Treat `models/best_model.zip` and root `best_vecnormalize.pkl` as one artifact. Only the allowlisted best-policy inference bundle is released from each active `output/` run. Intermediate/final checkpoints, logs, and TensorBoard streams remain local. Retained repository results are historical writing basis rather than evidence of complete post-fix retraining.
