CelerisLab/docs/validation/uniform_srt_diagnostic.md

First-failure uniform SRT diagnostic (prepared, not executed)

Purpose: locate the first physical failure in the unchanged uniform D80 SRT control that was healthy at executed step 160 and nonfinite at the next observation, executed step 320. The original failure's summary steps=160 is the last accepted observation, not the last executed step. This diagnostic does not select different numerical parameters, run an AMR case or perform a new benchmark campaign.

Exact source and configuration

The default source is the containing repository (override with --source); the exact pinned production/runtime source is commit 9805383f9c50d867e8555471baf5dbda8333a652. The manifest pins production files and the helper/runner dependencies directly against the original failure's source_before.json. All pinned contents match. Import guards require CelerisLab and the runner from the selected pinned checkout.

The script calls the original run_refined_dfg.config(helper,80) factory and unchanged Simulation(...).initialize(). Thus it retains D80, U=.01, ν=.008, τ=.524, ω≈1.9084, shifted FP32, SRT, double buffer, parabolic local Zou–He inlet, all-nine pressure-NEQ damping .5, backflow clamp disabled, identical cylinder geometry and initialization. No clipping, equilibrium rewrite or solver patch is added.

The inlet overload in this source may differ by FP32 rounding from older d79 code even where the algebra matches; that is not changed here and is not established as the cause of failure. MRT's outlet branch also differs. Neither prior MRT success nor this diagnostic establishes an SRT/outlet/body attribution.

Observation and stop contract

• At most 320 actual solver steps, starting from the original fresh initialization. Stop at the first physical health failure (or wall deadline).
• Read the production macroscopic path, its decoded populations, and force after every completed step. Preserve the actual executed count separately from the last observed count.
• Independently report nonfinite physical populations, nonfinite physical macros and nonfinite reaction. Apply the original density [.5,1.5] and Mach≤.2 checks to physical FLUID nodes. A new population-only check may stop earlier than the original macro-only health check; the report explicitly distinguishes it.
• Per-step Simulation.run(1) resets/reads the observation buffer for a one-step force window. This changes observation cadence and force accumulation windows, not configuration or fluid updates. A cumulative-force-only overflow from the original 100/60-step batching would not be reproduced identically; that limitation is explicit. Original macro warnings already show nonfinite populations/macros, so this instrument prioritizes locating the physical failure.
• Ghost/inactive/body-solid populations are reported in separate categories; their NaNs do not independently trigger physical failure. Category masks are disjoint and complete. Boundary subtypes are decoded as the nibble, not mistaken for independent bit flags.
• Physical region labels are body-near (within radius+3 cells), outlet/inlet-near (three fluid columns), wall-near (three fluid rows), and remaining bulk. These locate symptoms, not causal mechanisms.
• Initial, preceding and firstbad/final compact reports retain category counts, finite global extrema and positions, the first 16 offending nodes in row-major order, their bad q indices/all populations, and up to four 5×5 population/flag neighborhoods. “First” identifies earliest sampled step; within that step the list is row-major, not an asserted temporal order among GPU threads.

Bounded evidence

Only the immediately preceding and firstbad/final active-buffer states are retained. states.npz contains their exact raw shifted FP32 values, production-decoded FP32 populations and step numbers, plus flags. This is ~81 MiB at the exact D80 shape, with a 128 MiB artifact limit enforced. There are no saved field trajectories. Initial conditions are reproducible from the source/config; only their diagnostic summary is retained.

These are diagnostic snapshots, not full restart checkpoints (spare buffers and observation/control state are not a restart bundle). diagnostics.json stores nonfinite numbers as explicit NaN, +Inf, -Inf strings and is strict finite JSON; NPZ preserves actual IEEE values. A force-only failure can have no offending fluid node. A clean cap means no sampled failure through 320, not benchmark acceptance.

CPU validation

python tests/validation/uniform_srt_diagnostic/test_diagnostic.py
python tests/validation/uniform_srt_diagnostic/diagnose_uniform_srt.py --output /tmp/new-srt-preflight

Seven CPU-only tests pass: source/config, complete category partition, inactive nonfinite exclusion, independent population/macro/force reasons, health thresholds, bounded two-state artifact writing, and deadline/source guards. py_compile and the real-source default preflight pass. No CUDA import is required by preflight/tests and no GPU was executed here.

Root-only admitted execution

After independent review and root GPU admission:

CUDA_VISIBLE_DEVICES=<admitted-device> timeout --signal=TERM --kill-after=30s <hard-budget>s python tests/validation/uniform_srt_diagnostic/diagnose_uniform_srt.py --source <exact-9805383-checkout> --execute --gpu-admitted --wall-seconds <soft-budget> --output <new-output-directory>

The external watchdog is required for a blocked CUDA call; cooperative wall checks cannot interrupt one. Budget should allow per-step downloads and ~81 MiB evidence writing; this diagnostic is not a performance measurement. The step ceiling cannot be increased by a CLI flag. READY is emitted after the first completed step and all observation paths. First physical failure is expected diagnostic evidence and returns exit 2; step cap returns 0. preflight.json records the pre-execution configuration, not a completion receipt.

Original failure files, production source, original pilot runner and gates are untouched. This test-only handoff is based on exact commit 9805383; added diagnostic files are not part of the historical failure source manifest.

Historical reporting undercount (not changed here)

The failed pilot executed to step 320 before its second observation failed. Its summary steps=160 reflects begin, which advances only after an accepted observation. This is a reporting undercount, not evidence that the second batch did not execute. The original failure package and runner/gates are preserved unchanged. This diagnostic records both executed_steps and observation_step, and labels previous-state steps explicitly. Fixing the historical runner's reporting is a separate change.