CelerisLab/docs/audit/config_diagnosis.md

Config method diagnosis: FP16S and EsoPull analysis

Resolution status: EsoPull — FIXED (v0.5.1); FP16S — inherent Bouzidi incompatibility (not fixable without architecture change).

---

1. Overview

Method	Original Failure	Root Cause	Status
EsoPull	NaN at D=20; CD ~500-700% error at D=30	3 bugs: (1) InitEsoPull wrote physical layout not EsoPull layout; (2) wall BCs used raw load_ddf instead of EsoPull semantics; (3) PrepareEsoPullCurvedKernel used post-streaming read for pre-streaming force calc	FIXED v0.5.1 — EsoPull now bit-identical to double-buffer for Kan99b K2
FP16S	30-40% CD error even with ddf_shifting	Bouzidi per-direction DDF reads accumulate FP16S quantization noise without direction-averaging cancellation	Inherent — requires architecture change (TYPE_E equilibrium boundaries)

---

2. FP16S analysis (unchanged from v0.5.0)

2.1 Root cause

Bouzidi curved boundary reads INDIVIDUAL DDF values per direction per cut-link, with no averaging across directions. At omega~1.9, Re~100, NEQ magnitudes occupy only 1-3 FP16S quantization steps (~3e-5/step). The interpolation ratio (1/q, up to 2x) and moving-wall correction amplify the noise.

FluidX3D avoids this because update_force_field() loads ALL 9 (or 19) directions from neighbor cells into FP32, where averaging cancels quantization noise.

2.2 Known working combinations (FP16S)

ddf_shifting	Inlet	D	CD error	Notes
True	zou_he_local	30	~32%	Best FP16S result
True	zou_he_local	60	TBD	Larger D may fare better
False	any	any	>>100%	Shifting essential for FP16S

2.3 Recommendations

FP16S is only usable when D is large (D >= 60) OR when force accuracy is secondary. For production Kan99b validation, use FP32. A proper fix would require a FluidX3D-style TYPE_E equilibrium boundary with full-cell momentum summation.

---

3. EsoPull root causes and fixes (v0.5.1)

3.1 Bug #1 (critical): InitEsoPull physical-layout write

File: lbm/kernels/step/init_flow.cu

InitEsoPull called write_equilibrium which wrote fi[k, i] = f_eq[i] (physical layout). But load_f_esopull(t=0) expects EsoPull backing layout where paired directions are scattered across neighbor cells. At step 0, every direction was read from the wrong slot, producing garbage DDF for collision.

Fix: Added write_equilibrium_esopull and write_rest_equilibrium_esopull that scatter equilibrium values directly into the EsoPull t=0 backing layout: odd directions to neighbor forward slots, even directions to local reverse slots. Also corrected an odd/even slot swap bug discovered during testing.

3.2 Bug #2 (moderate): Wall boundary functions use raw load_ddf

File: lbm/kernels/boundary/bounce_back.cuh

apply_wall_freeslip_d2q9_y_pull and apply_wall_bb_d2q9_y_pull used load_ddf(fi, index_f(k, slot)) to read opposite-direction values. In EsoPull mode, fi[k, slot] is a backing slot, not a physical direction, so the wrong values were used for half-way bounce-back.

Fix: Extended function signatures with t and j parameters. When t > 0, use load_physical_dir_pre_streaming to resolve the correct pre-streaming value in EsoPull layout. When t == 0, fall back to raw load_ddf (correct for double-buffer).

3.3 Bug #3 (critical for CD): Pre-streaming vs post-streaming semantics error

File: lbm/kernels/step/aux_kernels.cu, lbm/kernels/streaming/esopull_semantic_helpers.cuh

PrepareEsoPullCurvedKernel used load_physical_dir_esopull() to read DDF values for Bouzidi force computation. This helper returns POST-streaming values (the distribution that arrived at a cell after streaming). However, Bouzidi force calculation requires PRE-streaming values — the post-collision distribution that sat at the fluid cell in each direction BEFORE streaming (matching what double-buffer CurvedBoundaryKernel reads from fi_in[k_f, dir]).

For odd directions (E, N, NE, SE), store_f_esopull writes the pre-streaming value to a NEIGHBOR cell. load_physical_dir_esopull then reads the WRONG cell's value for ~50% of directions, producing a systematic ~8x force error.

Fix: Added load_physical_dir_pre_streaming in esopull_semantic_helpers.cuh. This resolves the pre-streaming value by:

• Using tp = t ^ 1 (previous step's parity) for direction/slot resolution (because store_f_esopull at step t-1 wrote the values)
• Reading odd-direction values from the neighbor cell (where store_f_esopull scattered them)
• Reading even-direction values from the local cell
• Falling back to load_physical_dir_esopull at t=0 (init layout, not store layout)

All three DDF reads in PrepareEsoPullCurvedKernel (f_toward, f_toward_ff, f_opp_same) were updated to use the pre-streaming variant.

3.4 Results

Run	Mode	D	St	CD	CD error pre-fix	CD error post-fix
MR2	double-buffer (baseline)	20	0.16913	1.1367	—	2.96%
MR3	esopull	20	0.16508	1.1009	~716%	0.28%
MR1	double-buffer (baseline)	30	0.16992	1.1455	—	3.76%
B1	esopull	30	0.16647	1.1129	~539%	0.81%

EsoPull is now numerically equivalent to double-buffer for Kan99b K2 (MRT, D=20/30, rotating cylinder with curved boundaries).

3.5 What the original diagnosis got wrong

1. Section 3.3 (init mismatch): Correctly identified but underestimated — the suggested "skip first step" fix would NOT work because EsoPullStep(t=0) itself also operates on wrong semantics from init layout.

2. Section 3.5 (inlet secondary issue): Incorrect. The EsoPull inlet/outlet functions already used load_f_esopull() (not raw load_ddf) for neighbor DDF reads since they were added. This was never a contributing factor to the CD error.

3. Section 3.6 (verdict): Correctly identified init as root cause of NaN, but missed the pre-streaming bug which was the true cause of CD ~500-700% error. The init bug alone could not explain such a large persistent error.

---

4. Files modified in v0.5.1

File	Change
lbm/kernels/step/init_flow.cu	Added write_equilibrium_esopull / write_rest_equilibrium_esopull; modified InitEsoPull to use them
lbm/kernels/streaming/esopull_semantic_helpers.cuh	Added load_physical_dir_pre_streaming; updated usage comments
lbm/kernels/step/aux_kernels.cu	Changed 3 load_physical_dir_esopull calls to load_physical_dir_pre_streaming in PrepareEsoPullCurvedKernel
lbm/kernels/boundary/bounce_back.cuh	Extended wall function signatures; added EsoPull pre-streaming read path; updated module docstring
lbm/kernels/step/one_step_esopull.cu	Updated wall function calls to pass j and t
tests/validation/run_kan99b_rotating_cylinder.py	Added --streaming CLI parameter