DynamisLab/docs/ccd_correction_field_report.md

CCD Analysis Report: Correction-Field Decomposition of Illusion Control

What this report is: A self-contained summary of the CCD (Canonical Correlation Decomposition) analysis pipeline applied to the fluidic pinball illusion control problem. It assumes no prior knowledge of CCD or the project details — everything is explained from the ground up.

What this report is NOT: A complete physics investigation. It is a progress report documenting what analysis was done, what was found, what it means, and where the open questions are.

---

1. The Problem in Plain Language

1.1 The physical system

Imagine three identical cylinders arranged in a triangle pointing upstream, placed in a channel with water flowing past them. Each cylinder can spin independently at a controlled speed. When the cylinders do NOT spin, the flow behind them forms a chaotic oscillating wake (a "von Karman vortex street").

The DRL controller can rotate the three cylinders at different speeds to change this wake. The goal of illusion control is: "make the flow field downstream of the three cylinders look like the flow field that would be produced by a single cylinder of a different size."

We test three target sizes: a cylinder of diameter 0.75 (smaller than the pinball cylinders), 1.0 (same size), and 1.5 (larger).

1.2 What the controller sees and does

• Observations (input to controller): The forces on each cylinder (drag and lift) + the flow velocity measured at 3 points downstream
• Actions (output of controller): 3 rotation speeds (one per cylinder), updated every 800 simulation timesteps
• Reward (what the controller is trained to maximise): How closely the downstream sensors match the target cylinder's signal, plus how closely the total forces on the pinball match the target cylinder's forces

1.3 The key insight that changed everything

The naive approach is to ask: "does the controlled flow look like the target flow?" But this is not what the controller does. The controller works by modifying the existing pinball wake. A better question is: "what extra change does the controller add on top of the uncontrolled pinball wake, and does that change look like the change needed to transform the pinball wake into the target wake?"

This leads to the correction-field framework:

Symbol	Meaning	How to think of it
q_in	Clean channel flow (no pinball)	The baseline
q_blk	Pinball, no rotation	What the pinball does to the flow by its mere presence
q_ctl	Pinball with DRL control	The controlled flow
q_tar	The target cylinder alone	The flow we wish we had
dq_blk = q_blk - q_in	Blockage field: how the pinball disturbs the channel	Pinball's "mess"
dq_ctl = q_ctl - q_blk	Correction field: what control adds on top of the pinball	Controller's "fix"
dq_tar = q_tar - q_blk	Target correction: what change would turn pinball into target	The required "fix"

The main question becomes: does dq_ctl (the actual fix) look like dq_tar (the required fix)?

---

2. What is CCD? (For the Non-Expert)

2.1 The core idea

Proper Orthogonal Decomposition (POD) finds the flow patterns that contain the most energy. It answers: "what are the dominant oscillating structures in this flow?"

Canonical Correlation Decomposition (CCD) finds the flow patterns that are most correlated with a specific quantity you care about (an "observable"). It answers: "what flow structures most determine the force on the cylinders?" or "what flow structures most determine the downstream sensor reading?"

The difference is crucial. Imagine a jet engine: the most energetic flow structures might be in the turbulent exhaust, but the structures that generate noise might be much weaker and completely different. POD would miss them because it ranks by energy, not by relevance to noise.

2.2 How CCD works (simplified)

1. Take snapshots: Record 96 velocity field snapshots of dq_ctl at evenly spaced times over 4 vortex shedding cycles
2. Build a reference basis: Use POD to find the main energy-containing structures in the TARGET correction field dq_tar — this gives us a coordinate system defined by what the target looks like
3. Project into this basis: Express the CONTROLLED correction field dq_ctl in terms of the target's structures
4. Pick an observable: Choose something we care about — the total lift force (SigmaFy), the cylinder rotation speeds (action), or the future sensor error (signature)
5. Find the correlated patterns: CCD finds the directions in flow-structure-space that best predict/correlate with the observable
6. Measure compactness (m80): How many such directions do we need to capture 80% of the correlation? m80=1 means a single flow pattern explains most of the observable. m80=4 means we need more patterns.
7. Measure overlap (O_k): Do two cases (e.g., target vs illusion) use the same flow patterns to generate the observable? O=1 means identical, O=0 means completely different.

2.3 Validation: how do we know CCD is meaningful?

We use Leave-One-Cycle-Out (LOCO) cross-validation:

• We have 4 shedding cycles of data
• Train CCD on 3 cycles, predict the observable on the held-out 1 cycle
• Compute R2 (how well the prediction matches reality)
• Repeat for each cycle as the held-out set
• If R2 > 0.4-0.5, the CCD patterns are stable and predictive

---

3. Data Quality and Preprocessing

Before any analysis, the raw flow fields must be phase-aligned — each snapshot corresponds to the same phase in the vortex shedding cycle across all cases. This is done by detecting the dominant shedding frequency, finding cycle boundaries, and extracting 24 evenly-spaced snapshots per cycle for 4 cycles = 96 snapshots total.

All cases pass quality gates:

• Strict gate: cycle-to-cycle period variation (CV_T) < 10%
• Relaxed gate: CV_T < 12%

Case	Gate	Points/cycle	Interpolation factor	Strouhal
target_cylinder 0.75L	strict	30.0	0.80	0.128
target_cylinder 1.0L	strict	24.8	0.97	0.133
target_cylinder 1.5L	strict	25.8	0.93	0.143
illusion 0.75L	strict	29.9	0.80	—
illusion 1.0L	strict	24.5	0.98	—
illusion 1.5L	strict	24.2	0.99	—
pinball (uncontrolled)	strict	21.4	1.12	0.113

The interpolation factor (rho) indicates how close each case is to having an integer number of snapshots per cycle. rho=1 is perfect; all cases have rho <= 1.12, meaning almost no interpolation artifacts.

---

4. The Five Analysis Lines

We answer five questions, each requiring a different observable for CCD:

Line	Observable	Question	Symbol
Force line (primary)	Total lift force	Which correction structures most determine the lift?	SigmaFy
Force line (secondary)	Total drag force	Which correction structures most determine the drag?	SigmaFx
Action line	3 cylinder rotation speeds	Which correction structures does the controller directly modulate?	[Omega1, Omega2, Omega3]
Signature line	Future sensor error	Which correction structures most determine whether downstream sensors will match the target?	e_s(t+tau) = s_ctl(t+tau) - s_tar(t+tau)

The signature line has an extra dimension: time delay tau. We try tau=0 (instantaneous sensor error) and tau=tau_c (the time it takes for flow structures to convect from the pinball to the sensors, about 3-4 simulation steps).

---

5. Master Results

5.1 One-sentence summary per diameter

Diameter	Summary
1.0L	The controller's correction direction is nearly identical to the target's required correction (O=0.913), and a single flow pattern captures 80% of the force-relevant correction. This is the "natural scale" case.
0.75L	The controller's correction only partially aligns with the target's required correction (O=0.564). The force and sensor-error structures are strongly separated in space at any instant, but converge after convective propagation.
1.5L	This is not a failure but a special mechanism. The controller achieves 94% sensor similarity using a qualitatively different strategy: the cylinder rotation commands have very weak correlation with the target-basis structures (action sigma1 = 0.28 vs 1.1-1.4), the correction energy is concentrated near the cylinders, and the shedding phase drifts over time.

5.2 The master table

All values at r=6 (6 POD modes retained for the reference basis), unless noted.

Metric	0.75L	1.0L	1.5L
O(dqctl, dqtar) — how similar are the corrections? (1=identical, 0=orthogonal)	0.564	0.913	0.667
force_fy m80 — how many patterns needed for 80% of force? (lower = more concentrated)	2	1 (at r=8/10)	2
action m80 — how many patterns for 80% of action correlation?	2	3	3
action sigma1 — strength of action correlation (higher = actions more tied to target-basis structures)	1.39	1.13	0.28
signature m80 (zero delay)	3	3	2
signature m80 (convective delay)	2	3	2
O(force, sig) at zero lag (1=force and sensor error use same structures)	0.413	0.551	—
O(force, sig) at convective delay	0.806	0.768	—
Phase drift (cycle-to-cycle period variation)	low	low	high
Body-wake KE / sensor-zone KE (higher = more correction energy near cylinders)	0.73	1.17	2.58

5.3 Validation: can we trust these numbers?

LOCO cross-validation R2 (m80 reconstruction, r=6):

Observable	0.75L	1.0L	Threshold	Verdict
force_fy (lift)	0.65 +- 0.08	0.64 +- 0.02	> 0.4	PASS
force_fx (drag)	0.38 +- 0.23	0.43 +- 0.11	> 0.4	WARNING
signature (zero lag)	0.50 +- 0.09	0.49 +- 0.04	> 0.4	PASS
signature (convective delay)	0.51 +- 0.09	0.53 +- 0.03	> 0.4	PASS

The standard deviation across the 4 folds is small (0.02-0.09), meaning the patterns are stable across different data subsets. The drag channel (force_fx) is unreliable for detailed claims — its R2 is borderline and its variance is high. All other channels pass validation.

---

6. The Force vs Signature Separation: The Most Important Finding

6.1 Why this matters

One of the fundamental questions in flow control is: "are the flow structures that generate forces the same as the flow structures that determine what a downstream sensor sees?" If they are the SAME, then controlling the force automatically controls the sensor signal. If they are DIFFERENT, then the controller must manage two separate sets of structures.

Our finding: they are SEPARATED at any instant, but CONVERGE after the flow has time to convect downstream.

Evidence — full-field CCD:

tau (convective delay in steps)	0.75L O(force, sig)	1.0L O(force, sig)
0 (instantaneous)	0.413 (separated)	0.551 (partial)
~3-4 (convective delay)	0.806 (shared)	0.768 (shared)

At tau=0, the force-relevant and sensor-error-relevant structures share only 41-55% of their modal directions. After the flow convects downstream (tau=3-4), they share 77-81%. This makes physical sense: at any snapshot, the forces are determined by what is happening near the cylinders, while the sensor error is determined by what is further downstream. But after the near-body structures have had time to propagate downstream, they become the same thing.

6.2 Where does this separation happen spatially?

We divided the flow field into three zones and ran CCD separately in each:

Zone	x-range (pixels)	What's there
near_body	350-500	Around the cylinders (located at x=380-406)
body_wake	500-700	Just downstream of cylinders
sensor_zone	580-650	Where the velocity sensors measure the flow

0.75L — the separation is dramatic:

Zone	O(force, sig) at tau=0	O(force, sig) at tau=tau_c
near_body	0.262 (separated)	0.827 (shared)
body_wake	0.269 (separated)	0.917 (shared)
sensor_zone	0.010 (NEARLY ORTHOGONAL)	0.722 (shared)

In the sensor zone at zero lag, the force and signature structures are effectively perpendicular (O=0.01). This is the cleanest possible demonstration that force-relevant and sensor-error-relevant structures live in different spatial regions at any given instant. After the convective delay, the body_wake shows the strongest coupling (O=0.917), meaning the near-wake structures jointly determine future forces AND future sensor readings.

1.0L — more uniform, less separation:

Zone	O(force, sig) at tau=0	O(force, sig) at tau=tau_c
near_body	0.596	0.596
body_wake	0.509	0.483
sensor_zone	0.594	0.730

At the natural scale (1.0L), force and signature are more intrinsically linked across all zones. There is no zone with near-zero overlap. This makes sense: when the target shedding frequency matches the pinball's natural frequency, the same structures that produce forces are also those that the downstream sensors detect.

---

7. 1.5L Special Mechanism

The 1.5L case is not a failure (94.2% sensor similarity) but it operates differently:

1. Action correlation is dramatically weaker: sigma1 = 0.28 (vs 1.13-1.39 for other diameters). In the target's structural coordinate system, the cylinder rotation commands have very little explanatory power.

2. Phase drift: The shedding period varies significantly over time (CV_T across windows is high), unlike the stable periodic shedding of 0.75L and 1.0L.

3. Correction energy concentrated near cylinders: The KE ratio (body_wake / sensor_zone) = 2.58 (vs 0.73 for 0.75L, 1.17 for 1.0L). The controller is applying larger corrections near the cylinders, not just modifying the downstream wake.

4. Signature coupling is stronger than force coupling: The signature-line sigma1 (1.29) is greater than the force-line sigma1 (0.91). The correction field is more tightly tied to future sensor error than to instantaneous force.

---

8. Comparison: Correction Field vs Raw Field

Why go to the trouble of computing dq_ctl = q_ctl - q_blk instead of just working with q_ctl directly?

Measure	Raw field (q_ctl)	Correction field (dq_ctl)	What it tells us
1.0L O(target, illusion)	0.919	0.913	Similar — 1.0L is clean either way
0.75L O(target, illusion)	0.673	0.564	Raw field was contaminated — ~16% of the apparent overlap was just baseline similarity
1.0L force m80	2	1	Correction field is more concentrated — the controller's ADDED structures are simpler than the full flow
LOCO R2 force_fy	0.66-0.71	0.64-0.65	Comparable — correction field doesn't degrade predictability

Conclusion: Correction-field is the superior primary analysis object. The raw field can still be useful for historical comparison, but all mechanism claims should be based on correction-field analysis.

---

9. What We Learned About Steady Cloak

The steady cloak case (open-loop constant-speed rotation of the rear cylinders) was also analysed. The result: it does not work well. The RMS fluctuation suppression is essentially 0%, and the residual after cancellation is 81% of the original blockage. The downstream sensor region does better (13% residual) but that is mostly because the wake naturally recovers with distance.

This case is not suitable as a primary mechanism demonstration. A closed-loop steady cloak (using DRL) would be needed for meaningful analysis.

---

10. Limitations

1. POD-reduced CCD: All CCD results are constrained to the subspace spanned by the first 6-10 POD modes of the target correction field. If the controller uses structures that have very low energy in the target's natural basis, they will be truncated and invisible to CCD. For a "true" full-field CCD, significantly more data would be needed.

2. The drag channel is unreliable: force_fx (drag) fails LOCO validation (R2 ~0.4 with high variance). This is because the force reward in the DRL training matches drag statistics (mean, variance) but not the instantaneous waveform. Drag CCD results should only be used for O_k trend comparisons, not for mechanism claims.

3. 1.5L force-signature overlap not computed: The force-vs-signature comparison could not be run for 1.5L because of technical issues with the cross-correlation computation. This is a gap that should be filled.

4. Karman cloak not analysed: The data for Karman cloak (vortex street incoming) has been collected but the analysis was deferred. The physical question is different (distortion compensation vs target generation) and the full pipeline is ready for when this becomes a priority.

---

11. Available Figures and How to Read Them

Figure 1: Force sanity check — sanity_force_{diam}L.png

Three files: one per diameter.

What it shows: The raw force time series comparison between the target cylinder (red) and the controlled pinball illusion (blue). Four panels per figure:

• Top-left: Total drag force Fx over time
• Top-right: Total lift force Fy over time
• Bottom-left: Fx scatter plot (target vs illusion), with correlation r annotated
• Bottom-right: Fy scatter plot, with correlation r annotated

How to read it: The diagonal dashed line in the scatter plots indicates perfect tracking. For 1.0L, the Fy scatter (bottom-right) shows points clustering near the diagonal with r=0.82 — the controller tracks the lift waveform. For 1.5L, the Fy scatter shows r=-0.30 — the lift is negatively correlated, meaning the controller is doing something fundamentally different. The Fx scatter for all diameters shows r near zero — the controller matches mean drag but not the drag waveform.

Look at: The Fy scatter correlation coefficients. 0.75L: r=0.38 (weak positive), 1.0L: r=0.82 (strong positive), 1.5L: r=-0.30 (negative!).

---

Figure 2: Correction field comparison — corr_illusion_{diam}L_ctl_vs_tar.png

Three files: one per diameter.

What it shows: A 2x2 panel comparing the controller's correction (dq_ctl, left column) with the target's required correction (dq_tar, right column).

• Top row: Mean streamwise velocity (ux) of the correction field
• Bottom row: RMS magnitude of the correction field

How to read it: Look at the spatial patterns in the mean ux panels (top row). If the left and right panels look similar in structure (red/blue pattern), the controller is adding a correction that resembles what the target requires. For 1.0L, they look nearly identical. For 0.75L, there are similarities but also clear differences in the wake region. The RMS panels (bottom row) show where the fluctuations are — bright regions indicate high unsteadiness in the correction.

Look at: How similar the top-left and top-right panels are. The more similar, the more the controller's correction "knows" what the target needs.

---

Figure 3: Correction field maps — corr_illusion_{diam}L_dq_ctl_(control_correction).png

Three files: one per diameter, plus similar files for dq_blk and vorticity.

What it shows: Three panels of the dq_ctl correction field:

• Left: Mean streamwise velocity (ux)
• Centre: Mean cross-stream velocity (uy)
• Right: RMS magnitude

How to read it: Red in the ux panel means the controller is ACCELERATING the flow at that point; blue means DECELERATING. The RMS panel shows where the control is most unsteady. The pinball cylinder positions are at approximately x=380-406 (visible as blank regions).

Look at: The ux panel — where does the controller add positive (red) vs negative (blue) momentum? For 0.75L and 1.0L, there is a strong dipole pair in the wake. For 1.5L, the pattern is shifted and the amplitude is larger.

---

Figure 4: CCD mode 1 — ccd_mode1_fy_{diam}L_{target,illusion}.png

Four files: 2 diameters x 2 cases (target and illusion).

What it shows: The first (most important) CCD mode for the force-fy line (lift), expressed as a velocity field. Left panel = ux component, right panel = uy component. Red = positive, blue = negative.

How to read it: This is the single flow pattern that is most correlated with the lift force. If the target and illusion panels look similar, it means the controller is using the same kind of flow pattern to generate lift as the target cylinder naturally uses.

Look at: Compare the 1.0L target mode with the 1.0L illusion mode — they should look very similar (consistent with O=0.913). Compare 0.75L target with 0.75L illusion — more differences expected (O=0.564).

---

Figure 5: POD phase portraits — pod_phase_portraits_target_basis.png

One file, three panels (0.75L, 1.0L, 1.5L).

What it shows: The scatter of the first two POD coefficients (a1, a2) in the target-only basis. Red dots = target cylinder, blue dots = illusion (controlled), green dots = pinball (uncontrolled).

How to read it: Each dot represents one snapshot (96 per case). The spread of dots shows the "attractor" — the region of flow state space occupied by each case. If the blue dots overlap with the red dots, the illusion dynamics are similar to the target dynamics. If the blue dots are in a completely different region (like 1.5L), the controller is operating in a different dynamical regime.

Look at: For 1.0L, blue (illusion) should largely overlap with red (target) and be separated from green (pinball). For 0.75L, the separation is smaller. For 1.5L, the pattern may look different entirely.

---

Figure 6: Overlap heatmap — Ok_heatmap_fy_r6.png

One file, at r=6 POD rank.

What it shows: A 3x3 heatmap with columns = diameters (0.75L, 1.0L, 1.5L) and rows = comparison pairs (target-illusion, target-pinball, illusion-pinball). Colour = O_1 (the modal overlap of the first CCD mode).

How to read it: Each cell tells you how similar two cases are in their force-relevant flow structures. Dark cells (values near 0.9) mean the two cases use nearly identical lift-generating structures. Light cells (values near 0.2) mean they use very different structures.

Look at: The top row (target-illusion overlap) across diameters. For 0.75L: ~0.67, for 1.0L: ~0.92, for 1.5L: ~0.62. The progression shows the controller's force strategy diverging from the target's as the target size moves away from the pinball's natural scale.

---

Figure 7: Cross-diameter overlap — cross_diameter_overlap_fy.png

One file.

What it shows: A 3x3 heatmap showing how similar the illusion's force-CCD direction is between different diameters, when all are projected into the 1.0L target-only POD basis.

How to read it: Each cell shows O(diameter_i, diameter_j) — how aligned the force-relevant structures are between illusions at different target sizes. All values along the diagonal are 1.0 (a case is identical to itself). Off-diagonal values show cross-diameter similarity.

Look at: The O(0.75L, 1.0L) = ~0.85, O(0.75L, 1.5L) = ~0.96, O(1.0L, 1.5L) = ~0.92. Interestingly, the two "off-natural-scale" cases (0.75L and 1.5L) are MORE similar to each other in the 1.0L basis than either is to 1.0L. This suggests they use a similar "deviant" strategy.

---

Figure 8: z_1 verification — z1_verification_fy_{diam}L.png

Two files: 0.75L and 1.0L.

What it shows: The temporal coefficient of the first CCD mode (z_1, blue) overlaid with the normalised total lift force (red). Top panel: raw z_1(t). Bottom panel: both signals normalised and overlaid.

How to read it: If the blue and red lines track each other well in the bottom panel, the CCD mode is successfully capturing the lift-related structures. This is a sanity check — it shows that CCD found something real.

Look at: The overlap between the blue dashed and red solid lines in the bottom panel. Good tracking = CCD is working correctly.

---

Figure 9: 1.5L special diagnostics (3 files)

• 15L_raw_timeseries.png: Raw sensor, force, and action time series for 1.5L. Shows sensor tracking (how well illusion = target for sensors), force comparison, and the DRL action signals.
• 15L_windowed_periodicity.png: The cycle-to-cycle period variation (CV_T) over time for 1.5L. If CV_T exceeds the dashed lines, the shedding is not perfectly periodic. This confirms the "phase drift" behaviour.
• 15L_overlap_summary.png: A bar chart comparing O(target, illusion) across diameters. The 1.5L bar is annotated as "special mechanism."

How to read the periodicity figure: The top panel shows CV_T over time — values below 0.10 (red dashed line) indicate stable periodic shedding. If values frequently exceed this, the shedding period is drifting. The middle panel shows the cycle period itself. The bottom panel shows the dominant frequency. Together, they reveal whether the flow is stably periodic or drifting.

---

Figure 10: Steady cloak cancel test — steady_cloak_cancel_test.png

One file.

What it shows: Three panels comparing dq_blk (the blockage field — pinball's disturbance), dq_ctl (the control correction), and dq_ctl + dq_blk (the residual — what's left after control tries to cancel blockage).

How to read it: If the control perfectly cancels the blockage, the right panel (dq_ctl + dq_blk) should be near zero everywhere. Blue/red patterns in the right panel indicate incomplete cancellation. The presence of strong colour shows the control does not fully restore the flow.

Look at: The third panel — if it's mostly blank (near zero), the cancellation is working well. For this case, it is NOT blank, confirming the open-loop steady cloak does not effectively cancel the pinball disturbance.

---

12. Summary of Conclusions

1. The correction-field framework is the correct way to analyse this problem. It isolates what the controller actually changes, removing baseline similarity contamination.

2. 1.0L illusion is a low-rank, target-aligned correction. O(dqctl, dqtar)=0.913, m80=1. When the target matches the pinball's natural scale, the controller modulates the existing shedding channel in a near-optimal way.

3. Force and signature structures are spatially separated at zero lag but converge after convective delay. This is confirmed by both full-field CCD (O=0.41-0.55 at tau=0, rising to 0.77-0.81 at tau=tau_c) and by zone-restricted CCD (sensor zone shows O=0.01 for 0.75L at tau=0).

4. 1.5L is a genuine special mechanism, not a failure. It achieves 94% sensor similarity despite weak action coupling, strong phase drift, and a near-body-focused correction pattern.

5. The drag channel (force_fx) is unreliable for mechanism claims. It fails validation and should only be used for trend comparisons.

6. The open-loop steady cloak is ineffective (0% fluctuation suppression) and should not be a focus for mechanism analysis.

---

13. Data and Code Availability

All analysis scripts: src/CCD_analysis/correction_analysis/*.py All results: src/CCD_analysis/data/ccd/*.json All figures: src/CCD_analysis/data/figures/*.png 81 total figures, 8 JSON result files

Key result files:

• ccd_results.json — raw-field CCD (Round 5 baseline)
• correction_ccd_results.json — correction-field force/action CCD
• correction_validation_results.json — LOCO validation
• signature_ccd_results.json — signature-line CCD
• 15L_correction_results.json — 1.5L analysis
• zone_ccd_results.json — zone-restricted CCD
• steady_metrics.json — steady cloak quantitative metrics