Files
MeltMind/docs/claude/stage-review-S2-result-2026-11-15.md
T

3.9 KiB

Stage review: S2 result → next stage (Claude, 2026-11-15)

Source: MeltMind 1dea89d6 (docs/s2-custody-results-2026-11-15.md). Predictions and scoring code were frozen at c47dd6a before custody decoding. Nothing here re-scores or re-trains anything.

1. What S2 established (100 % pool, 209 components / 617 parts, 15/15 pairs)

Arm mean rel. L2 near-wall Kendall tau-b last-filled dist/Rg
N (Transolver w128, E_max 42) 0.1080 0.1101 0.8924 0.477
E (N + 4 edge q) 0.1094 0.1113 0.8924 0.494
R2 (Gre24-style GBRT) 0.0730 0.0749 0.9245 0.438
B0 (V*_f x t_set) 0.1040 0.0872 0.9071 0.342
  • Primary, pre-registered: Delta = 1 - E/N = -1.29 %, 95 % component bootstrap CI [-2.12 %, -0.56 %], so the region is harmful. With this backbone, the edge-direction readings slightly raise the error. Same sign at the 50 % pool (folds 0, 2, 3, 4) and in the not-known-exposed row. At the 25 % pool the result is inconclusive.
  • Usefulness condition fails: N is 48 % worse than R2 (CI [-56 %, -40 %]). N is not distinguishable from the naive B0 on relative L2: its point estimate is worse, but the 95 % interval of the N-vs-B0 contrast includes 0. B0 has better point estimates on Kendall tau-b, near-wall error and last-filled location; those comparisons are descriptive. (Corrected 2026-11-15 after MeltAstra and Mold pointed out the N-vs-B0 interval; my first version wrote 'worse than B0' as if it were established.)

2. Reading

  • The E-vs-N result is valid for this backbone and training budget only. The backbone does not beat even the naive B0, so the result cannot be generalised to "edge readings hurt a strong model". It is a negative result inside a weak model.
  • The bigger finding is that the neural route as configured (width 128 chosen for feasibility, 42 epochs, batch of one part, direct t* target) is not yet a useful surrogate. R2 sets the bar.
  • R2 does not dominate every metric: B0 locates the last-filled region better. The volume-fraction prior carries real information that the learned models partly lose.
  • Nothing here concerns O1, which is still unscheduled.

3. Back to the original intent

The goal is a practically useful surrogate for unseen parts; geometry organisation is a means to generalisation and sample efficiency. After S2, the strongest working model is the distance-feature GBRT. The O2 question therefore has to be asked inside the strongest model, and the neural route has to clear R2 before any neural feature ablation means anything.

4. Proposed next steps (for discussion; nothing launched)

D. Read-only diagnostics on existing S2 outputs (no training, no new labels; CPU only):

  • D1 learning curves: N/E/R2/B0 at 25/50/100 % (component fraction, actual counts). Does N improve with data at all?
  • D2 training dynamics: the best epoch and the validation curve for the 100 % runs. Is the model still improving at epoch 42, i.e. under-trained?
  • D3 scale vs shape: for each part, split the relative L2 into a part-level scale error (the best single scalar rescaling, used only as an oracle diagnostic) and the remaining shape error, for all four arms. This shows whether N loses on the overall part fill time or on the arrival-order field.
  • D4 where N loses to B0: per-component differences against size, thickness proxy (t_part/Rg) and node count, descriptive only.

S3-A (cheap, CPU): the O2 edge readings inside the strong model. R2 versus R2 plus the same 4 edge q readings (and optionally g12/g23 as a separate arm), on the same 617 roster, folds, HPO grid, statistics and decision table. This is a one-factor test on a model that already works.

S3-B (after D1-D3): neural backbone repair, only if the diagnostics point to a fixable cause. For example, a residual target over B0 or R2, a longer schedule, or R2 features as inputs. The usefulness gate stays: the model must beat R2.

O1 remains the next design item and is not refuted by anything here.