Stage review: S2 result and proposed next steps (diagnostics, O2-in-R2, backbone repair)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
dfe2c65b9b
commit
47259fee57
1 file changed
+36
@@ -0,0 +1,36 @@
|
||||
# Stage review: S2 result → next stage (Claude, 2026-11-15)
|
||||
|
||||
Source: MeltMind 1dea89d6 (docs/s2-custody-results-2026-11-15.md). Predictions and scoring code were frozen at c47dd6a before custody decoding. Nothing here re-scores or re-trains anything.
|
||||
|
||||
## 1. What S2 established (100 % pool, 209 components / 617 parts, 15/15 pairs)
|
||||
| Arm | mean rel. L2 | near-wall | Kendall tau-b | last-filled dist/Rg |
|
||||
|---|---|---|---|---|
|
||||
| N (Transolver w128, E_max 42) | 0.1080 | 0.1101 | 0.8924 | 0.477 |
|
||||
| E (N + 4 edge q) | 0.1094 | 0.1113 | 0.8924 | 0.494 |
|
||||
| R2 (Gre24-style GBRT) | 0.0730 | 0.0749 | 0.9245 | 0.438 |
|
||||
| B0 (V*_f x t_set) | 0.1040 | 0.0872 | 0.9071 | 0.342 |
|
||||
|
||||
- **Primary, pre-registered:** Delta = 1 - E/N = -1.29 %, 95 % component bootstrap CI [-2.12 %, -0.56 %], so the region is **harmful**. With this backbone, the edge-direction readings slightly raise the error. Same sign at the 50 % pool (folds 0, 2, 3, 4) and in the not-known-exposed row. At the 25 % pool the result is inconclusive.
|
||||
- **Usefulness condition fails:** N is 48 % worse than R2 (CI [-56 %, -40 %]). N is also worse than the naive B0 on relative L2, Kendall tau-b, near-wall error and last-filled location.
|
||||
|
||||
## 2. Reading
|
||||
- The E-vs-N result is valid **for this backbone and training budget only**. The backbone loses even to B0, so the result cannot be generalised to "edge readings hurt a strong model". It is a negative result inside a weak model.
|
||||
- The bigger finding is that the neural route as configured (width 128 chosen for feasibility, 42 epochs, batch of one part, direct t* target) is **not yet a useful surrogate**. R2 sets the bar.
|
||||
- R2 does not dominate every metric: B0 locates the last-filled region better. The volume-fraction prior carries real information that the learned models partly lose.
|
||||
- Nothing here concerns O1, which is still unscheduled.
|
||||
|
||||
## 3. Back to the original intent
|
||||
The goal is a practically useful surrogate for unseen parts; geometry organisation is a means to generalisation and sample efficiency. After S2, the strongest working model is the distance-feature GBRT. The O2 question therefore has to be asked **inside the strongest model**, and the neural route has to clear R2 before any neural feature ablation means anything.
|
||||
|
||||
## 4. Proposed next steps (for discussion; nothing launched)
|
||||
**D. Read-only diagnostics on existing S2 outputs** (no training, no new labels; CPU only):
|
||||
- D1 learning curves: N/E/R2/B0 at 25/50/100 % (component fraction, actual counts). Does N improve with data at all?
|
||||
- D2 training dynamics: the best epoch and the validation curve for the 100 % runs. Is the model still improving at epoch 42, i.e. under-trained?
|
||||
- D3 scale vs shape: for each part, split the relative L2 into a part-level scale error (the best single scalar rescaling, used only as an oracle diagnostic) and the remaining shape error, for all four arms. This shows whether N loses on the overall part fill time or on the arrival-order field.
|
||||
- D4 where N loses to B0: per-component differences against size, thickness proxy (t_part/Rg) and node count, descriptive only.
|
||||
|
||||
**S3-A (cheap, CPU): the O2 edge readings inside the strong model.** R2 versus R2 plus the same 4 edge q readings (and optionally g12/g23 as a separate arm), on the same 617 roster, folds, HPO grid, statistics and decision table. This is a one-factor test on a model that already works.
|
||||
|
||||
**S3-B (after D1-D3): neural backbone repair, only if the diagnostics point to a fixable cause.** For example, a residual target over B0 or R2, a longer schedule, or R2 features as inputs. The usefulness gate stays: the model must beat R2.
|
||||
|
||||
**O1** remains the next design item and is not refuted by anything here.
|
||||
Reference in new issue
Block a user