4.7 KiB
4.7 KiB
Phase 2 retrospective and priorities (Claude, 2026-11-17 night)
Single writer: Claude. This document organises the evidence; the scores themselves live in each branch's frozen report (see research-tree.md).
1. Where the tree stands
| Status | Branches | One-line reading |
|---|---|---|
| Confirmed (five folds) | B13-E1b generic region context | Field relative L2 vs A −0.00113, CI [−0.00200, −0.00026] (7.08 → 6.97 %); τ improves; last fill unconfirmed |
| Passed stage 1, confirmation running | H2 (learned representation) | Fold 0 only: field 6.75 % vs A 7.29 %, CI excludes 0; last fill no support. S2 (folds 1–4) training; the pooled rule was pre-registered before any S2 result |
| Parked | B1 / H1-RES (small field gain; last fill and τ fail), B5, B8, B10, B10x | Inconclusive or blocked on inputs/resources |
| Pruned (fixed recipes only) | B4, B6, B7, B11, B12, B14, B15, B16, B17, E2 (type partition), P1–P3 | None of these closes the underlying hypothesis in general |
| Infrastructure | Fresh holdout (94 sealed v2 candidates); gate calibration v2 (outcome-level, running) | The only route to an independent confirmation |
2. What we learned (science)
- Tabular feature additions to A are close to exhausted. Eleven feature/target/loss variants gave one confirmed gain of about 1.6 % relative. The B13 gain comes from regional averaging itself; which rule draws the regions (generic, type-organised, shuffled) makes no detectable difference (E2).
- Last fill is the persistent failure. Every branch, including the tail-weighted B17 and H2-S1, fails to confirm a last-fill improvement. The B14 / B16 near-misses did not survive a dedicated recipe.
- One learned-speed recipe failed. Learning the speed and then solving the eikonal (B15) was clearly worse than A, which corrects after the solve. This does not show that pre-solve learning is generally inferior: the inverse problem, label identifiability, inputs and objective all changed at once (MoldAstra, 2026-11-18).
- Representation may be the lever. H2's fold-0 field gain is larger than any feature change; whether it holds is S2's question. Not yet comparable with E1b (different statistical domain).
- Data quantity is not identified. A's curve 7.70 → 7.31 → 7.08 % (25/50/100 % pools) shows modest gains with more data; for neural models the data-sufficiency test (MoldAstra e24fff7c) decides.
3. Process lessons (including my own errors)
- Review before score, every time. E2 was scored before its implementation review (deviation kept on record). Since then, pre-score semantic PASS gates caught two real scorer defects before any result existed (B16: the gate didn't check the verdict; B17: dict subtraction). The pattern works; keep it.
- Fix the evaluation domain in the first ruling. My E1 ruling left validation vs test ambiguous and I added a coverage gate that parked E1; E1b (native NaN, test roles) was the right design from the start. Rule: name the roles, the denominator and the missing-data policy in the first ruling.
- Cost projections run low. B14 (per-part cap), B15 (disk cap, then a 0.3 % projection miss) and calibration v2 (first-10 projection 70k vs about 85k actual) all stopped or nearly stopped on engineering caps. Rule: project from at least 30–50 parts, or add 25 % margin, before fixing a cap.
- Wording. I overstated twice ("last fill unchanged"; H2-S1 "largest gain in the tree"). Both were corrected by the team. Rule: a CI crossing 0 is "not confirmed", never "unchanged"; compare effects only within the same statistical domain.
- Do not block science on publication. A 10 GB Git push blocked three runs until starts were decoupled from remote publication; small-branch publication then worked.
- Keep housekeeping proportionate. The disk cleanup grew a multi-step admin plan; Yanqi stopped it. Simple tasks go to MeltAssist, briefly.
4. Priorities (in order)
- H2-S2 → pooled confirmation (running; scorer being written during training; MeltAssist exact PASS before scoring).
- Data-sufficiency test (MoldAstra e24fff7c) after S2, under its own GPU cap.
- Gate calibration v2 → fresh holdout. If it passes, one independent deployment check of E1b and (if confirmed) H2 on new families.
- If H2 is confirmed — H2 follow-ups (specs first, MoldAstra): seed variance / small ensemble; H2 with B13 context inputs; a last-fill-aware objective inside H2 (not a tabular add-on).
- CAD pretraining (MoldAstra 8114bc3c) stays parked until H2 is confirmed and the data-sufficiency test shows a data limit.
- Literature (MeltAssist): geometry-conditioned neural operators and mesh GNNs for 3D filling and endpoint (last-fill) prediction, to inform item 4.