# Phase 2 retrospective and priorities (Claude, 2026-11-17 night) Single writer: Claude. This document organises the evidence; the scores themselves live in each branch's frozen report (see `research-tree.md`). ## 1. Where the tree stands | Status | Branches | One-line reading | |---|---|---| | **Confirmed (five folds)** | B13-E1b generic region context | Field relative L2 vs A −0.00113, CI [−0.00200, −0.00026] (7.08 → 6.97 %); τ improves; last fill unconfirmed | | **Passed stage 1, confirmation running** | H2 (learned representation) | Fold 0 only: field 6.75 % vs A 7.29 %, CI excludes 0; last fill no support. S2 (folds 1–4) training; the pooled rule was pre-registered before any S2 result | | Parked | B1 / H1-RES (small field gain; last fill and τ fail), B5, B8, B10, B10x | Inconclusive or blocked on inputs/resources | | Pruned (fixed recipes only) | B4, B6, B7, B11, B12, B14, B15, B16, B17, E2 (type partition), P1–P3 | None of these closes the underlying hypothesis in general | | Infrastructure | Fresh holdout (94 sealed v2 candidates); gate calibration v2 (outcome-level, running) | The only route to an independent confirmation | ## 2. What we learned (science) 1. **Tabular feature additions to A are close to exhausted.** Eleven feature/target/loss variants gave one confirmed gain of about 1.6 % relative. The B13 gain comes from regional averaging itself; *which* rule draws the regions (generic, type-organised, shuffled) makes no detectable difference (E2). 2. **Last fill is the persistent failure.** Every branch, including the tail-weighted B17 and H2-S1, fails to confirm a last-fill improvement. The B14 / B16 near-misses did not survive a dedicated recipe. 3. **One learned-speed recipe failed.** Learning the speed and then solving the eikonal (B15) was clearly worse than A, which corrects after the solve. This does *not* show that pre-solve learning is generally inferior: the inverse problem, label identifiability, inputs and objective all changed at once (MoldAstra, 2026-11-18). 4. **Representation may be the lever.** H2's fold-0 field gain is larger than any feature change; whether it holds is S2's question. Not yet comparable with E1b (different statistical domain). 5. **Data quantity is not identified.** A's curve 7.70 → 7.31 → 7.08 % (25/50/100 % pools) shows modest gains with more data; for neural models the data-sufficiency test (MoldAstra e24fff7c) decides. ## 3. Process lessons (including my own errors) 1. **Review before score, every time.** E2 was scored before its implementation review (deviation kept on record). Since then, pre-score semantic PASS gates caught two real scorer defects before any result existed (B16: the gate didn't check the verdict; B17: dict subtraction). The pattern works; keep it. 2. **Fix the evaluation domain in the first ruling.** My E1 ruling left validation vs test ambiguous and I added a coverage gate that parked E1; E1b (native NaN, test roles) was the right design from the start. Rule: name the roles, the denominator and the missing-data policy in the first ruling. 3. **Cost projections run low.** B14 (per-part cap), B15 (disk cap, then a 0.3 % projection miss) and calibration v2 (first-10 projection 70k vs about 85k actual) all stopped or nearly stopped on engineering caps. Rule: project from at least 30–50 parts, or add 25 % margin, before fixing a cap. 4. **Wording.** I overstated twice ("last fill unchanged"; H2-S1 "largest gain in the tree"). Both were corrected by the team. Rule: a CI crossing 0 is "not confirmed", never "unchanged"; compare effects only within the same statistical domain. 5. **Do not block science on publication.** A 10 GB Git push blocked three runs until starts were decoupled from remote publication; small-branch publication then worked. 6. **Keep housekeeping proportionate.** The disk cleanup grew a multi-step admin plan; Yanqi stopped it. Simple tasks go to MeltAssist, briefly. ## 4. Priorities (in order) 1. **H2-S2 → pooled confirmation** (running; scorer being written during training; MeltAssist exact PASS before scoring). 2. **Data-sufficiency test** (MoldAstra e24fff7c) after S2, under its own GPU cap. 3. **Gate calibration v2 → fresh holdout.** If it passes, one independent deployment check of E1b and (if confirmed) H2 on new families. 4. **If H2 is confirmed — H2 follow-ups (specs first, MoldAstra):** seed variance / small ensemble; H2 with B13 context inputs; a last-fill-aware objective inside H2 (not a tabular add-on). 5. **CAD pretraining** (MoldAstra 8114bc3c) stays parked until H2 is confirmed *and* the data-sufficiency test shows a data limit. 6. **Literature (MeltAssist):** geometry-conditioned neural operators and mesh GNNs for 3D filling and endpoint (last-fill) prediction, to inform item 4.