Files
MeltMind/docs/claude/phase2-retrospective-2026-11-17.md
T
2026-11-18 01:37:16 +00:00

4.7 KiB
Raw Blame History

Phase 2 retrospective and priorities (Claude, 2026-11-17 night)

Single writer: Claude. This document organises the evidence; the scores themselves live in each branch's frozen report (see research-tree.md).

1. Where the tree stands

Status Branches One-line reading
Confirmed (five folds) B13-E1b generic region context Field relative L2 vs A −0.00113, CI [−0.00200, −0.00026] (7.08 → 6.97 %); τ improves; last fill unconfirmed
Passed stage 1, confirmation running H2 (learned representation) Fold 0 only: field 6.75 % vs A 7.29 %, CI excludes 0; last fill no support. S2 (folds 1–4) training; the pooled rule was pre-registered before any S2 result
Parked B1 / H1-RES (small field gain; last fill and τ fail), B5, B8, B10, B10x Inconclusive or blocked on inputs/resources
Pruned (fixed recipes only) B4, B6, B7, B11, B12, B14, B15, B16, B17, E2 (type partition), P1–P3 None of these closes the underlying hypothesis in general
Infrastructure Fresh holdout (94 sealed v2 candidates); gate calibration v2 (outcome-level, running) The only route to an independent confirmation

2. What we learned (science)

  1. Tabular feature additions to A are close to exhausted. Eleven feature/target/loss variants gave one confirmed gain of about 1.6 % relative. The B13 gain comes from regional averaging itself; which rule draws the regions (generic, type-organised, shuffled) makes no detectable difference (E2).
  2. Last fill is the persistent failure. Every branch, including the tail-weighted B17 and H2-S1, fails to confirm a last-fill improvement. The B14 / B16 near-misses did not survive a dedicated recipe.
  3. One learned-speed recipe failed. Learning the speed and then solving the eikonal (B15) was clearly worse than A, which corrects after the solve. This does not show that pre-solve learning is generally inferior: the inverse problem, label identifiability, inputs and objective all changed at once (MoldAstra, 2026-11-18).
  4. Representation may be the lever. H2's fold-0 field gain is larger than any feature change; whether it holds is S2's question. Not yet comparable with E1b (different statistical domain).
  5. Data quantity is not identified. A's curve 7.70 → 7.31 → 7.08 % (25/50/100 % pools) shows modest gains with more data; for neural models the data-sufficiency test (MoldAstra e24fff7c) decides.

3. Process lessons (including my own errors)

  1. Review before score, every time. E2 was scored before its implementation review (deviation kept on record). Since then, pre-score semantic PASS gates caught two real scorer defects before any result existed (B16: the gate didn't check the verdict; B17: dict subtraction). The pattern works; keep it.
  2. Fix the evaluation domain in the first ruling. My E1 ruling left validation vs test ambiguous and I added a coverage gate that parked E1; E1b (native NaN, test roles) was the right design from the start. Rule: name the roles, the denominator and the missing-data policy in the first ruling.
  3. Cost projections run low. B14 (per-part cap), B15 (disk cap, then a 0.3 % projection miss) and calibration v2 (first-10 projection 70k vs about 85k actual) all stopped or nearly stopped on engineering caps. Rule: project from at least 30–50 parts, or add 25 % margin, before fixing a cap.
  4. Wording. I overstated twice ("last fill unchanged"; H2-S1 "largest gain in the tree"). Both were corrected by the team. Rule: a CI crossing 0 is "not confirmed", never "unchanged"; compare effects only within the same statistical domain.
  5. Do not block science on publication. A 10 GB Git push blocked three runs until starts were decoupled from remote publication; small-branch publication then worked.
  6. Keep housekeeping proportionate. The disk cleanup grew a multi-step admin plan; Yanqi stopped it. Simple tasks go to MeltAssist, briefly.

4. Priorities (in order)

  1. H2-S2 → pooled confirmation (running; scorer being written during training; MeltAssist exact PASS before scoring).
  2. Data-sufficiency test (MoldAstra e24fff7c) after S2, under its own GPU cap.
  3. Gate calibration v2 → fresh holdout. If it passes, one independent deployment check of E1b and (if confirmed) H2 on new families.
  4. If H2 is confirmed — H2 follow-ups (specs first, MoldAstra): seed variance / small ensemble; H2 with B13 context inputs; a last-fill-aware objective inside H2 (not a tabular add-on).
  5. CAD pretraining (MoldAstra 8114bc3c) stays parked until H2 is confirmed and the data-sufficiency test shows a data limit.
  6. Literature (MeltAssist): geometry-conditioned neural operators and mesh GNNs for 3D filling and endpoint (last-fill) prediction, to inform item 4.