research-tree: item b fails screen; open phase-3 stage review
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ce31bec73e
commit
98a754a636
2 files changed
+35
No files matched your search
@@ -34,6 +34,7 @@ Rules:
|
||||
| P3 | O1 with size-matched K = 8 partitions | — | — | PRUNED (eligibility 23 %; lesson: control cut area as a covariate, do not match it) |
|
||||
|
||||
Log (newest first):
|
||||
- 2026-11-19: **Item b (H2 + frozen B13 context inputs) DOES_NOT_PASS its fixed screen; PRUNED (this recipe).** Recovery postfit (PASS 029a77a2) COMPLETE, 552 / 204, evidence 192ce24a. b − H2 seed 0: field −.00006, CI [−.0018, +.0017] (needed upper bound < −.001 and |effect| > seed spread .00015: neither met); last fill −.044 Rg, CI [−.130, +.035]; τ-b +.0002, CI crosses 0. Descriptive strata (no gate): 510 parts with valid B13 context −.0008, CI [−.0026, +.0009]; 42 historically missing +.0089, CI [+.0026, +.0156] (worse where the context is absent and zero-filled). **Reading:** B13's tabular context adds nothing measurable to H2; the representation appears to already carry the regional information that helped A. Phase-3 scored set is now complete: H2 confirmed on field; seed noise small for field, large for last fill; c and b pruned; 1b parked; Track A shows H2 < A on our random splits without a paper-level reproduction. **Stage review opened** (docs/claude/stage-review-2026-11-19.md).
|
||||
- 2026-11-19: **Item c (last-fill-aware objective inside H2) FAILS its pre-registered screen; PRUNED (this recipe).** Recovery postfit (PASS 59a6255f, one labelled infrastructure recovery after a pre-worker guard KeyError, 0 targets read in the stop) COMPLETE, 552 / 204. c − H2 seed 0: last fill +.0193 Rg, CI [−.0559, +.0948] (gate needed ΔD ≤ −.010 with upper bound < 0, and |effect| > the same-domain seed spread .0482: not met); field +.00029, CI [−.0013, +.0019] (upper bound above +.001: fails); τ-b +.00037, CI crosses 0. Secondary (reported separately, no gate): last-1 % tail error E_tail +.066, CI [+.044, +.087], i.e. *worse* tail placement; peak certificate 0 for every arm. **Reading:** this location-aware objective did not move the last-filled point, and by the secondary metric it moved the late region the wrong way. That closes this recipe, not the question: the S2 diagnostic (more far-from-tail picks) still stands as H2's main weakness. 630.73 GPU-s postfit including the stop.
|
||||
- 2026-11-19: **Track A (Gre24-style random-split protocol) scored, and seed-1 noise reference scored** (MeltSol, evidence fee71ea6; original bindings 958abd8c/a5fa4a6e and 7b4da089, each run once). **Track A:** splits 0/1/2/4 (split 3 stopped on budget, disclosed; 14 preparation failures disclosed; our own random splits), 1228 paired test visits / 571 unique parts. Split-averaged median per-part RMSE/max: A 3.99 % → H2 3.25 %; per-part paired Δ −.00742, CI [−.00844, −.00646]. **Reading:** on the same re-created random splits H2 clearly beats A. This is *not* a SOTA claim against Gre24's table: our A on these splits gives 3.99 %, not the paper's 3.31-3.38 %, so the plan's reproduction gate is not met and the gap between H2 3.25 % and the paper number is not interpretable (four splits, different seeds, random splits leak families). **Seed-1 vs seed-0 (H2, grouped protocol, 552/204):** field +.00015, CI [−.00143, +.00204]; last fill −.048 Rg, CI [−.122, +.024]; τ +.00004, CI crosses 0. **Reading:** the seed spread is small for the field (about 1/30 of H2's −.005 gain, so the field confirmation is not a seed artefact). For last fill it is about .05 Rg, comparable to H2's +.080 last-fill loss, so single-seed last-fill differences are noisy. c and b will be read against the same-domain seed-1 reference, as their bindings specify.
|
||||
- 2026-11-19: **All phase-3 training terminal, no scores yet** (TRON read-only, 07:10 and 07:30 CST). seed-1 v2: five folds PASS, 105,759.84 GPU-s; postfit PASS 7b4da089 already adopted, postfit not started. b: five folds PASS, 95,685.04 GPU-s on the ledger (9,647.52 of it charged waiting), postfit binding not yet written. For c, seed-1 and b: no prediction, scoring or test-target read; 7,200 GPU-s postfit reserve each. Horsea has no compute processes. Native routing is still missing; order once it returns: c postfit review → Track A shared-slot narrow review → seed-1 postfit → b postfit binding + review.
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# Stage review after phase 3 (Claude, 2026-11-19)
|
||||
|
||||
Yanqi: at big stage points, everyone summarises, discusses and re-researches. This file is the brief. Claude rules afterwards and sends Yanqi one short stage report.
|
||||
|
||||
## Scored facts (details and CIs in `research-tree.md`)
|
||||
|
||||
| Item | Result |
|
||||
|---|---|
|
||||
| H2 (learned representation), grouped 552/204 | Field 7.08 → 6.58 % vs A, confirmed; last fill worse (+.080 Rg) |
|
||||
| seed 1 | Field seed spread ~.00015 (tiny); last-fill seed spread ~.048 Rg (large) |
|
||||
| Item 1 data sufficiency | Unidentified / overfit signs; more data helps on validation only |
|
||||
| c (last-fill objective in H2) | Fails; secondary tail error worse |
|
||||
| b (H2 + B13 context) | Fails; worse on the 42 parts without context |
|
||||
| Track A (four random splits) | H2 3.25 % vs A 3.99 % (paired CI excludes 0); A does not reproduce Gre24's 3.31-3.38 %, so no paper claim |
|
||||
| 1b (100-epoch schedule) | Parked |
|
||||
| Fresh holdout | Blocked (gate calibration v2 failed) |
|
||||
| Track B (GINO ShapeNet) | Not started |
|
||||
|
||||
## Questions for each member (independent answers first, then discussion)
|
||||
|
||||
1. What do these results say about where H2's remaining error comes from (field vs last-fill location), and what is the single most promising next branch? Justify against the evidence above.
|
||||
2. Track A: what would make a Gre24 comparison legitimate (reproduce A at 3.31-3.38 % on their protocol, or argue the gap)? Is it worth the cost now, or should Track B (GINO public benchmark, cleaner protocol) take priority for the SOTA goal?
|
||||
3. Last fill: given the seed spread (.048 Rg) and two failed recipes (B17, c), is last fill a target to keep attacking, to measure differently (set-based / multi-seed), or to park?
|
||||
4. Process: what went well and what cost us time in phase 3 (routing outage, budget stops, missing contract field, review rounds)? One concrete rule each.
|
||||
|
||||
## Who
|
||||
|
||||
- MeltSol: executor view: costs, failures, what infrastructure would make the next phase cheaper. Also the bulk literature view on Track B SOTA.
|
||||
- MeltAssist: reviewer view: where review caught or missed defects.
|
||||
- MoldAstra: maths view on Q1 and Q3 (short).
|
||||
- TRON: routing and coordination lessons (Q4 only).
|
||||
- NewMeltAstra, MeltAstra: not asked (save Astra quota); MeltAstra retires now that c, b and Track A are scored.
|
||||
|
||||
Each answer: at most one page, posted to Git (MeltSol publishes), link in Slack. No new compute during the review.
|
||||
Reference in new issue
Block a user