Files
MeltMind/docs/s1-real-scoring-results-2026-11-14.md
T

52 lines
8.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# S1 M2 v1.2 — 24 supervised LOPO folds
All 24 folds completed and passed verification. **The two primary contrasts have opposite directions:** B ranks behind its shuffled node/gate control, while C ranks ahead of the ordinary edge-shuffle control. This does not meet the proposed joint interpretation that both organized blocks make inputs more familiar. No prediction model was trained.
## Primary result table
Decisions use the **five-permutation-seed mean** and the **固定 LOPO 配对差的来源重加权区间** (source-reweighting interval over fixed LOPO paired differences). Differences are absolute M2 rank-distance units; smaller is more familiar. These are two-sided 95% linear-percentile intervals from seed 1407 with 2,000 resamples. They do not refit ranks or reference pools and do not guarantee 95% coverage for new sources. Overlapping LOPO training folds remain fixed.
| Contrast | Query slice | Five-seed mean difference | Fixed-LOPO source-reweighting interval | Ranking | Seeds with same point direction / interval classification |
|---|---|---:|---|---|---|
| B − (A+S) | Whole | +0.00262105 | [+0.00137530, +0.00408964] | B behind | 5/5 / 5/5 |
| B − (A+S) | Near wall | +0.00171341 | [+0.00039639, +0.00318377] | B behind | 5/5 / 5/5 |
| C − (B+edge shuffle) | Whole | −0.01495009 | [−0.01620100, −0.01378732] | C ahead | 5/5 / 5/5 |
| C − (B+edge shuffle) | Near wall | −0.01582829 | [−0.01691546, −0.01481519] | C ahead | 5/5 / 5/5 |
Per-seed results are stability diagnostics only; they do not vote on the primary decision. An interval including or touching zero would mean **未分先后 / not separated**, never equivalent. `decision_table.csv` contains full precision, both source and query intervals, valid replicate counts, source-equal and part-equal differences, and the two separately labeled seed counts. Every contrast/slice has all 2,000 valid source replicates and 200 valid query replicates.
The secondary normalized edge-shuffle contrast is much smaller: C minus the stratified control is −0.00055111, interval [−0.00091199, −0.00023184] for whole queries, and −0.00055050, interval [−0.00095840, −0.00015434] for near-wall queries. Both point directions and interval classifications agree across 5/5 seeds. It remains a secondary diagnostic, not a replacement for a failed primary comparison.
B−A and C−B are included as descriptive cross-dimension differences. RMS normalization does not remove dimension/redundancy effects. These coverage measurements establish neither filling prediction accuracy nor improved sample efficiency.
## Complete evidence
All six arms, all five seeds, both slices and all 24 sources are retained. `arm_summary.csv` gives aggregate scores and intervals; `part_scores.csv` gives per-part scores, coverage and query-bootstrap diagnostics; `seed_stability.csv` reports each contrast/seed without assigning it primary decision status. Source-equal and part-equal totals agree to floating precision because this panel has one part per source.
`query_clipping.csv`, `control_clipping_summary.csv`, `neighbor_diagnostics.csv` and `shuffle_audits.json` expose numerical and control behavior. Full per-reference-part clipping details, rank maps/weights, physical reference IDs, eight selected neighbors and distances, node scores, query draw-slot bootstrap counts, and replicate estimates are retained in each fold's `structure.json` plus `arrays.npz` inside the evidence archive. Nothing was dropped from failed/rejected intermediates; there were no failed real folds.
Across 24 distinct parts × 5 permutations (218,280 eligible slots, counted once per part/seed, not 23 times for repeated training folds):
- Ordinary edge shuffle produces 187,525 nodes with raw-bound excess above 1e-10 and 106,156 normalized cross-channel order violations above 1e-10. This control deliberately breaks the association with receiver gaps. Its large contrast alone does not isolate physically feasible local structure.
- Stratified normalized shuffle has **zero** bound or cross-channel order violations above 1e-10. Exact comparisons find 325 bound and 339 order violations, with maximum excess 8.881784197001252e-16 in either category; these are retained without repair.
- Query clipping occurs in all ten columns somewhere in the panel. The largest combined tail fraction for one part/column/slice is 0.1865. No training column is constant in these folds. Complete ranges, sides and denominators are published.
## Execution and verification
- Frozen scorer is unchanged from `e425419c8012127400e8945baa8d04e2953646bd`. Original O2/draw inputs remain those released at `a86a74cbd943e0687ee69fb1cccdc003250113cc` / `94011fd8efe0512e6579c7c541ccd585dc710c2e`.
- [Claude main review and conditional release](https://arcticnimbus.slack.com/archives/C0C78UFKGRM/p1794643516496129), [Grok independent seed/formula PASS](https://arcticnimbus.slack.com/archives/C0C78UFKGRM/p1794643523014799), [interval label agreement](https://arcticnimbus.slack.com/archives/C0C78UFKGRM/p1794643532162559), [actual continuation receipt](https://arcticnimbus.slack.com/archives/C0C78UFKGRM/p1794643986876739). Claude reviewed source and hashes but did not rerun synthetic tests in his environment; Grok independently replayed the vectors and formula, not the whole scorer.
- Execution contract SHA256: `c12c54f825c7c1c958a6951e3ea62c6339404bb1328511482790f4dfff528210`. Both presentation pins were fixed before launch. The supervisor adapter reuses the existing whole-process memory-tree implementation, 20 ms polling target, TERM/KILL handling and final peak check; only job invocation and receipt handling were adapted to S1.
- 24 folds ran sequentially with 1,800 s / 1.25 GiB limits per complete child process, including input load, scoring and serialization. No budget was enlarged. Total supervised fold time **270.5283 s**, maximum **13.2551 s**, maximum peak RSS **146.0078 MiB**. One child process was observed at a time.
- Polling was a **20 ms target**, not a timing guarantee: the largest observed gap was **0.586656 s**. Per-process high-water marks and final `ru_maxrss` supplement polling. All raw traces and supervisor receipts are retained.
- Exactly **785,808 logical 8NN queries**, **785,808 radius searches**, **432 tree builds**, and **6,286,464 direct candidate distances**. Every fold matches `query_cost.csv`; candidate count is exactly eight per logical query on this dataset. Near-wall slicing and bootstrap add no kNN queries.
- Four synthetic execution/storage tests pass, including normal completion, timeout, memory-stop and byte-preserving NPZ roundtrip. Three additional reporting tests prove that only the mean interval decides, seed agreement is separate, zero endpoints mean not separated, and the secondary control is not promoted to a primary decision. The original 14 scorer tests and their source are unchanged.
- Verification checks every saved file against the supervision receipts; recomputes every stored point/query-replicate ratio; checks eight distinct physical neighbors, score/distance means, source isolation, rank/weight invariants, baseline reuse and part permutations across folds. Aggregation used no new kNN queries or rank fits. Its separate runtime was 10.7197 s and peak RSS 125.9844 MiB.
## Reading and replaying the archive
The archive includes every real-run artifact, exact execution/reporting source, frozen input metadata needed for verification, original approvals, test logs and a file-by-file SHA256 manifest. It uses workspace-relative paths. `structure.json` refers to numbered arrays in `arrays.npz`; `run_s1_fold.unpack_tree` restores numeric arrays and tuple-key diagnostics with `allow_pickle=False`. Nonfinite scalar tags are explicit; array bit patterns are preserved.
For a report-only replay after extraction, keep the extracted workspace as the current directory. Create a new empty report directory, copy `execution_contract.json` and `completion.json` into it, and symlink its `folds` to the extracted real-run `folds`. Run `tools/report_s1_scoring.py NEW_REPORT_DIRECTORY` using Python 3.10 / NumPy 1.26.4 / SciPy 1.15.3 and `OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2`. This validates and aggregates fixed stored scores without loading real descriptors for new scoring. Compare its eight CSV tables and `aggregate.json` with the originals. Re-running the real folds would require the earlier full O2/draw evidence packages and a separate authorized execution scope.
The package is submitted to Claude for main result review. Grok's required targeted check is complete; no duplicate full review is requested.