87 lines
11 KiB
Markdown
87 lines
11 KiB
Markdown
# S1 M2 v1.2 scorer: code review packet
|
||
|
||
This packet implements the six-arm scorer and its synthetic checks. It contains **no real S1 scores, fitted real-data ranks, or model training**. Only the released panel CSV and fold CSV were read to calculate the query cost. The input adapter has been exercised on generated temporary NPZ files only.
|
||
|
||
Authority and fixed inputs:
|
||
|
||
- [Claude code-review scope, 15:36:39](https://arcticnimbus.slack.com/archives/C0C78UFKGRM/p1794641799442609).
|
||
- [Actual start and implementation questions](https://arcticnimbus.slack.com/archives/C0C78UFKGRM/p1794642190451619).
|
||
- [Claude interval/RNG rulings, 15:43:44](https://arcticnimbus.slack.com/archives/C0C78UFKGRM/p1794642224144079). This adds source bootstrap 1407 to the earlier Drive seed table.
|
||
- Released O2 panel/shortfall parent `a86a74cbd943e0687ee69fb1cccdc003250113cc`; original draws `94011fd8efe0512e6579c7c541ccd585dc710c2e`. No numerical producer, original draw, field or eligibility mask was modified.
|
||
- Current M2 source snapshot and both Slack rulings are included verbatim. Historical sections of that document retain earlier definitions; its v1.2 section and the later rulings take precedence.
|
||
|
||
## What to review
|
||
|
||
`s1_scoring.py` is the pure numerical API. `score_fold(training, test)` returns all node scores, neighbors, rank maps, actual weights, permutation/clipping diagnostics, per-part ratios and paired query replicates. `summarize_folds` reports all arms and contrasts, each seed and the mean of all five seeds. `s1_scoring_io.py` reads frozen sampled outputs and converts NA to JSON null. There is deliberately no real-panel launch command in this code-review packet: the future supervised launcher must pin the released input manifests, verify the current process release, fix wall/RSS limits and persist each complete fold before moving on. This is an execution prerequisite, not another information-sharing approval.
|
||
|
||
### Coordinates and ranks
|
||
|
||
Columns are `[d/t_part, gate_distance/R_g, g12, g23, gate_q1, gate_q2, edge_mean_q1, edge_max_q1, edge_mean_q2, edge_max_q2]`.
|
||
|
||
| Arm | Dimensions | Training modification |
|
||
|---|---:|---|
|
||
| A | 2 | none |
|
||
| A+S | 6 | permute columns 2:6 as a single block within each part |
|
||
| B | 6 | none |
|
||
| B+edge_shuffle | 10 | permute columns 6:10 as a single block within each part |
|
||
| C | 10 | none |
|
||
| B+stratified_edge_shuffle | 10 | jointly permute gap-normalized edge block within exact positive-gap strata; multiply receiver gaps |
|
||
|
||
Queries always retain their true descriptors. Each fold fits ten true-column midrank maps once. Eligible node weight is `c/(P*K_pE)`; there is no second mass factor. Exact equal values have a common midrank, constant columns remain in the denominator and take 0.5 at their training value. Strict tails map to 0/1. Query audits include both slices, weighted draw-slot tail fractions and raw ranges. Reconstructed control references receive per-part tail audits against the same frozen maps. Shuffled values are never refitted.
|
||
|
||
### Distances and identity
|
||
|
||
Each distinct eligible physical node contributes exactly one reference slot, even when its draw multiplicity exceeds one. Different physical nodes with equal descriptors remain distinct. Physical key is `(source_key, case, stable anchor node ID)`; raw-node identity is checked against the draw file by the adapter. Training parts are sorted by `(source,case)` and nodes retain ascending anchor ID order. One representative per source is enforced; test source leakage fails.
|
||
|
||
Distance is `sqrt(mean((rank(query)-rank(reference))**2))`. Node M2 is the mean of eight nearest distances. Exact cKDTree Euclidean search finds the eighth radius; a second radius query with `nextafter(d8*(1+64*eps), +inf)` gathers boundary candidates, then direct RMS distances and physical IDs determine the final eight. This retrieval padding does not round distances into ties or alter them. Independent brute-force tests include repeated descriptor ties in 2, 6 and 10 dimensions. A tie-heavy dataset can expand to all references; the cost plan includes this bound. Zero-distance counts and neighbor-source concentration are retained per query.
|
||
|
||
### Random numbers
|
||
|
||
All generators are PCG64 with NumPy SeedSequence; NumPy 1.26.4 is the replay environment. Different numerical library versions are not claimed bitwise identical.
|
||
|
||
- Ordinary permutations: `[2026111401..2026111405, prefix]`, fresh generator per part/seed. A+S and plain edge shuffle share the same row index. Mapping is reproducibly regenerated from the same sorted eligible nodes across folds.
|
||
- Normalized shuffle: a separate fresh generator initialized with the same part/seed. Strata order `(0,0),(0,1),(1,0),(1,1)` uses exact `gap>0`. Only strata of size at least eight consume a permutation. Smaller strata keep exact original raw edge blocks. Zero-gap normalized channels are zero. No rejection, clipping, repair or redraw. Audits report exact violations and diagnostics above tolerance `1e-10`, plus donor modes, index changes, value changes and denominators.
|
||
- Query bootstrap: `[2026111406,prefix]`, 200 sequential replicates; each draws 2,000 original draw-slot indices on the real panel. Six arms, slices and five seeds share the resulting counts. Synthetic tests use smaller explicitly generated draw lists. No new nearest-neighbor queries.
|
||
- Source bootstrap: `[2026111407]`, 2,000 sequential resamples of the 24 lexicographically sorted source keys with replacement. One shared index matrix for all arms/contrasts/slices/seeds. No prefix for this source-level generator.
|
||
|
||
`synthetic_seed_vectors.json` contains complete small permutations, query/source first-three vectors and digests over dtype, shape and full bytes. This is the targeted independent replay material for Grok, together with the hand-ranked example below.
|
||
|
||
### Ratios, intervals and interpretation
|
||
|
||
Per-part whole/near scores use `sum(c*H*score)/sum(c*H)`, selecting accepted entries before multiplication. Zero denominator is NA. Near-wall only restricts queries to `0<d/t<=0.25`; the reference pool is unchanged. Coverage reports accepted fraction among positive-distance draws, its product with the full-part positive-mass fraction, and the unscored boundary share. The near fraction of eligible draws has a separate denominator.
|
||
|
||
Aggregate within source with equal part weight, then across sources with equal weight; also report part-equal totals. Required missing parts make the fixed-source aggregate NA, never silently renormalize. Fewer than eight total distinct references produces missing scores; a real shortfall fold returns its noncomparable status and is not aggregated. The small-fixture gate bypass rejects parts tagged with real provenance. Shortfall uses strict `<500`, `<1000` and integer `5*count_below_1000 > P`.
|
||
|
||
Query SE is the valid replicate sample standard deviation (ddof=1), without division by sqrt(200). Both intervals use two-sided 95% percentile quantiles with linear interpolation. Original point NA or fewer than two valid replicates suppresses its interval; valid/total replicate counts are recorded. Pair differences are formed inside each replicate. The five-seed mean is taken before interval construction, with all five seed results retained.
|
||
|
||
For each primary contrast, the **source interval** strictly below zero means ranks ahead, strictly above zero ranks behind, otherwise no ranking difference; touching zero is not a win. No effect-size margin. Primary contrasts are B−(A+S) and C−(B+edge_shuffle); C−(B+stratified_edge_shuffle) is secondary, B−A and C−B descriptive. No joint scientific success claim is inferred from one contrast. All arms' scores and both levels of interval are available, but cross-dimension differences do not establish improved geometry information.
|
||
|
||
Source intervals resample the fixed LOPO source statistics; they do not refit overlapping training folds, ranks, or reference pools. Query intervals condition on these same realized objects and measure draw-slot variation only. Neither interval validates filling accuracy, physical bias, remeshing robustness or model generalization.
|
||
|
||
## Hand expectations and tests
|
||
|
||
The tests write expected values independently of the scorer:
|
||
|
||
- Three-part midrank fixture: eligible weights `[1/9,2/9]`, `[1/6,1/6]`, `[1/4,1/12]`. Masses at values 0,1,2 are 5/18,17/36,1/4. Ranks at 0,1,2 are **5/36,37/72,7/8**; rank at unseen 1.5 is 3/4.
|
||
- Three training parts each have ten distinct physical nodes at identical ten descriptor levels across columns. Eight-neighbor endpoint score is `(3*0+3*.1+2*.2)/8 = 7/80`; interior is `(3*0+5*.1)/8 = 1/16`. Ten test nodes with multiplicities `[1,2,1,1,1,1,1,1,1,1]` give whole **59/880**, first-three near slice **11/160**.
|
||
- Part statistics 1,3,10 from sources a,a,b give source-equal **6**, part-equal **14/3**. Replicates 1,2,3 give SE **1**, interval **[1.05,2.95]**. Hand-chosen paired source resamples produce **[-2.925,-1.075]**.
|
||
- Additional checks cover constant-column `a -> a/sqrt(2)`, all-arm rigid motion with the gate (plus an unmoved-gate negative control), seed byte replay, ordinary block tuples, normalized zero/positive strata and cross-channel order, strict shortfall endpoints, distinct node identity, NA propagation, source leakage, input hashes and JSON null serialization.
|
||
|
||
Initial test correction is disclosed: an assertion that `sqrt(a*a/2)` and `a/sqrt(2)` are bitwise equal failed by one FP64 ULP (0.21213203435596426 versus 0.21213203435596423). The test now checks exact agreement with the defined RMS evaluation and agreement with the algebraic alternative within relative `2e-16`. No scorer formula was changed for this failure.
|
||
|
||
Run synthetic verification from the packet directory with Python 3.10, NumPy 1.26.4, SciPy 1.15.3:
|
||
|
||
```
|
||
OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 python test_s1_scoring.py
|
||
python s1_seed_vectors.py replay.json
|
||
python plan_s1_queries.py --summary summary.csv --folds fold_table.csv --out local-plan
|
||
```
|
||
|
||
## Cost and next execution boundary
|
||
|
||
Every fold evaluates 18 configurations: A/B/C once plus three shuffled controls times five seeds. This is one 2D, six 6D and eleven 10D configurations. All true baselines are reused across seeds. Near-wall slices reuse query scores; neither 200 query replicates nor 2,000 source replicates add kNN calls.
|
||
|
||
Across 24 folds there are **43,656 eligible unique test nodes**, **785,808 logical 8NN queries**, **785,808 additional radius queries**, and **432 tree builds**. `query_cost.csv` lists every fold. Worst-case radius tie expansion totals 32,868,059,868 candidate distance evaluations; it is a conservative bound, not a measured workload. Matrix byte counts in the table are component sizes, not an RSS guarantee. No scoring-runtime extrapolation is claimed from the earlier O2 integrations.
|
||
|
||
Claude is the full reviewer. Grok's scope is independent seed replay and weighted-midrank formula checking. Real S1 execution remains pending this review and an explicit supervised execution scope with wall/RSS limits. Information-sharing authorization is already present and does not need to be requested again.
|