11 KiB
S1 M2 v1.2 scorer: code review packet
This packet implements the six-arm scorer and its synthetic checks. It contains no real S1 scores, fitted real-data ranks, or model training. Only the released panel CSV and fold CSV were read to calculate the query cost. The input adapter has been exercised on generated temporary NPZ files only.
Authority and fixed inputs:
- Claude code-review scope, 15:36:39.
- Actual start and implementation questions.
- Claude interval/RNG rulings, 15:43:44. This adds source bootstrap 1407 to the earlier Drive seed table.
- Released O2 panel/shortfall parent
a86a74cbd943e0687ee69fb1cccdc003250113cc; original draws94011fd8efe0512e6579c7c541ccd585dc710c2e. No numerical producer, original draw, field or eligibility mask was modified. - Current M2 source snapshot and both Slack rulings are included verbatim. Historical sections of that document retain earlier definitions; its v1.2 section and the later rulings take precedence.
What to review
s1_scoring.py is the pure numerical API. score_fold(training, test) returns all node scores, neighbors, rank maps, actual weights, permutation/clipping diagnostics, per-part ratios and paired query replicates. summarize_folds reports all arms and contrasts, each seed and the mean of all five seeds. s1_scoring_io.py reads frozen sampled outputs and converts NA to JSON null. There is deliberately no real-panel launch command in this code-review packet: the future supervised launcher must pin the released input manifests, verify the current process release, fix wall/RSS limits and persist each complete fold before moving on. This is an execution prerequisite, not another information-sharing approval.
Coordinates and ranks
Columns are [d/t_part, gate_distance/R_g, g12, g23, gate_q1, gate_q2, edge_mean_q1, edge_max_q1, edge_mean_q2, edge_max_q2].
| Arm | Dimensions | Training modification |
|---|---|---|
| A | 2 | none |
| A+S | 6 | permute columns 2:6 as a single block within each part |
| B | 6 | none |
| B+edge_shuffle | 10 | permute columns 6:10 as a single block within each part |
| C | 10 | none |
| B+stratified_edge_shuffle | 10 | jointly permute gap-normalized edge block within exact positive-gap strata; multiply receiver gaps |
Queries always retain their true descriptors. Each fold fits ten true-column midrank maps once. Eligible node weight is c/(P*K_pE); there is no second mass factor. Exact equal values have a common midrank, constant columns remain in the denominator and take 0.5 at their training value. Strict tails map to 0/1. Query audits include both slices, weighted draw-slot tail fractions and raw ranges. Reconstructed control references receive per-part tail audits against the same frozen maps. Shuffled values are never refitted.
Distances and identity
Each distinct eligible physical node contributes exactly one reference slot, even when its draw multiplicity exceeds one. Different physical nodes with equal descriptors remain distinct. Physical key is (source_key, case, stable anchor node ID); raw-node identity is checked against the draw file by the adapter. Training parts are sorted by (source,case) and nodes retain ascending anchor ID order. One representative per source is enforced; test source leakage fails.
Distance is sqrt(mean((rank(query)-rank(reference))**2)). Node M2 is the mean of eight nearest distances. Exact cKDTree Euclidean search finds the eighth radius; a second radius query with nextafter(d8*(1+64*eps), +inf) gathers boundary candidates, then direct RMS distances and physical IDs determine the final eight. This retrieval padding does not round distances into ties or alter them. Independent brute-force tests include repeated descriptor ties in 2, 6 and 10 dimensions. A tie-heavy dataset can expand to all references; the cost plan includes this bound. Zero-distance counts and neighbor-source concentration are retained per query.
Random numbers
All generators are PCG64 with NumPy SeedSequence; NumPy 1.26.4 is the replay environment. Different numerical library versions are not claimed bitwise identical.
- Ordinary permutations:
[2026111401..2026111405, prefix], fresh generator per part/seed. A+S and plain edge shuffle share the same row index. Mapping is reproducibly regenerated from the same sorted eligible nodes across folds. - Normalized shuffle: a separate fresh generator initialized with the same part/seed. Strata order
(0,0),(0,1),(1,0),(1,1)uses exactgap>0. Only strata of size at least eight consume a permutation. Smaller strata keep exact original raw edge blocks. Zero-gap normalized channels are zero. No rejection, clipping, repair or redraw. Audits report exact violations and diagnostics above tolerance1e-10, plus donor modes, index changes, value changes and denominators. - Query bootstrap:
[2026111406,prefix], 200 sequential replicates; each draws 2,000 original draw-slot indices on the real panel. Six arms, slices and five seeds share the resulting counts. Synthetic tests use smaller explicitly generated draw lists. No new nearest-neighbor queries. - Source bootstrap:
[2026111407], 2,000 sequential resamples of the 24 lexicographically sorted source keys with replacement. One shared index matrix for all arms/contrasts/slices/seeds. No prefix for this source-level generator.
synthetic_seed_vectors.json contains complete small permutations, query/source first-three vectors and digests over dtype, shape and full bytes. This is the targeted independent replay material for Grok, together with the hand-ranked example below.
Ratios, intervals and interpretation
Per-part whole/near scores use sum(c*H*score)/sum(c*H), selecting accepted entries before multiplication. Zero denominator is NA. Near-wall only restricts queries to 0<d/t<=0.25; the reference pool is unchanged. Coverage reports accepted fraction among positive-distance draws, its product with the full-part positive-mass fraction, and the unscored boundary share. The near fraction of eligible draws has a separate denominator.
Aggregate within source with equal part weight, then across sources with equal weight; also report part-equal totals. Required missing parts make the fixed-source aggregate NA, never silently renormalize. Fewer than eight total distinct references produces missing scores; a real shortfall fold returns its noncomparable status and is not aggregated. The small-fixture gate bypass rejects parts tagged with real provenance. Shortfall uses strict <500, <1000 and integer 5*count_below_1000 > P.
Query SE is the valid replicate sample standard deviation (ddof=1), without division by sqrt(200). Both intervals use two-sided 95% percentile quantiles with linear interpolation. Original point NA or fewer than two valid replicates suppresses its interval; valid/total replicate counts are recorded. Pair differences are formed inside each replicate. The five-seed mean is taken before interval construction, with all five seed results retained.
For each primary contrast, the source interval strictly below zero means ranks ahead, strictly above zero ranks behind, otherwise no ranking difference; touching zero is not a win. No effect-size margin. Primary contrasts are B−(A+S) and C−(B+edge_shuffle); C−(B+stratified_edge_shuffle) is secondary, B−A and C−B descriptive. No joint scientific success claim is inferred from one contrast. All arms' scores and both levels of interval are available, but cross-dimension differences do not establish improved geometry information.
Source intervals resample the fixed LOPO source statistics; they do not refit overlapping training folds, ranks, or reference pools. Query intervals condition on these same realized objects and measure draw-slot variation only. Neither interval validates filling accuracy, physical bias, remeshing robustness or model generalization.
Hand expectations and tests
The tests write expected values independently of the scorer:
- Three-part midrank fixture: eligible weights
[1/9,2/9],[1/6,1/6],[1/4,1/12]. Masses at values 0,1,2 are 5/18,17/36,1/4. Ranks at 0,1,2 are 5/36,37/72,7/8; rank at unseen 1.5 is 3/4. - Three training parts each have ten distinct physical nodes at identical ten descriptor levels across columns. Eight-neighbor endpoint score is
(3*0+3*.1+2*.2)/8 = 7/80; interior is(3*0+5*.1)/8 = 1/16. Ten test nodes with multiplicities[1,2,1,1,1,1,1,1,1,1]give whole 59/880, first-three near slice 11/160. - Part statistics 1,3,10 from sources a,a,b give source-equal 6, part-equal 14/3. Replicates 1,2,3 give SE 1, interval [1.05,2.95]. Hand-chosen paired source resamples produce [-2.925,-1.075].
- Additional checks cover constant-column
a -> a/sqrt(2), all-arm rigid motion with the gate (plus an unmoved-gate negative control), seed byte replay, ordinary block tuples, normalized zero/positive strata and cross-channel order, strict shortfall endpoints, distinct node identity, NA propagation, source leakage, input hashes and JSON null serialization.
Initial test correction is disclosed: an assertion that sqrt(a*a/2) and a/sqrt(2) are bitwise equal failed by one FP64 ULP (0.21213203435596426 versus 0.21213203435596423). The test now checks exact agreement with the defined RMS evaluation and agreement with the algebraic alternative within relative 2e-16. No scorer formula was changed for this failure.
Run synthetic verification from the packet directory with Python 3.10, NumPy 1.26.4, SciPy 1.15.3:
OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 python test_s1_scoring.py
python s1_seed_vectors.py replay.json
python plan_s1_queries.py --summary summary.csv --folds fold_table.csv --out local-plan
Cost and next execution boundary
Every fold evaluates 18 configurations: A/B/C once plus three shuffled controls times five seeds. This is one 2D, six 6D and eleven 10D configurations. All true baselines are reused across seeds. Near-wall slices reuse query scores; neither 200 query replicates nor 2,000 source replicates add kNN calls.
Across 24 folds there are 43,656 eligible unique test nodes, 785,808 logical 8NN queries, 785,808 additional radius queries, and 432 tree builds. query_cost.csv lists every fold. Worst-case radius tie expansion totals 32,868,059,868 candidate distance evaluations; it is a conservative bound, not a measured workload. Matrix byte counts in the table are component sizes, not an RSS guarantee. No scoring-runtime extrapolation is claimed from the earlier O2 integrations.
Claude is the full reviewer. Grok's scope is independent seed replay and weighted-midrank formula checking. Real S1 execution remains pending this review and an explicit supervised execution scope with wall/RSS limits. Information-sharing authorization is already present and does not need to be requested again.