65 lines
21 KiB
Markdown
65 lines
21 KiB
Markdown
# MeltMind project collaboration
|
||
|
||
- Before moving on from a major result or entering a new stage, follow `docs/collaboration-stage-review-policy.md`: consolidate evidence across the team, revisit the user's research goal and prior work, discuss disagreements, and record the next-stage decision.
|
||
- Continue the existing shared design and its fixed commits. Check for an existing draft or running task before starting another. Claude integrates and reviews; MeltAstra supplies execution evidence; MoldAstra supplies mathematical/grouping boundaries; Grok performs only needed targeted checks.
|
||
- Keep topology/local-type partitioning and true connectivity as independent open hypotheses. A failed input-coverage mechanism is not a failure of all partition or topology routes.
|
||
- Design/document review and information sharing do not authorize new scientific computation or training. Carry out the scope already authorized; record explicit scope and budgets before expanding it.
|
||
- The user has authorized project information sharing in their controlled Drive, Git and Slack spaces; see `docs/sharing-authorization-2026-11-14.md`. Do not ask for the same sharing authorization again. Git identity discussion is not a priority.
|
||
- Preserve original data, failures, review limitations and immutable evidence. Do not label proposals as executed or reviewed results.
|
||
|
||
- Current E1 scope follows the standing continue-work authorization relayed in Slack `1794651141.035699` and Claude caps `1794651169.320839`: bounded CPU-only label-free preparation and synthetic checks may proceed after the count checkpoint, without another user approval loop. Total 16 CPU-hours / 25 GiB, at most 8 processes, per-job 30 min / 1.25 GiB tree RSS, at most 2000 registration pairs; the 10-part stage additionally stays sequential within 3 h / 5 GiB. Never read labels, fit models, use GPU, run pilot/full matrix, or exceed caps in this scope. See `docs/s2-e1-budget-checkpoint-2026-11-14.md`.
|
||
|
||
- Post-witness update (2026-11-14 21:12) supersedes treating the preceding E1-only scope as the entire remaining project scope. Follow the latest section of `docs/collaboration-stage-review-policy.md` and Claude Slack `1794661944.308269`: custody join, label-free copy calibration, k=3 per-route candidate count, bounded registration (up to 4000 pairs / 100 worker CPU-h / 16 workers), band review, component freeze, then coordinated S2 continuation. No repeated approval gates or reruns of completed review; preserve concrete execution limits and distinguish future stages from completed work. User preference: tens of CPU-hours alone are not a reason to block progress. No additional scientific task is implied by this preference.
|
||
|
||
- S2 execution update (2026-11-15): components/splits and the fixed ten-part measurement are complete; ten-part 10/10 PASS is in `docs/s2-ten-part-results-2026-11-15.md`. The 499 non-test preparation queue has started. Read `docs/s2-pilot-live-handoff-2026-11-15.md` and existing receipts before any launch; never restart from chat idle. Claude accepted implementation fixes at `1794683016.134929`, accepted the frozen ID-order dispatch at `1794683049.057259`, and raised the first-50 continuation ruling to 80 worker CPU-h / 100 GiB at `1794682712.505709`. The running original supervisor still enforces its pinned 60-hour cap; any required continuation may contain only never-attempted parts and must preserve cumulative accounting. After all 499 terminal attempts and required evidence, the authorized single-GPU pilot includes an E256 timing probe and four runs within 2 GPU-hours total, per-node train/validation extrapolation and a frozen common epoch limit. Keep 125 test labels sealed; no repeated user-go or main-review round is required for these accepted fixes. This does not authorize the full training matrix.
|
||
|
||
- Infrastructure-recovery update (2026-11-15 05:16 UTC+8) supersedes the original live sessions above: original preparation16682 exited INFRA_ABORTED and idle waiter13519 is retired; never restart either or manufacture original batch completion. Active recovery82149 and chain42022 use `artifacts/s2-pilot-recovery-inputs-2026-11-15` and `artifacts/s2-pilot-recovery-continuation-2026-11-15`. Read the leading section of `docs/s2-pilot-live-handoff-2026-11-15.md`. Fixed134 =16 explicitly authorized infrastructure recoveries +118 never attempted; original362 PASS and3 real timeouts are retained. Cumulative60 worker CPU-h/100GiB includes old aborted CPU once; no80h extension, no new first50 barrier or automatic retry. Preserve active source pins. Three-source verification precedes the existing bounded probe/four-run pilot; no main matrix.
|
||
|
||
- Terminal update (2026-11-15 06:14 UTC+8): recovery82149 and chain42022 have finished. Recovery134=131 PASS+3 wall_limit; canonical499=493 PASS+6 wall_limit. E256 probe passed; N128-seed0 failed at epoch10 with nonfinite encoder output. Pilot stopped without retry/width selection after296.023387s of7200s; other three runs never started. Read `docs/s2-recovery-pilot-stage-result-2026-11-15.md`; preserve all evidence,125-test seal,and source pins. Do not restart from stale progress. Claude received stop notice1794694508.930959; next changes require the existing collaboration review, not a repeat sharing approval.
|
||
|
||
- Precision-v2 terminal correction (2026-11-15 06:30 UTC+8): diagnostic41846 completed and reproduced fp16 slice overflow; fixed8a548cc preserves original sources and implements the approved shared FP32 aggregation. New v2 session26792 has already exited: probe PASS at187345 nodes, VRAM12586057728B, E_max42, but SELECTED_GPU_BUSY_BEFORE_RUN stopped before any training run (runs=[]), total11.143491s. Earlier inference from pilot-started to N128 running was incorrect and retracted. Do not restart or rerun probe without the engineering continuation ruling; read latest handoff and raw complete receipts.125 tests remain sealed.
|
||
|
||
- Latest engineering continuation (2026-11-15): Claude1794695421.177269 authorized reuse of v2 probe after its zero-run infrastructure stop. Fixedea5ac05; sole active session34632/root `artifacts/s2-pilot-v2-continuation-2026-11-15`, entryPID209380 and N128 workerPID209415 verified running. Three idle polls passed over20.285703s, sameGPU. ReuseE_max42/probe; original11.143491s plus all new waits/setup/runs share7200s. Keep v2 zero-run stop and original numerical failure immutable. Inspect real worker/start/progress/terminal receipts before any action; never duplicate or modify pinned code.125 tests remain sealed.
|
||
|
||
- Latest terminal update (2026-11-15 07:14:59 UTC+8): session34632 and all pilot workers have exited. N128/E128 completed42 epochs with guardPASS; N256 failed at epoch2 on4/382 AMP skips exceeding1%, E256 never started. PILOT_INCOMPLETE/no width; cumulative2439.912972s/7200. N256 history/progress also records reserved16416505856B above15GiB; the failed-result path did not inspect recorded peaks. Preserve this resource-boundary gap. Read `docs/s2-v2-pilot-stop-results-2026-11-15.md`; no restart, reprobe, threshold/source change or full matrix is authorized by a polling request.125 tests sealed; evidence submitted toClaude.
|
||
|
||
- Claude1794698179.161599 records a separate feasibility decision: main-matrix width128, width effect not tested, no use of validation scores to justify selection. Original pilot selected_global_width staysnull. Before any main-matrix execution: fix recorded progress/result VRAM-peak enforcement with synthetic test, allocator configuration per ruling, unchangedAMP1% andE_max42, and budget/execution plan. This read-only stage audit has not implemented or launched that work.
|
||
|
||
- Main engineering plan update: independent `s2_main_memory_guard.py` checks all progress/result/history peak records during and after failed workers;7 CPU synthetic checks passed. No old pinned source changed. Allocatorexpandable_segments parsedtrue withV100 VMM support, no GPU tensors/workload. Claude1794698734.967459 supersedes pilot-cell reuse: separately prepare6 timeouts at60min and125 fold0 test geometries without label decoding, freeze one final roster, then all90 neural fits fresh; pilot weights remain historical only. Read `docs/s2-main-engineering-and-budget-2026-11-15.md`. Proposed costs/caps are plans; no131 queue or main matrix launched in this engineering turn.
|
||
|
||
- Latest actual launch (2026-11-15 07:51 UTC+8): Claude1794699027.366469 approved budgets/execution. Sole active131 preparation is session32694/coordinatorPID220180/root `artifacts/s2-main-preparation-2026-11-15`, fixedimplementationbf5203c, contract0a3c628c81c008eab53e9337406b84ec2063edcd49529bcc7a82e3e4a3d719a3. Host verified16guards+16geometryworkers. Do not restart or mutate pinned sources. 493successes reuse only; all131 label-free;6×60min+125×30min/1.25GiB;16workers;new20workerCPU-h with old34.75965h separate; cumulative100GiB.10syntheticchecksPASS. Automatic roster freeze only after131terminal withoutglobalstop. Then90fresh width128/Emax42 with46GPUh includingidle/setup,30min/run,up to4V100,N/Eadjacent,first8projection;R2singlefit30CPUmin,total100CPUh,leaveGPUfeedersfree. Currentmatrix/R2notstarted;all90+15terminal/checkpointsfrozen beforeunifiedcustodypredictions/scoring. Read leading handoff and actual-start-audit; older pending/running entries are historical.
|
||
|
||
- Latest terminal (09:25): preparation32694/PID220180 exited,131=124PASS+7wall_limit; no globalstop,monitor_errors0. Final617 roster SHA fb0c984f2e553343a598ce2e3c7012d3017053dc0fef58b5aaf46dd3c67a207e automatically frozen;7globallyexcluded,493oldsuccesses reused. Newworker13.8436103492h/20h; cumulativehistory48.6032611436h. Do not restart preparation. Read final-roster/roster-receipt/terminal-audit/final-fold-pool-counts in main-preparation root. Matrix implementation and synthetic acceptance now proceeding under Claude1794699027; no training launch yet.
|
||
|
||
- Latest matrix launch (09:36,09:37 actual worker evidence): sole session92182/coordinatorPID223006/root artifacts/s2-main-matrix-2026-11-15; implementatione3fa0af, contract198e5711ed2e7485167068769652c866867229e09ecf0d09b21612debc3b7a56,617rosterfb0c984f. Four V100 training workers223127/223141/223131/223145 observed epoch1; initialN f0pool25seed0/1/2 and f0pool50seed0, each followed byE.17syntheticchecksPASS. First4blocks/8runs barrier and46GPUhprojection automatic; allidle/setup/failure charged,30min/run,no retries,pins immutable. CPUaffinity0-7 GPUfeeders;8-31availableforR2. R2 notyetimplemented/started; all90+15terminal/checkpointsfrozen beforecustody. Check actual-start-audit and live receipts; do not duplicate matrix or restart completed131 preparation.
|
||
|
||
- Latest R2 launch: implementationf560a3d;10newtestsPASS; soleR2session51525/coordinatorPID237829/root artifacts/s2-r2-main-2026-11-15; contractedec86197769f44bf98f234795950f36c9fa42084ccb3c68448b63b440aa126e. FirstHPOguardPASS worker21.929337CPU-s/fit7.789711311CPU-s; first-fitprojection18685.543199s<=360000 permits automatic4worker continuation onCPU8/10/12/14,coord30-31. Claude1794707190: finalrefit samelearningonly; failedHPOconfigsNA,choosebestfinished,fixedgridtie; all12fail orfinalfail=>cellNA. Max180HPO+15final,1800CPUseconds/fit,total100workerCPUh,no retries. Neural92182/PID223006 stillactive,sourcepinsunchanged; first8 gatepending as oflaunch. Do not duplicate either queue or restart preparation. Bothallterminal/checkpointsfrozen beforecustody/testpredictions/scoring. Read livehandoff and R2actual-start-audit.
|
||
|
||
- Latest first-eight milestone (10:04): same neural92182/PID223006 first8 all42epochs/PASS,4pairs valid, checkpoints hashed; automatic budget checkpointpassed, projected83657.259515GPU-s=23.238128h<=46h, chargedfirst8 4813.381680s. Existingqueue continuedblocks4-7; no newlaunch. R2same51525/PID237829 had48HPO PASS/4active withnoerrors. Read first-eight-audit/checkpoint and liveprogress, retainall90+15terminal custodygate.
|
||
|
||
- Latest custody launch (16:13): training all terminal frozen6903c07 (89neuralPASS/1NA,15R2finalPASS). Claude1794730198 main-review conditions closed in522ee84;14testsPASS/617columnmetadataPASS. Sole prediction session42606/coordinator455501/worker455512, root artifacts/s2-custody-evaluation-2026-11-15, contracte8b081b31d0f8975da1c2f1e92843181ee788979d98e61d2a8571e5e5d94850c. Firstfold3/121 observed, GPU0active; others wait firstfoldcost. No labels decoded. Follow leading handoff; never restart oldtraining or duplicate prediction. All predictions hashed then scoring code/config Git freeze before custody. Approved4GPUh/45minfold/15GiBVRAM/16GiBRSS/20GiBfold, custody10CPUh/3h/16GiB,100GiBtotal,0retries.
|
||
|
||
- Latest custody COMPLETE (2026-11-15,终态核验): prediction42606 and custody87444 exited0, allguardsPASS.13449predictions frozen+codelocked c47dd6a before one decode of617parts; results1dea89d/report docs/s2-custody-results-2026-11-15.md. Main100pct209components/617parts: Delta1-E/N=-0.01292367747,95CI[-0.02122332192,-0.00563340237],harmful;R2bestprimaryerror,N usefulness vsR2 fails. Allnewmetricsvalid,undefinedbootstrap0. Prediction410.070561GPU-s,custody72.962534CPU-s. Do not restart/redecode/retrain/change frozen stats; all original NA preserved. Next team stage review ledClaude; no O1/organisation inference from this result. Read newest handoff, older running entries historical.
|
||
|
||
- Latest posthoc D1-D4 complete (2026-11-15): S2 acceptedClaude1794731320, stage47259fee/correctiond2d69cac. Authorized read-only diagnostics fixeddaed8e873afa7bf56fd3a58f6701f78ce566c5b0; report docs/s2-posthoc-diagnostics-2026-11-15.md; completion.json is latest. D1242.573114CPU-s,D317.133535CPU-s,13449D3rows,0undefined,2lowpoolE signed-oracle reversals retained; no newlabels/model/training/prediction change. S3-A/B not launched; require stage discussion and explicit execution contract. Do not replay earlier queues or treat oracle as deployable calibration.
|
||
|
||
- Latest S3-A preparation (2026-11-15): Claude1794732043 approved direction, requires one main review before running. Contract263b0b42b832d958ccf5843215403d629cf9d037084951d10f2af40bb89348d8 at artifacts/s3a-planning-2026-11-15;17syntheticchecksPASS. NewR2_E4/R2_G2,360HPO+30final planned, old15R2fits/predictions reused. CPU20h including18hfit+2hpostfit,0retry,noGPU,newdecode0. Scoring/data/budget helpers frozen; actual queue implementation and launch pending main review. Mold1794732176 final D3 read accepted, no recomputation; H not pure ordering, D2 cannot decide longer training success/failure. S3-B not launched/O1open. Do not restart S2.
|
||
|
||
- Latest S3-A actual launch (2026-11-15 17:01 UTC+8): Claude1794732829.567629 approved implementation+launch, no further main review before results. Sole session5217/coordinatorPID458991/root artifacts/s3a-execution-2026-11-15; implementation6142ad6/contract263b0b42 unchanged. FirstE4job0+G2job12 PASS47.590461CPU-s; projection43360.732672s=12.044648h<=18h, then4guards+4workers CPU8/10/12/14 verified. Original15R2 no refit;390maxnewfits,20workerCPUh incl2hpostfit,0retry,noGPU/newdecode. 8newqueuechecksPASS. Read actual-start-audit/first-fit-cost-check/liveprogress; do not duplicate/restart/mutate sources. Queue stops at all-fit terminal freeze; predictions/scoring not yet started. Parallel stage/literature review does not expand compute scope.
|
||
|
||
- Latest naive-ordering launch (2026-11-15 17:23 UTC+8): Claude1794733395 permits one617-full-node Fill_time custody;1794733619 corrected Dw fallback alreadyB0, c/ties/alpha accepted, fixed contract then run with no further review. Sole49189/PID460410/root artifacts/naive-ordering-2026-11-15, d2adb6b/contractc49313e.13checksPASS. FirstpartPASS1.148115CPU-s/projection885.196665s<=8h,4workers16/18/20/22;10workerCPUh incl2hpost,0retry/noGPU/no newmodels. S3-A5217/PID458991 unchanged, workers8/10/12/14, sourcepinsverified; neverduplicate/restart either. Full617PASS before clocks/maps/score; originalVTU/Tet/querymax distinct, strictCDF withties, PPSmultiplicityonce/alphaestimate, eikonalnot tested. Checklive receipts and latesthandoff.
|
||
|
||
- Latest naive and LOG terminal update (2026-11-15): Naive ordering complete: original610PASS+7failures retained, seven cache-only continuations PASS; canonical617,23938maps frozen,score PASS. Worker cumulative836.697435206s/36000; no new fits or decodes in continuation. Original49189,continuation2411,map63422,score10304 all exited; do not rerun. Results9b7121a/docs/naive-ordering-results-2026-11-15.md. LOG92800 complete624/624,411within1pp,611zero-printed-flow undefined,13central-window printed-flow within5%; see docs/log-time-map-results-2026-11-15.md. Both retain complete raw evidence. S3-A original5217/PID458991 still active: 228 receipts/0 failures,progress228,4active,sourcepinsPASS,worker12282.190599s/64800fit cap;20h includes2hpostfit. No terminal freeze, prediction/scoring not started. Do not restart or modify running pins; after all390terminal follow existing approved postfit freeze/predict/score scope. Earlier two-queue running entries are historical.
|
||
|
||
- Latest S3-A terminal correction (2026-11-15): S3-A original5217/PID458991 has now exited1: S3A_STOPPED_INCOMPLETE/S3A_COORDINATOR_INTERRUPTED.235attempted=231PASS+4interrupted(230,232,233,234;returncode-9),155never-attempted235..389,all30finalnotrun. SourcepinsPASS,no live queue/fit processes,accounting_complete true,worker12507.336661CPU-s; original failures and attempts immutable. Signal origin unknown; handler merges SIGINT/SIGTERM, do not call this numerical failure. No restart/retry authorized by polling; Claude notified1794736881.943279 for infrastructure continuation ruling. Read artifacts/s3a-interruption-audit-2026-11-15/audit.json. Naive/LOG complete; prior S3-A running snapshots are historical.
|
||
|
||
- Latest S3-A infrastructure continuation actual launch (2026-11-15): S3-A authorized159 continuation now active: session7921/PID464772/root artifacts/s3a-recovery-2026-11-15, orchestration259d1d685bdced1920eb3d483019cd18fedc1bf5 (local fixed; remote publication queued behind087f63b full raw upload). Scientific contract263b0b42 and original6142ad6 workers unchanged. Claude1794736905 explicit4infra recoveries230/232/233/234+155never attempted; reuse231PASS, prior12507.336661CPU-s counted once within20h/18hfit+2hpostfit. Six scope/accounting/barrier testsPASS, actual sourcepinsPASS and4guards+4workers observed; old458991 absent. All360HPO terminal then30final,390canonical thenprediction/scoring. Original partial failures immutable; do not duplicate/restart or mutate either pinned implementation. Read fresh receipts, recovery-scope/code-freeze and actual-start-audit; older stopped/running entries are historical.
|
||
|
||
- Latest S3-A/naive handoff (2026-11-15): Sole S3-A recovery7921/PID464772 remains active: 287 canonical terminal (56 new receipts, 0 new failures), prior231PASS reused; cumulative15954.571443workerCPU-s includes old12507.336661 once. Source259d1d6, old scientific6142ad6/contract263b0b42 unchanged. Four explicit recovered jobs230/232/233/234 allPASS. Sole postfit chain81572/PID465797 waits at WAIT_EXISTING_RECOVERY; fixed58e4e4c,4newsyntheticchecksPASS. It starts nofits, waits390complete/workerexit,Gitfreezes models,predicts once,Gitfreezes exactmanifest,scores using original frozen scorer within cumulative20h and postfit2h/combinedwall3h; failure stops without retry. Original full2.2GB publication58603/PID464160 still uploading087f63b,followup91597 waits then publishes fixed4d76d35 afterremoteverification. Do not duplicate uploads or queues. Latest head may be locally fixed before remote publication; do not claim remote completion. SmallreviewZIPF0CHEAFDB4P is shared and received/reviewed byMold/Claude. Claude1794737961 withdrew Spearman request; no correlations/bootstrap were run. Read artifacts/s3a-current-handoff-2026-11-15.json and real receipts.
|
||
|
||
- Latest final-fit process-count policy and actual waiter replacement (2026-11-15): Claude1794738923.689619 final-fit process_count_limit policy is enforced by sole replacement postfit45375/PID466888/root artifacts/s3a-postfit-final-policy-chain-2026-11-15, fixed/pushedcb786589448447be1fba86baed9020b8b6c52456,4testsPASS. Old idle81572/PID465797 intentionally retired(exit143), no children or predictions at retirement; never restart it. Recovery7921/PID464772 unchanged, sourcesPASS; 364canonical terminal at audit. If anyFINAL process_count_limit occurs, new chain writes awaiting-final-infra-ruling and stops before checkpoint adoption/prediction/scoring; no auto recovery, raw worker/batch NA is not adopted as scientific cellNA. HPO323 remains unavailable config11, other11cell26configurations actualPASS with result/modelhash verified. Continue existing360HPO→30final→390freeze workflow under original budget when no final-policy hold. Existing raw087f63b/small4d76d35 publication complete; do not duplicate uploads. New final-result publication expects remote basecb786589.
|
||
|
||
- Latest S3-A terminal: 390logical terminal,359HPOPASS+1configNA+30finalPASS; sole45375 exited0 and result1e49c1572227116e821d3cd414ebcecd7c430753 published with independent remote readback. No restart/retry/re-score. See leading handoff and docs/s3a-results-2026-11-15.md. Mold1794740467 actual10JSON review and Claude1794740503 acceptance recorded. S3-B(a) remains a proposed <=10workerCPUh CPU-only/no-learning contract subject to one review; no automatic launch, no S3-B(b)/O1 training authorization.
|
||
|
||
- Latest contract-only S3-B(a) update: see docs/s3b-a-contract-2026-11-15.md. Claude1794740834 fixes both paired-CI gates, total-volume denominator and exact midpoint rule; Mold1794740985 clarifies zero wall edges/UNKNOWN failures. Fixed-ten cost checkpoint and possible20workerCPUh cap supersede early10h-only proposal, but contract awaits single main review and this task authorizes documents only. No implementation, tests or real workload launched. S3-A report delivered bd2473c with original results1e49c157 independently HTTP/hash verified.
|