12 KiB
MeltMind project collaboration
-
Before moving on from a major result or entering a new stage, follow
docs/collaboration-stage-review-policy.md: consolidate evidence across the team, revisit the user's research goal and prior work, discuss disagreements, and record the next-stage decision. -
Continue the existing shared design and its fixed commits. Check for an existing draft or running task before starting another. Claude integrates and reviews; MeltAstra supplies execution evidence; MoldAstra supplies mathematical/grouping boundaries; Grok performs only needed targeted checks.
-
Keep topology/local-type partitioning and true connectivity as independent open hypotheses. A failed input-coverage mechanism is not a failure of all partition or topology routes.
-
Design/document review and information sharing do not authorize new scientific computation or training. Carry out the scope already authorized; record explicit scope and budgets before expanding it.
-
The user has authorized project information sharing in their controlled Drive, Git and Slack spaces; see
docs/sharing-authorization-2026-11-14.md. Do not ask for the same sharing authorization again. Git identity discussion is not a priority. -
Preserve original data, failures, review limitations and immutable evidence. Do not label proposals as executed or reviewed results.
-
Current E1 scope follows the standing continue-work authorization relayed in Slack
1794651141.035699and Claude caps1794651169.320839: bounded CPU-only label-free preparation and synthetic checks may proceed after the count checkpoint, without another user approval loop. Total 16 CPU-hours / 25 GiB, at most 8 processes, per-job 30 min / 1.25 GiB tree RSS, at most 2000 registration pairs; the 10-part stage additionally stays sequential within 3 h / 5 GiB. Never read labels, fit models, use GPU, run pilot/full matrix, or exceed caps in this scope. Seedocs/s2-e1-budget-checkpoint-2026-11-14.md. -
Post-witness update (2026-11-14 21:12) supersedes treating the preceding E1-only scope as the entire remaining project scope. Follow the latest section of
docs/collaboration-stage-review-policy.mdand Claude Slack1794661944.308269: custody join, label-free copy calibration, k=3 per-route candidate count, bounded registration (up to 4000 pairs / 100 worker CPU-h / 16 workers), band review, component freeze, then coordinated S2 continuation. No repeated approval gates or reruns of completed review; preserve concrete execution limits and distinguish future stages from completed work. User preference: tens of CPU-hours alone are not a reason to block progress. No additional scientific task is implied by this preference. -
S2 execution update (2026-11-15): components/splits and the fixed ten-part measurement are complete; ten-part 10/10 PASS is in
docs/s2-ten-part-results-2026-11-15.md. The 499 non-test preparation queue has started. Readdocs/s2-pilot-live-handoff-2026-11-15.mdand existing receipts before any launch; never restart from chat idle. Claude accepted implementation fixes at1794683016.134929, accepted the frozen ID-order dispatch at1794683049.057259, and raised the first-50 continuation ruling to 80 worker CPU-h / 100 GiB at1794682712.505709. The running original supervisor still enforces its pinned 60-hour cap; any required continuation may contain only never-attempted parts and must preserve cumulative accounting. After all 499 terminal attempts and required evidence, the authorized single-GPU pilot includes an E256 timing probe and four runs within 2 GPU-hours total, per-node train/validation extrapolation and a frozen common epoch limit. Keep 125 test labels sealed; no repeated user-go or main-review round is required for these accepted fixes. This does not authorize the full training matrix. -
Infrastructure-recovery update (2026-11-15 05:16 UTC+8) supersedes the original live sessions above: original preparation16682 exited INFRA_ABORTED and idle waiter13519 is retired; never restart either or manufacture original batch completion. Active recovery82149 and chain42022 use
artifacts/s2-pilot-recovery-inputs-2026-11-15andartifacts/s2-pilot-recovery-continuation-2026-11-15. Read the leading section ofdocs/s2-pilot-live-handoff-2026-11-15.md. Fixed134 =16 explicitly authorized infrastructure recoveries +118 never attempted; original362 PASS and3 real timeouts are retained. Cumulative60 worker CPU-h/100GiB includes old aborted CPU once; no80h extension, no new first50 barrier or automatic retry. Preserve active source pins. Three-source verification precedes the existing bounded probe/four-run pilot; no main matrix. -
Terminal update (2026-11-15 06:14 UTC+8): recovery82149 and chain42022 have finished. Recovery134=131 PASS+3 wall_limit; canonical499=493 PASS+6 wall_limit. E256 probe passed; N128-seed0 failed at epoch10 with nonfinite encoder output. Pilot stopped without retry/width selection after296.023387s of7200s; other three runs never started. Read
docs/s2-recovery-pilot-stage-result-2026-11-15.md; preserve all evidence,125-test seal,and source pins. Do not restart from stale progress. Claude received stop notice1794694508.930959; next changes require the existing collaboration review, not a repeat sharing approval. -
Precision-v2 terminal correction (2026-11-15 06:30 UTC+8): diagnostic41846 completed and reproduced fp16 slice overflow; fixed8a548cc preserves original sources and implements the approved shared FP32 aggregation. New v2 session26792 has already exited: probe PASS at187345 nodes, VRAM12586057728B, E_max42, but SELECTED_GPU_BUSY_BEFORE_RUN stopped before any training run (runs=[]), total11.143491s. Earlier inference from pilot-started to N128 running was incorrect and retracted. Do not restart or rerun probe without the engineering continuation ruling; read latest handoff and raw complete receipts.125 tests remain sealed.
-
Latest engineering continuation (2026-11-15): Claude1794695421.177269 authorized reuse of v2 probe after its zero-run infrastructure stop. Fixedea5ac05; sole active session34632/root
artifacts/s2-pilot-v2-continuation-2026-11-15, entryPID209380 and N128 workerPID209415 verified running. Three idle polls passed over20.285703s, sameGPU. ReuseE_max42/probe; original11.143491s plus all new waits/setup/runs share7200s. Keep v2 zero-run stop and original numerical failure immutable. Inspect real worker/start/progress/terminal receipts before any action; never duplicate or modify pinned code.125 tests remain sealed. -
Latest terminal update (2026-11-15 07:14:59 UTC+8): session34632 and all pilot workers have exited. N128/E128 completed42 epochs with guardPASS; N256 failed at epoch2 on4/382 AMP skips exceeding1%, E256 never started. PILOT_INCOMPLETE/no width; cumulative2439.912972s/7200. N256 history/progress also records reserved16416505856B above15GiB; the failed-result path did not inspect recorded peaks. Preserve this resource-boundary gap. Read
docs/s2-v2-pilot-stop-results-2026-11-15.md; no restart, reprobe, threshold/source change or full matrix is authorized by a polling request.125 tests sealed; evidence submitted toClaude. -
Claude1794698179.161599 records a separate feasibility decision: main-matrix width128, width effect not tested, no use of validation scores to justify selection. Original pilot selected_global_width staysnull. Before any main-matrix execution: fix recorded progress/result VRAM-peak enforcement with synthetic test, allocator configuration per ruling, unchangedAMP1% andE_max42, and budget/execution plan. This read-only stage audit has not implemented or launched that work.
-
Main engineering plan update: independent
s2_main_memory_guard.pychecks all progress/result/history peak records during and after failed workers;7 CPU synthetic checks passed. No old pinned source changed. Allocatorexpandable_segments parsedtrue withV100 VMM support, no GPU tensors/workload. Claude1794698734.967459 supersedes pilot-cell reuse: separately prepare6 timeouts at60min and125 fold0 test geometries without label decoding, freeze one final roster, then all90 neural fits fresh; pilot weights remain historical only. Readdocs/s2-main-engineering-and-budget-2026-11-15.md. Proposed costs/caps are plans; no131 queue or main matrix launched in this engineering turn. -
Latest actual launch (2026-11-15 07:51 UTC+8): Claude1794699027.366469 approved budgets/execution. Sole active131 preparation is session32694/coordinatorPID220180/root
artifacts/s2-main-preparation-2026-11-15, fixedimplementationbf5203c, contract0a3c628c81c008eab53e9337406b84ec2063edcd49529bcc7a82e3e4a3d719a3. Host verified16guards+16geometryworkers. Do not restart or mutate pinned sources. 493successes reuse only; all131 label-free;6×60min+125×30min/1.25GiB;16workers;new20workerCPU-h with old34.75965h separate; cumulative100GiB.10syntheticchecksPASS. Automatic roster freeze only after131terminal withoutglobalstop. Then90fresh width128/Emax42 with46GPUh includingidle/setup,30min/run,up to4V100,N/Eadjacent,first8projection;R2singlefit30CPUmin,total100CPUh,leaveGPUfeedersfree. Currentmatrix/R2notstarted;all90+15terminal/checkpointsfrozen beforeunifiedcustodypredictions/scoring. Read leading handoff and actual-start-audit; older pending/running entries are historical. -
Latest terminal (09:25): preparation32694/PID220180 exited,131=124PASS+7wall_limit; no globalstop,monitor_errors0. Final617 roster SHA fb0c984f2e553343a598ce2e3c7012d3017053dc0fef58b5aaf46dd3c67a207e automatically frozen;7globallyexcluded,493oldsuccesses reused. Newworker13.8436103492h/20h; cumulativehistory48.6032611436h. Do not restart preparation. Read final-roster/roster-receipt/terminal-audit/final-fold-pool-counts in main-preparation root. Matrix implementation and synthetic acceptance now proceeding under Claude1794699027; no training launch yet.
-
Latest matrix launch (09:36,09:37 actual worker evidence): sole session92182/coordinatorPID223006/root artifacts/s2-main-matrix-2026-11-15; implementatione3fa0af, contract198e5711ed2e7485167068769652c866867229e09ecf0d09b21612debc3b7a56,617rosterfb0c984f. Four V100 training workers223127/223141/223131/223145 observed epoch1; initialN f0pool25seed0/1/2 and f0pool50seed0, each followed byE.17syntheticchecksPASS. First4blocks/8runs barrier and46GPUhprojection automatic; allidle/setup/failure charged,30min/run,no retries,pins immutable. CPUaffinity0-7 GPUfeeders;8-31availableforR2. R2 notyetimplemented/started; all90+15terminal/checkpointsfrozen beforecustody. Check actual-start-audit and live receipts; do not duplicate matrix or restart completed131 preparation.
-
Latest R2 launch: implementationf560a3d;10newtestsPASS; soleR2session51525/coordinatorPID237829/root artifacts/s2-r2-main-2026-11-15; contractedec86197769f44bf98f234795950f36c9fa42084ccb3c68448b63b440aa126e. FirstHPOguardPASS worker21.929337CPU-s/fit7.789711311CPU-s; first-fitprojection18685.543199s<=360000 permits automatic4worker continuation onCPU8/10/12/14,coord30-31. Claude1794707190: finalrefit samelearningonly; failedHPOconfigsNA,choosebestfinished,fixedgridtie; all12fail orfinalfail=>cellNA. Max180HPO+15final,1800CPUseconds/fit,total100workerCPUh,no retries. Neural92182/PID223006 stillactive,sourcepinsunchanged; first8 gatepending as oflaunch. Do not duplicate either queue or restart preparation. Bothallterminal/checkpointsfrozen beforecustody/testpredictions/scoring. Read livehandoff and R2actual-start-audit.
-
Latest first-eight milestone (10:04): same neural92182/PID223006 first8 all42epochs/PASS,4pairs valid, checkpoints hashed; automatic budget checkpointpassed, projected83657.259515GPU-s=23.238128h<=46h, chargedfirst8 4813.381680s. Existingqueue continuedblocks4-7; no newlaunch. R2same51525/PID237829 had48HPO PASS/4active withnoerrors. Read first-eight-audit/checkpoint and liveprogress, retainall90+15terminal custodygate.