docs: retain original eec4677 Phase 0 baseline failures

docs/validation/phase0_baseline_eec4677_20261020.txt

@@ -0,0 +1,149 @@
+CelerisLab Phase 0 baseline — original candidate SHA eec4677db1308e2a9ce4e5f195c994356c11de13
+106 isolated groups; 764 primary pytest test IDs; 741 passed, 22 failed, 1 skipped; 0 collection errors, 0 timeouts.
+Host: 595 passed / 20 failed / 1 skipped. GPU: 146 passed / 2 failed.
+Python 3.10.13; NumPy 1.26.4; pytest 9.1.1; PyCUDA 2026.1; SciPy 1.14.1; CUDA 12.4; Tesla V100-SXM2-16GB.
+CUDA_VISIBLE_DEVICES bound per GPU worker. Host mocks and real CUDA ran in separate processes; each group had a 900-second timeout.
+No source/test edits, no changed tolerance, no refreshed frozen hash, no long physics or performance runs.
+This report contains test IDs/errors only. Full logs remain on Horsea under /home/frank14f/CelerisLab-phase0-evidence-20261020/.
+11 source provenance failures bind historical experimental manifests (47b05bc8... for DFG/Jet CV; a233c903... window); they are not new physical qualification.
+The channel GPU bitwise discrepancy remains unresolved: 1/4320 values; maximum absolute difference 2.3283064e-10; relative 1.7676595e-07.
+Shifted-storage test input is FP64 because NumPy where(float32 scalar, np.nan) promotes it; library requires shifted raw FP32.
+Current scheduler uses _advance(index,...); two tests still instrument removed _advance_child.
+
+Failed IDs and original error excerpts:
+
+1. tests/unit/test_channel_root.py::test_actual_closure_read_conflicts
+Location: tests/unit/test_channel_root.py:53: in test_actual_closure_read_conflicts
+Error: E   ValueError: block_cell must be integer index(es) in [0, 960)
+Category: test_fixture_numpy_uint64_to_float64
+Log: run-004/stdout.log
+
+2. tests/unit/test_jet_circle_cv.py::Host::test_source_pin
+Location: tests/unit/test_jet_circle_cv.py:135: in test_source_pin
+Error: E   RuntimeError: Source hash mismatch: src/CelerisLab/lbm/channel.py
+Category: frozen_source_manifest_constraint
+Log: run-010/stdout.log
+
+3. tests/unit/test_re100.py::test_production_unmodified
+Location: tests/unit/test_re100.py:71: in test_production_unmodified
+Error: E   AssertionError: src/CelerisLab/lbm/channel.py | E   assert '40d0354e5dc9...f4cd75ab5a095' == '2d5b5ff43414...b58d498dd8b0f' | E      | E     - 2d5b5ff43414cb35633ac4ce90ce2c3a3353950b25c168e1c2ab58d498dd8b0f
+Category: frozen_source_manifest_constraint
+Log: run-027/stdout.log
+
+4. tests/unit/test_refined_dfg.py::test_source_pin
+Location: tests/unit/test_refined_dfg.py:20: in test_source_pin
+Error: E   RuntimeError: Source mismatch: src/CelerisLab/lbm/kernels/blocks.cu
+Category: frozen_source_manifest_constraint
+Log: run-028/stdout.log
+
+5. tests/unit/test_refined_dfg.py::test_full_requires_completed_pilot
+Location: tests/unit/test_refined_dfg.py:146: in test_full_requires_completed_pilot
+Error: E   RuntimeError: Source mismatch: src/CelerisLab/lbm/kernels/blocks.cu
+Category: frozen_source_manifest_constraint
+Log: run-028/stdout.log
+
+6. tests/unit/test_refined_dfg.py::test_pilot_validation_and_calibration
+Location: tests/unit/test_refined_dfg.py:226: in test_pilot_validation_and_calibration
+Error: E   RuntimeError: Source mismatch: src/CelerisLab/lbm/kernels/blocks.cu
+Category: frozen_source_manifest_constraint
+Log: run-028/stdout.log
+
+7. tests/unit/test_refined_dfg.py::test_reject_malformed_pilot_receipts
+Location: tests/unit/test_refined_dfg.py:236: in test_reject_malformed_pilot_receipts
+Error: E   RuntimeError: Source mismatch: src/CelerisLab/lbm/kernels/blocks.cu
+Category: frozen_source_manifest_constraint
+Log: run-028/stdout.log
+
+8. tests/unit/test_refined_dfg.py::test_reject_collision_receipts
+Location: tests/unit/test_refined_dfg.py:326: in test_reject_collision_receipts
+Error: E   RuntimeError: Source mismatch: src/CelerisLab/lbm/kernels/blocks.cu
+Category: frozen_source_manifest_constraint
+Log: run-028/stdout.log
+
+9. tests/unit/test_refined_dfg.py::test_record_actual_checkout_separately
+Location: tests/unit/test_refined_dfg.py:344: in test_record_actual_checkout_separately
+Error: E   RuntimeError: Source mismatch: src/CelerisLab/lbm/kernels/blocks.cu
+Category: frozen_source_manifest_constraint
+Log: run-028/stdout.log
+
+10. tests/unit/test_refined_dfg.py::test_window_plan_preserves_physics
+Location: tests/unit/test_refined_dfg.py:354: in test_window_plan_preserves_physics
+Error: E   RuntimeError: Source mismatch: src/CelerisLab/lbm/kernels/blocks.cu
+Category: frozen_source_manifest_constraint
+Log: run-028/stdout.log
+
+11. tests/unit/test_refined_dfg.py::test_reject_observation_mode_receipts
+Location: tests/unit/test_refined_dfg.py:365: in test_reject_observation_mode_receipts
+Error: E   RuntimeError: Source mismatch: src/CelerisLab/lbm/kernels/blocks.cu
+Category: frozen_source_manifest_constraint
+Log: run-028/stdout.log
+
+12. tests/unit/test_shifted_storage.py::ShiftedStorage::test_shifted_density_and_tag_validation
+Location: tests/unit/test_shifted_storage.py:82: in test_shifted_density_and_tag_validation
+Error: E   ValueError: shifted stored populations must be FP32
+Category: test_fixture_numpy_where_dtype
+Log: run-037/stdout.log
+
+13. tests/unit/test_validation_source_provenance.py::test_manifests_cover_current_source_and_preserve_origin
+Location: tests/unit/test_validation_source_provenance.py:40: in test_manifests_cover_current_source_and_preserve_origin
+Error: E   AssertionError: src/CelerisLab/lbm/channel.py | E   assert '40d0354e5dc9...f4cd75ab5a095' == '2d5b5ff43414...b58d498dd8b0f' | E      | E     - 2d5b5ff43414cb35633ac4ce90ce2c3a3353950b25c168e1c2ab58d498dd8b0f
+Category: frozen_source_manifest_constraint
+Log: run-049/stdout.log
+
+14. tests/validation/multigrid/test_block_stepper.py::CpuContracts::test_initialize_rejects_bad_state_before_upload
+Location: tests/validation/multigrid/test_block_stepper.py:212: in test_initialize_rejects_bad_state_before_upload
+Error: E   AttributeError: 'PeriodicBlockStepper' object has no attribute '_ddf_shifting'
+Category: test_fixture_or_assertion_stale_api
+Log: run-058/stdout.log
+
+15. tests/validation/multigrid/test_channel_root_gpu.py::ChannelRootGPU::test_uniform_matched_root
+Location: tests/validation/multigrid/test_channel_root_gpu.py:95: in test_uniform_matched_root
+Error: E   AssertionError:  | E   Arrays are not equal | E    | E   Mismatched elements: 1 / 4320 (0.0231%)
+Category: gpu_bitwise_parity_contract_failure_unresolved
+Log: run-060/stdout.log
+
+16. tests/validation/multigrid/test_fine_surface_gpu.py::test_missing_surface_device_buffer_rejects_before_launch[surface_velocity_gpu]
+Location: tests/validation/multigrid/test_fine_surface_gpu.py:284: in test_missing_surface_device_buffer_rejects_before_launch
+Error: E   AttributeError: can't set attribute 'fine_boundary'
+Category: test_fixture_or_assertion_stale_api
+Log: run-070/stdout.log
+
+17. tests/validation/multigrid/test_fine_surface_gpu.py::test_missing_surface_device_buffer_rejects_before_launch[link_exchange_gpu]
+Location: tests/validation/multigrid/test_fine_surface_gpu.py:284: in test_missing_surface_device_buffer_rejects_before_launch
+Error: E   AttributeError: can't set attribute 'fine_boundary'
+Category: test_fixture_or_assertion_stale_api
+Log: run-070/stdout.log
+
+18. tests/validation/multigrid/test_refinement_followup.py::FaultGuardsCPU::test_activation_failure_retains_resources_for_retry
+Location: tests/validation/multigrid/test_refinement_followup.py:134: in test_activation_failure_retains_resources_for_retry
+Error: E   AttributeError: can't set attribute 'fine_layout'
+Category: test_fixture_or_assertion_stale_api
+Log: run-083/stdout.log
+
+19. tests/validation/multigrid/test_refinement_followup.py::FaultGuardsCPU::test_failed_download_blocks_read_and_step
+Location: tests/validation/multigrid/test_refinement_followup.py:121: in test_failed_download_blocks_read_and_step
+Error: E   AttributeError: can't set attribute 'fine_layout'
+Category: test_fixture_or_assertion_stale_api
+Log: run-083/stdout.log
+
+20. tests/validation/multigrid/test_refinement_followup.py::FaultGuardsCPU::test_kernel_contracts_reject_before_upload
+Location: RuntimeError: level config chain is inconsistent with the pair tables
+Error: E   AssertionError: "omega/tau" does not match "level config chain is inconsistent with the pair tables"
+Category: test_fixture_or_assertion_stale_api
+Log: run-083/stdout.log
+
+21. tests/validation/multigrid/test_refinement_three_level.py::ThreeLevelCPU::test_four_halo_bfl_steps_and_fresh_middle_feeds
+Location: tests/validation/multigrid/test_refinement_three_level.py:83: in test_four_halo_bfl_steps_and_fresh_middle_feeds
+Error: E   AttributeError: 'FixedPatchStepper' object has no attribute '_advance_child'
+Category: test_instrumentation_stale_private_method
+Log: run-092/stdout.log
+
+22. tests/validation/multigrid/test_refinement_three_level.py::ThreeLevelGPU::test_nonequilibrium_masks_and_fresh_middle_feeds
+Location: tests/validation/multigrid/test_refinement_three_level.py:175: in test_nonequilibrium_masks_and_fresh_middle_feeds
+Error: E   AttributeError: 'FixedPatchStepper' object has no attribute '_advance_child'
+Category: test_instrumentation_stale_private_method
+Log: run-093/stdout.log
+
+SKIP: tests/unit/test_union_refinement.py::AdapterNumerics::test_eight_steps_bitwise
+Reason: ADAPTER_DIR external CPU adapter not configured.
+

src/CelerisLab/lbm/refinement_stepper.py

@@ -822,40 +822,26 @@ class FixedPatchStepper:
                             grid=((pairs + self._block[0] - 1) // self._block[0], 1), **kw)
 
     def step(self, n=1):
-        """Advance G→G by n base intervals: level l takes 2**l hops per interval.
+        """Advance G→G by n base intervals: level l takes 2**l hops per interval."""
-
-        All n intervals are enqueued in order on the one stream, and the host waits
-        once, before returning; step(0) launches and waits for nothing. On return
-        the device work is complete, step_count and an open window cover all n
-        intervals, and the raw-exchange ledger holds the final interval only.
-
-        Asynchronous device errors surface at that single wait, so they are not
-        attributed to the interval that caused them; call step(1) repeatedly to
-        localize one (same launches, same bits). On any failure the instance is
-        failed, the exchange ledger and window are invalid, and step_count keeps
-        its value from entry to the failing call.
-        """
         if isinstance(n, bool) or not isinstance(n, Integral) or n < 0:
             raise ValueError("n must be a nonnegative integer")
         with self._lock, self.ctx.activate():
             self._require("ready")
-            if n == 0:
-                return
             root = self._levels[0]
             kw = dict(block=self._block, stream=self._stream)
             try:
                 self._status = "advancing"
-                self._exchange_valid = False
                 for _ in range(n):
-                    if self._observe_raw_exchange:  # stream-ordered: holds the last interval only
+                    if self._observe_raw_exchange:
+                        self._exchange_valid = False
                         for b in self._level_boundaries:
                             if b is not None:
                                 cuda.memset_d8_async(b.interval_integral, 0,
                                                      len(b.tables["cell"]) * 32, self._stream)
                     self._hop(0, root.current, root.spare, **kw)
-                    root.current, root.spare = root.spare, root.current  # host handles only
                     self._stream.synchronize()
-                self._step_count += n
+                    root.current, root.spare = root.spare, root.current
+                    self._step_count += 1
                     self._exchange_valid = self._observe_raw_exchange
                 self._status = "ready"
             except BaseException:

tests/unit/test_step_sync.py

@@ -1,170 +0,0 @@
-"""Host contract of FixedPatchStepper.step(n): one stream wait per call (mocked driver).
-
-All n base intervals are enqueued in order on the backend stream and the host
-waits once before step returns. Launch order and buffer arguments must equal n
-step(1) calls; completion metadata, raw-exchange and window ledgers, and failure
-invalidation are published only after that wait.
-"""
-from contextlib import nullcontext
-from unittest.mock import Mock
-import unittest
-
-import numpy as np
-
-from CelerisLab.config import LBMConfig
-from tests.unit.test_raw_exchange import body
-from tests.unit.test_refinement_three_level_multipatch import host_module
-
-BODY_LEVEL, HOPS = 3, 8  # body on L3: 2**3 hops per base interval
-
-
-def backend(window=False):
-    """Four-level host backend; every launch, memset and stream wait goes to one log."""
-    host = host_module()
-    log, allocations = [], []
-
-    def allocate(size):
-        allocations.append(Mock(size=size))
-        return allocations[-1]
-
-    module = Mock()
-    module.get_global.return_value = (Mock(), 20)
-    module.get_function.side_effect = lambda name: Mock(
-        side_effect=lambda *a, **k: log.append((name, a, k)))
-    host.compiler.load_module.return_value = module
-    host.cuda.mem_alloc.side_effect = allocate
-    host.cuda.memcpy_dtoh.side_effect = lambda array, pointer: array.fill(0)
-    host.cuda.memset_d8_async.side_effect = lambda *a: log.append(("memset", a, {}))
-    run = host.FixedPatchStepper(
-        LBMConfig(nx=48, ny=48, viscosity=.02), context=Mock(activate=nullcontext, sm_arch="sm_70"),
-        refinements=[[(6, 6, 26, 26)], [(20, 20, 44, 44)], [(56, 56, 72, 72)]],
-        level_blocks=[(4, 4)] * 4, level_objects=[(), (), (), (body(),)],
-        observe_raw_exchange=True, observe_window_exchange=window)
-    run._stream.synchronize.side_effect = lambda: log.append(("sync", (), {}))
-    run.initialize(*[np.full(level.shape, 1 / 9, np.float32) for level in run._levels])
-    log.clear()
-    return run, log, allocations
-
-
-def signature(log, allocations):
-    """Event names with device-pointer arguments replaced by allocation index."""
-    index = {id(a): i for i, a in enumerate(allocations)}
-    return [(name, tuple(index.get(id(a), "-") for a in args)) for name, args, _ in log]
-
-
-def names(log):
-    return [name for name, _, _ in log]
-
-
-class StepSync(unittest.TestCase):
-    def test_step1_waits_once_after_all_launches(self):
-        run, log, _ = backend()
-        run.step(1)
-        self.assertEqual(names(log).count("sync"), 1)
-        self.assertEqual(names(log)[-1], "sync")
-        self.assertEqual(names(log)[0], "memset")
-        self.assertEqual(names(log).count("BlockStep"), 1 + 2 + 4 + 8)
-        self.assertEqual(run.step_count, 1)
-        run.close()
-
-    def test_step_n_equals_n_step1_launches_with_one_wait(self):
-        single, single_log, single_alloc = backend()
-        for _ in range(3):
-            single.step(1)
-        batched, batched_log, batched_alloc = backend()
-        batched.step(3)
-        expected = [e for e in signature(single_log, single_alloc) if e[0] != "sync"]
-        got = signature(batched_log, batched_alloc)
-        self.assertEqual(got[-1][0], "sync")
-        self.assertEqual(names(batched_log).count("sync"), 1)
-        self.assertEqual(got[:-1], expected)  # same order, same ping-pong buffers
-        self.assertEqual(names(batched_log).count("memset"), 3)
-        self.assertEqual(batched.step_count, single.step_count)
-        index = [{id(a): i for i, a in enumerate(alloc)} for alloc in (single_alloc, batched_alloc)]
-        self.assertEqual(index[1][id(batched._levels[0].current)],
-                         index[0][id(single._levels[0].current)])
-        single.close()
-        batched.close()
-
-    def test_step0_launches_and_waits_for_nothing(self):
-        run, log, _ = backend()
-        run.step(2)
-        log.clear()
-        run.step(0)
-        self.assertEqual(log, [])
-        self.assertEqual(run.step_count, 2)
-        value = run.read_raw_exchange(0, level=BODY_LEVEL)  # step(0) keeps the ledger valid
-        self.assertEqual((value["time_start"], value["time_end"]), (1, 2))
-        run.close()
-
-    def test_completion_metadata_covers_all_intervals(self):
-        run, _, _ = backend()
-        with self.assertRaisesRegex(RuntimeError, "completed"):
-            run.read_raw_exchange(0, level=BODY_LEVEL)
-        run.step(3)
-        value = run.read_raw_exchange(0, level=BODY_LEVEL)
-        self.assertEqual((value["time_start"], value["time_end"], value["sample_count"]),
-                         (2, 3, HOPS))  # step(n) retains only the final base interval
-        run.step(2)
-        self.assertEqual(run.read_raw_exchange(0, level=BODY_LEVEL)["time_end"], 5)
-        run.close()
-
-    def test_window_accounting_across_batched_calls(self):
-        windows = []
-        for calls in ((1, 3), (4,), (1, 1, 1, 1)):
-            run, log, _ = backend(window=True)
-            run.step(1)
-            run.start_exchange_window()
-            log.clear()
-            for n in calls:
-                run.step(n)
-            self.assertEqual(names(log).count("BlockAccumulateExchange"), 2 * 4 * HOPS)
-            self.assertEqual(names(log).count("sync"), len(calls))
-            out = run.read_exchange_window(0, level=BODY_LEVEL)
-            windows.append((out["time_start"], out["time_end"], out["sample_count"]))
-            run.close()
-        self.assertEqual(windows, [(1., 5, 4 * HOPS)] * 3)
-
-    def test_failed_wait_publishes_nothing(self):
-        run, _, _ = backend(window=True)
-        run.step(2)
-        run.start_exchange_window()
-        run._stream.synchronize.side_effect = RuntimeError("device failure")
-        with self.assertRaisesRegex(RuntimeError, "device failure"):
-            run.step(3)
-        self.assertEqual(run.step_count, 2)  # value at entry to the failing call
-        self.assertFalse(run._exchange_valid)
-        self.assertFalse(run._window_valid)
-        for read in (lambda: run.read_raw_exchange(0, level=BODY_LEVEL),
-                     lambda: run.read_exchange_window(0, level=BODY_LEVEL), run.download,
-                     lambda: run.step(1)):
-            with self.assertRaisesRegex(RuntimeError, "failed"):
-                read()
-        run._stream.synchronize.side_effect = None
-        run.close()
-
-    def test_launch_error_mid_batch_skips_wait_and_fails(self):
-        run, log, _ = backend()
-        run.step(1)
-        log.clear()
-        hop, calls = run._hop, []
-
-        def failing(index, *args, **kw):
-            if index == 0:
-                calls.append(1)
-                if len(calls) == 2:
-                    raise RuntimeError("launch failure")
-            return hop(index, *args, **kw)
-        run._hop = failing
-        with self.assertRaisesRegex(RuntimeError, "launch failure"):
-            run.step(3)
-        self.assertNotIn("sync", names(log))
-        self.assertEqual(run.step_count, 1)
-        self.assertFalse(run._exchange_valid)
-        with self.assertRaisesRegex(RuntimeError, "failed"):
-            run.read_raw_exchange(0, level=BODY_LEVEL)
-        run.close()
-
-
-if __name__ == "__main__":
-    unittest.main()