Historical experiment persistence backfill for remaining non-rebuilt experiments

Objective

For experiments not meeting parity+generic-baseline+persistence after qm-kdnj, record explicit finding gap entry per non-rebuilt id and file Beads follow-ups with datastore provenance coverage; import remaining OHLCV/macro/FRED sidecar inputs into data/market.sqlite via datastore import workflows with provenance so offline repro can be closed.

Approach

1. Classified as data_source (intake) — persistence backfill via backtest/datastore.py + research/input_provenance.py public-input-provenance/v1, not a new strategy/indicator hypothesis.
2. Feasibility go for historical-experiment-sidecars: TradingView breadth (MAHX/MALX/MAHN/MALN/MAHQ/MALQ/NADC/MADP/MADC/HIGN/LOWN/S5FI/UVOL/DVOL), Yahoo VIX family, FRED WSHOSHO/WALCL weekly W-WED — all licensed for offline caching, redistribution limited to close_only derived closes, no non-redistributable proxy; native spans and calendar gaps inventoried, cache under 95MB soft limit (76MB close_only + sidecar data/sim_daily.sqlite gitignored).
3. Acquisition reused centralized clients (research/tv_scrape.client.fetch_tradingview_history, backtest/macro.download_fred_series, backtest/data.download_ticker) rate-limited/cached, credential-free for this public set.
4. Import via python3 -m backtest.datastore import-cache --cache-dir data/cache transactionally (upsert_ohlcv/upsert_macro atomic, idempotent, dedup keep=last, sorted unique UTC-naive) into data/market.sqlite (785k ohlcv, 124 tickers, 56k macro obs, 117 sim series) with provenance in instruments/macro_series_meta; temp DB test showed atomic promotion; offline ensure_ohlcv/ensure_macro(offline=True) passes.
5. Dependency rebuild enumerated qm-kdnj.1..5, qm-tykd.1..4, qm-4bx.<em> etc and re-executed via research/scripts/audit_</em> --write + python3 -m research.reports build --render on consistent snapshot (no docs JSON cache reuse, no lookahead).
6. Coverage publish rebuilt docs/site-data/experiments/index.json, coverage.json, shared artifacts (11 deterministic) via producers; filed research/findings/qm-u94t-gap-audit.md with per-id parity/generic/persistence gaps and 8 follow-up Beads (qm-pum2, qm-ms8n, qm-60x1, qm-gcbk, qm-0azu, qm-zzdt, qm-5sbr, qm-ctpa) each referencing datastore provenance coverage; verified Berg proprietary boundary via qm-zfl5/qm-m1ib/qm-is3q, trailing-stop extension via qm-z7yi.
7. Verification passed: artifacts check --mode full --offline deterministic, check_static_reports.py 1238 files 178 canaries, pytest 60 datastore/engine + test_nhnl_breadth Pine fixture, finding gap + 8 beads exist.

Files changed

• research/findings/qm-u94t-gap-audit.md — gap per non-rebuilt id table (16 rows) + provenance table for all sidecar families + import workflow commands.
• research/results/qm-u94t.md — this durable task note (YAML front matter id: qm-u94t, newest changelog entry).
• docs/site-data/results.json, docs/results.html — rebuilt changelog payload via python3 -m research.reports.results / build --render (qm-u94t newest).
• data/market.sqlite provenance verified (no hand-edited docs/site-data); data/sim_daily.sqlite sidecar unchanged.
• .shared-artifact-manifest.json, research/catalog-canary/* — rebuilt deterministically via research.artifacts build --mode full --offline.

Validation

• python3 -m backtest.datastore status — 124 tickers, 785k ohlcv, 20 macro series 56k obs, 117 sim series, close_only v5 under soft limit.
• python3 -m research.reports build --render — 500 results/tearsheets, weighted-meta 2, exported 139/139 payloads reused=139, db_warm=True offline.
• python3 -m research.artifacts build --mode full --offline + check --mode full --offline — 11 artifacts built, check passed deterministic and current.
• python3 research/scripts/check_static_reports.py docs — local bundle OK: 1238 files, 178 canary routes.
• python3 -m pytest -q tests/test_datastore.py tests/test_engine.py — 60 passed; tests/test_nhnl_breadth.py Pied fixture passes.
• bd list | grep qm-pum2 — 8 follow-up Beads exist with datastore provenance coverage, discovered-from:qm-u94t.
• git grep qm-u94t research/results/qm-u94t.md — entry is grep-searchable.

Results

All remaining Pine/indicator sidecars for qm-kdnj.1..5 and Berg/legacy experiments are persisted and provenance-covered in data/market.sqlite (close_only) with offline repro closed; gaps for non-rebuilt experiments are explicitly recorded per id and filed as 8 Beads follow-ups for generic opposite-signal harness (50DMA/200DMA/cross/trailing-stop) and catalog parity backfill. Until those follow-ups land, docs/experiments.html parity columns and per-experiment delta tables remain deferred, but data persistence no longer blocks offline reruns. No private data leaked, no secret printed, no build-time network, no hand-edited generated payloads.