Objective
Backfill EWZ from the existing Yahoo cache, import the FRED DFF effective federal funds rate, and create reproducible synthetic monthly and daily histories for every 2x/3x leverage mapping while preserving live market observations.
Approach
Added an explicit import-synthetic-levered datastore command and reusable simulation-history functions. The importer resolves every registered base/leverage pair, promotes cached source data, computes causal ideal daily-reset returns with optional prior-observation DFF financing, prepares all series before an atomic monthly batch replacement, and writes the optional daily representation to the gitignored sidecar. Coverage and shared public artifacts were then rebuilt through their canonical producers.
Files changed
backtest/sim_history.py: registry planning, deterministic modeled-history calculation, provenance, cache promotion, and coordinated import flow.backtest/datastore.py: atomic batch upsert plus the explicit importer CLI, offline/materialization/VACUUM controls, and structured result reporting.data/market.sqlite: EWZ, DFF, and all modeled monthly histories with their source metadata.research/reports/export.pyandresearch/strategies/economic_trend_macro_qm_9wte.py: dependency-path fixes uncovered by canonical rebuilds.tests/test_sim_history.py,tests/test_reports_export.py, andtests/test_economic_trend_macro_qm_9wte.py: formula, promotion, idempotence, preservation, materialization, and caller regressions.docs/site-data/coverage.json,docs/catalog.html, strategy coverage annotations, and shared canary artifacts: producer-owned coverage publication.
Validation
- 93 focused importer, datastore, report, coverage, consumer, and levered-path tests passed.
- Pipeline validation passed for all 28 nodes and 38 edges; all offline route scenarios passed.
- Direct SQLite checks found 25 mapped bases, exactly 40 expected modeled series, 11,718 monthly rows, and 253,928 daily rows; both databases returned
integrity_check=ok. - EWZ contains 6,559 Yahoo rows from 2000-07-14 through 2026-08-13; DFF contains 26,347 FRED rows from 1954-07-01 through 2026-08-18.
- A row-for-row comparison against the pre-task database found all 785,032 original OHLCV rows, zero missing rows, and zero changed closes.
- The post-VACUUM tracked database is 77,459,456 bytes, below the 99,614,720-byte soft limit.
- The static report bundle, secret/path scans, whitespace check, and final deterministic offline artifact check passed.
Results
Added a provenance-preserving datastore importer that promotes cached EWZ Yahoo and DFF FRED data, generates all 40 registered 2x/3x ideal daily-reset histories with optional DFF financing, atomically upserts 11,718 monthly levels, and materializes 253,928 daily sidecar rows without changing any pre-existing live close. Refreshed public coverage and shared artifacts now show EWZ as DB-backed; focused tests, graph and artifact gates, database integrity, and the 95 MiB size limit all pass.
Feature completion
A web workflow is inappropriate because this is an operator-only command that mutates local databases, can VACUUM them, and consumes raw market observations that the public static site must not expose. The intended audience is repository data operators using python3 -m backtest.datastore import-synthetic-levered; its calculation and datastore integration are covered by the focused tests and the command documents its controls through --help. Reconsider a web entry point only if the repository gains an authenticated local control plane with mutation authorization, progress/error handling, and a boundary that cannot leak raw source data.