Objective
Generate auditable pre-inception daily total-return histories for registered 2x and 3x leveraged ETF paths, using underlying returns plus explicit funding, swap-exposure, financing-spread, and expense assumptions. Preserve live ETF observations exactly from each registry inception boundary onward and keep raw Yahoo/Testfol data out of the repository.
Approach
- Added
synthetic_levered_daily, which appliesL<em>r - SW</em>(L-1)<em>(FR+SP)/(100</em>252) - E/(100*252)on underlying trading sessions and compounds from a fixed level anchor. - Resolved one preferred live substitute for every registered base/leverage level: 40 paths across 25 bases, restricted to 2x and 3x.
- Reused the existing
EFFR,DGS3MO,TB3MS, and__IRX_TBILL_TR__funding chain. The importer records normalized input/output hashes and formula parameters in its report and canonical provenance JSON. - Scaled only the modeled prefix at the first common live session on or after
earliest_month; every later value is copied from the preferred live close unchanged. - Added injected underlying, live, and funding clients so acquisition and persistence are fully testable offline. Monthly composites are written atomically to
sim_monthly; true daily composites use an atomic batch in the regenerablesim_daily.sqlitesidecar.
Files changed
backtest/sim_history.py— exact daily transform, funding combination, registry path selection, live splice, injected acquisition, provenance, and importer orchestration.backtest/datastore.py— atomic daily-sidecar batch replacement and CLI controls for ordered funding fallbacks, SW, SP, E, gross mode, and materialization.tests/test_sim_history.py— hand-calculated formula/borrow-drag tests, IRX conversion, SPY/TLT four-path offline import, exact live suffix, span, idempotence, schema/integrity, no-raw-OHLCV, and rollback assertions.
Validation
.venv/bin/python -m pytest -q tests/test_sim_history.py tests/test_datastore.py tests/test_levered_path_returns.py tests/test_leverage_feasibility.py tests/test_refresh_market_data.py tests/test_vix_backwardation.py tests/test_reports_coverage.py— 127 passed.python3 -m research.artifacts check --mode full --offline— deterministic and current.python3 -m research.pipeline validateandpython3 -m research.pipeline verify --offline— graph and route contracts passed.python3 research/scripts/check_static_reports.py docs— 1,252 files and 178 canary routes passed before this result entry was generated..venv/bin/python -m pytest -q— diagnostic baseline completed with 1,741 passed and 3 skipped; 75 unrelated existing web/catalog/fixture failures remained, plus two missing Firefox/WebKit binary setup errors. None intersected the modified modules or affected dependency slice.- Diff/privacy checks confirmed no change to
data/market.sqlite, the registry, or raw Yahoo/Testfol inputs.
Results
The importer now deterministically produces source-tagged synthetic prefixes for SPY→SSO/UPRO, TLT→UBT/TMF, and every other preferred registry path. Tests verify the output span against the injected underlying, financing and expense drag against hand calculations, exact preservation of the live suffix, stable rerun hashes, atomic rollback, and absence of live OHLCV writes.
No production database import was performed during this task. Existing strategies do not silently opt into modeled paths, so canonical coverage had no semantic change.
Web UI exception and limitations
A public static web workflow is inappropriate for this capability because it is an operator-controlled local database mutation that depends on internal/regenerable market inputs; exposing execution or modeled observations on the static site would blur the public/private data boundary. The intended audience is repository data operators and researchers, using python3 -m backtest.datastore import-synthetic-levered --help. That path shares the tested core implementation, reports inputs, parameters, spans, and errors, and is covered by the 127-test validation slice above.
Reconsider a local web administration surface only if the repository gains an authenticated mutable operator UI that can execute imports without publishing raw or modeled observations. Synthetic prefixes remain modeled estimates rather than native ETF history and can differ because of tracking error, index construction, fees, swaps, taxes, and later provider revisions.