Berg proprietary data proxy feasibility

Objective

For any Berg disclosure dependent on proprietary or otherwise unavailable inputs (30k-indicator system internals, non-redistributable series), assess public proxy feasibility, license/redistribution, history length, and point-in-time availability; document as data_source investigation if proxy is viable or as explicit untestable documentation if not — with no private data in public artifacts.

Parent epic qm-tykd (Research Milton Berg turning-point signals) and its coverage ledger v1 (research/findings/berg-coverage-ledger-qm-tykd.5.md §3) defer all proprietary-data vetting to this bead. Sibling qm-m1ib closed as a deferral companion; qm-tykd.1..4 already rejected proxy-only experiments on public data.

Approach

1. Intake classified as data_source with external_data + temporal_research and publication_required=true (public_artifact auto-added). No signal; not an indicator. split_required=false at intake — child data_source tasks only if a public proxy is go and needs new acquisition.
2. Planning pinned two blocking work items: feasibility_assessment (data_source_feasibility) then coverage_publication (data_coverage_publish).
3. Feasibility (data_source_feasibility source berg-proprietary-inputs, go_no_go=no_go) audited four inputs per ledger §3 families and gap record research/findings/qm-zzdt-berg-proxy-boundary-gap.md §§1-2: native 30k, NYSE TRIN/share-volume, NYSE advance/decline paid, native composite. Each was checked for authority/publishers, license/ToS, redistribution, history length/PIT, proxy chain, and historical-range maximization checklist (alternative proxies with relaxed thresholds, synthetic extension, constituent rebuild, third-party extended history).
4. Coverage publication writes this durable result note and rebuilds docs/site-data/results.json via the canonical research.reports results producer. No data/market.sqlite mutation and no docs/ hand-edit.

Per-Input Dispositions

1. Berg 30k native daily — F1, F2, F3, F4 (institutional only)

• Native status: proprietary unavailable. Authority: Milton Berg Advisors institutional system — research-process page verbatim “His system tracks over 30,000 daily — incorporating research from all disciplines…”, R3 Model Building Using Data Extremes (2023) confirms 30,000+ daily proprietary indicators.
• License/ToS: No public license/ToS for bulk daily redistribution; institutional-only. Ledger policy no_pdf_redistribution=true, proxy_disclosures_required=true. No Berg ticker in data/market.sqlite (instruments WHERE ticker LIKE '%BERG%' = 0 rows) and no __BERG*__ synthetic.
• Redistribution: not_allowed for native. Public proxies are allowed via research/input_provenance.py:public-input-provenance/v1 (TradingView/Yahoo free tier, close_only).
• History length: Native internal unknown; public proxy history already maximized: SPY 1993-01-29 (8428 rows), INDEX:HIGN 2001-06-05 (6331), INDEX:LOWN same (6312), INDEX:S5FI 2006-12-29 (4956), USI:UVOL 1996-04-19 (7616), USI:DVOL 1992-01-02 (8643), ^VIX 9215; 252-day warmup where applicable.
• Point-in-time: Native PIT unavailable; proxies are daily close with causal T_close → T+1 close via backtest/engine.py:run_backtest one-bar lag (not same_close), close_only schema (ticker, date, close), next-close tradable; T+2 extra lag verified per family.
• Proxy chain: SPY close vs SMA distance / rolling percentile (F1), HIGN/LOWN/S5FI breadth (F1/F3), UVOL/DVOL advancing/declining volume proxies (F2); all persisted idempotently; quality lower than native but acceptable with disclosure per ledger §3; F4 composite is a constituent-level rebuild from F1–F3 proxies.
• Historical-range maximization: (a) alternative proxies relaxed to high/low counts for A/D and return-derived capitulation for TRIN; (b) synthetic extension rejected (no calibration); (c) constituent rebuild for F4; (d) third-party scrape not pursued for proprietary internals.
• Disposition: explicit untestable entry with rationale and acceptable proxy analysis noted (ledger §3, gap record §2, qm-tykd.1..4 rejections). No child data_source task — proxies already maximized and qm-tykd.1..4 already evaluated on them.

2. NYSE TRIN / share-volume — F2 (volume climaxes, capitulation)

• Native status: paid NYSE share-volume unavailable free tier. Authority: TRIN = (adv_vol/dec_vol)/(adv_issues/dec_issues); qm-tykd.2 lane TRIN_share_volume_lane = insufficient_history offline.
• License/ToS: NYSE share volume/TRIN requires licensed history; free public sources incomplete; full paid history not redistributable per ledger §3 F2.
• Redistribution: not_allowed for true TRIN/volume; allowed for price-derived proxy.
• History length: Ideal NYSE daily 1990s; proxies UVOL 1996-04-19 (7616), DVOL 1992-01-02 (8643) maximize free history; qm-tykd.2 primary capitulation proxy SPY ret1 <= -2.5% (183 signals, 2.17%) used due to close_only Volume=0 (now close_only with no volume column).
• Point-in-time: TRIN lag T+0 close; proxy uses same derived-close T_close → T+1 with T+2 late-input test (retains at T+2 per qm-tykd.2).
• Proxy chain: USI:UVOL/DVOL as declining/advancing volume proxies plus SPY return capitulation; labeled with close_only caveat; |ret|/VIX and count-derived proxies considered but rejected as unlike; TradingView source via public-input-provenance/v1.
• Historical-range maximization: relaxed threshold to count-derived down-day proxy but retained UVOL/DVOL as closest defensible free breadth/volume; third-party scrape for paid NYSE not authorized.
• Disposition: explicit untestable for exact TRIN formula with acceptable proxy analysis and insufficient_history lane; no new licensable feed — existing proxies sufficient and experiments already rejected.

3. NYSE advance/decline paid breadth — F3 (thrust)

• Native status: paid NYSE A/D not redistributable. Authority: A/D thrust; ledger §3 F3; qm-tykd.3 uses HL Zweig 10-day low 0.40 → high ≥0.615 within 10 days.
• License/ToS: Full NYSE A/D daily is paid subscription; free tier via INDEX:HIGN/LOWN is index-level highs/lows, not true A/D — mismatch retained per gap record §2.
• Redistribution: not_allowed for true A/D; allowed for index highs/lows proxies.
• History length: Ideal 1990s; proxies HIGN/LOWN 2001-06-05 (6331/6312), S5FI 2006-12-29 (4956), MAH/MAL 2005-01-03 (5440) adjacent; SPY/QQQ/IWM multi-index 1993+ where DIA 0 rows.
• Point-in-time: Daily breadth at close; thrust on T_close HIGN/(HIGN+LOWN) with T+1 execution and T+2 extra lag (qm-tykd.3 retains at T+2 −0.00pp).
• Proxy chain: INDEX:HIGN/LOWN as highs-lows count proxy for A/D, S5FI adjunct; multi-index 4-index reduced to 3-index where DIA unavailable; provenance table in gap record §2.
• Historical-range maximization: constituent-level rebuild considered (full NYSE constituent OHLCV) but rejected as not defensible for paid A/D ratio; alternative high/low count relaxation already applied.
• Disposition: explicit untestable for exact A/D thrust ratio (missing thrust threshold/lookback/index list per ledger §3 F3) with acceptable proxy analysis; no child task — free breadth already maximized.

4. Berg native composite series — F4 (extreme composites, confirmation, overlay)

• Native status: no proprietary composite series exists. Authority: R3 describes composite building from extremes but discloses no weight vector/aggregation/confirmation/overlay (ledger §3 F4, §4 cross-report synthesis).
• License/ToS: No licensable Berg composite data product; research-process page lists only thesis whitepapers.
• Redistribution: not_applicable — derived composite is public-proxy aggregation only.
• History length: Inherits shortest leg S5FI 2006-12-29; rule-based proxies count>=2 and z-mean>=1.5 span 1994-06-01 to 2026-07-24 (8069 days, qm-tykd.4 common spine); maximization via F1–F3 constituent rebuild, not synthetic.
• Point-in-time: Derived daily composite count at T close from F1–F3 proxy signals, traded T+1 close with fee_bps=10 (_run → allocatesmartly next_close); T+2 extra lag retains (−1.7bps) per qm-tykd.4 frozen protocol.
• Proxy chain: count>=2 (F1 dist>4% & |z252|>1.5, F2 vol_z>2 & |ret|>1%, F3 |ret10|>5%) and z-mean>=1.5, equally-weighted aggregation of F1–F3 proxies; labeled proxy (no Berg native performance claim).
• Disposition: explicit untestable for exact composite weights/logic with acceptable proxy composite and overlay backtest; no child task — derivable from already-persisted F1–F3 proxies and already rejected (qm-tykd.4 overlay 8.44% vs SPY 11.00%, −2.56pp).

Behavioral Proof

• python3 -m backtest.datastore status → schema_version=5, ohlcv_layout=close_only, 127 tickers, 806897 ohlcv, under soft limit; connect_readonly query-only.
• sqlite3 data/market.sqlite "SELECT COUNT(*) FROM instruments WHERE ticker LIKE '%BERG%'" → 0 rows (no proprietary adoption).
• sqlite3 data/market.sqlite "SELECT ticker, start_date, end_date, n_rows, source FROM instruments WHERE ticker IN ('INDEX:HIGN','INDEX:LOWN','INDEX:S5FI','USI:UVOL','USI:DVOL')" → 6331/6312/4956/7616/8643 as above, sources tradingview.
• sqlite3 data/market.sqlite "SELECT sql FROM sqlite_master WHERE type='table' AND name='ohlcv'" → (ticker TEXT, date TEXT, close REAL, PRIMARY KEY (ticker, date)) — close_only, no volume column.
• grep -R "miltonberg.com" research/findings/berg-coverage-ledger-qm-tykd.5.md research/findings/qm-zzdt-berg-proxy-boundary-gap.md → three whitepaper URLs plus research-process landing page cited; proxy_disclosures_required=true.
• bd show qm-zfl5 qm-m1ib qm-tykd.4 --json → qm-zfl5 OPEN P3 → this publish, qm-m1ib closed investigation deferral, qm-tykd.4 closed rejected (n=358).
• python3 -m research.reports results → rebuilt docs/site-data/results.json and docs/results.html + detail pages via canonical producer.
• python3 -m research.artifacts check --mode full --offline → deterministic shared/canary bundle current (no hand-edited docs/ bytes).

Files Changed

• research/results/qm-zfl5.md — this durable result note (YAML front matter id: qm-zfl5, searchable summary; four per-input dispositions with license/history/PIT/proxy chain).
• docs/site-data/results.json, docs/results.html, docs/results/qm-zfl5.html — rebuilt via canonical research.reports results producer (qm-zfl5 newest).
• data/market.sqlite — not mutated (read-only verification only; 30k and paid NYSE history not imported; close_only preserved).

Validation

• No private data in public artifacts: only public closes cited; no credentials, cookies, authorization headers, or full authenticated URLs logged; public-input-provenance/v1 sanitized.
• python3 -m research.pipeline validate and python3 -m research.pipeline verify --offline — graph and offline verification pass in fixture-free run for this task's profile (data_source feasibility complete, no new acquisition required).
• python3 research/scripts/check_static_reports.py docs — static bundle OK.

Results

Every Berg unavailable input now has a written disposition with go/no-go and proxy chain: all four are not redistributable / not licensable for public import (no_go for native Berg and paid NYSE feeds) and therefore receive explicit untestable entries with rationale and acceptable proxy analysis noted in this ledger and in the upstream ledger/gap records. Acceptable public proxies are already persisted and maximize free history at reasonable precision cost with disclosed mismatches (high/low counts for A/D, return-derived capitulation for TRIN, constituent rebuild for composite). No child data_source tasks are spawned — there is no licensable series to acquire; the existing proxy coverage already supports the qm-tykd.1..4 rejections. No private data leaked to public artifacts. Follow-on ledger v2 refresh (qm-ckab blocked) may incorporate these dispositions verbatim if the parent ledger is rereleased.