VIG
the house takes its cut — the research record
2026-08-02 17:49
← 06 RESEARCH / deep_research_data_2026-07-31.md

Data: sources and independent collectors

2026-07-31. Deep-research pass over five dimensions: on-chain mining, already-indexed datasets, official Polymarket APIs, cross-venue references, and collector engineering at $0. All claims sourced in parentheses; verification verdicts weighed; unconfirmed items marked; refuted/unverifiable items collected at the end.

Verdict

The single highest-value new collector is the CLOB websocket book recorder, because order-book state is the only data that can never be backfilled: there is no historical order-book endpoint anywhere in the CLOB API (confirmed absent from clob-openapi.yaml path enumeration, verified), and book snapshots + price_change diffs on the market websocket are the sole record of depth, spread, queue, and per-trade fee_rate_bps — the raw material for maker fill-rate, markout, and event-window supply work. Everything else on the list is either backfillable later or cheap enough to add opportunistically. Ranked build order:

  1. wss book recorder (windowed capture around event windows; CLOB market channel, verified live) — irreplaceable data, fee_rate_bps only streams here.
  2. Kalshi REST sweeper (markets + orderbooks + /series/fee_changes joined to the existing Polymarket census) — full market-data surface verified keyless and no-auth on both hosts (live curl, HTTP 200); the cheapest genuine cross-venue arb measurement available.
  3. UMA dispute ledger — gamma umaResolutionStatuses transition poller (current-state-only array, verified) plus UMA subgraph backfill (entities verified in UMA docs).
  4. On-chain post-fee arb miner via Envio HyperSync — replicates AFT-2025 on the post-fee regime, which the paper explicitly could not do ("we do not take into account the fees because Polymarket currently does not charge per trade", AFT-2025 Sec 7.1, verified); realized fees are now directly on-chain in both event regimes.
  5. Sharp-reference feeds: Atlanta Fed xlsx daily ingest (verified working download), the-odds-api credit-budgeted snapshots (verified 500 credits/mo free), VoteHub polls poller (verified 200 no-auth).

Ranked datasets to pull (no collector needed): HuggingFace TimeSeventeen/Polymarket-v1 (verified: full v1 tape 2022-11-21→2026-04-28, 1.2B OrderFilled rows, ~49GB parquet, CC-BY-4.0 — includes ~4 weeks of post-fee trading plus the split/merge/redeem events needed for realized-arb computation with zero mining); Dune polymarket_polygon and unified prediction_markets.* for retrospective queries (verified, ~24h latency — never for live sweeps); warproxxx/poly_data self-hosted for the post-Apr-28 v2 tape (verified: GPL-3.0, 2,258 stars, pushed 2026-06-29, HyperSync free tier in production).

On-chain: post-fee arb replication

The AFT-2025 measurement (Saguillo/Ghafouri/Kiffer/Suarez-Tangil, "Unravelling the Probabilistic Forest") mined Jan 1 2024–Apr 1 2025 via Alchemy's public node, filtering CTF contract 0x4D97…6045 for OrderFilled/PositionSplit/PositionsMerge, and explicitly excluded fees (paper PDF Sec 4.2 and 7.1, verified against the extracted text). A post-fee rerun is therefore new measurement, not replication theater — and the fee expansion of Mar 30 2026 gives it a natural event study.

But the paper's exact pipeline is broken by a protocol migration:

Extraction path: free RPCs cannot carry this. dRPC public is the best keyless option (100-block getLogs cap, 10K-log responses handled — reproduced) but returned intermittent empty arrays in the original pass (silent-empty did not reproduce in verification — one transient 403 did; treat zero-result windows as retry-mandatory either way); polygon-rpc.com 401s getLogs (reproduced); publicnode/nodies gate archive requests behind tokens/paid plans (verification corrected the original "non-browser blocking" framing); 1rpc.io caps at exactly 50 blocks (reproduced). A full pass is ≥63K requests over 1–3 days. Alchemy free caps getLogs at 10 blocks/request at 60 CU (docs — verified); a full pass ≈38M CU vs a reported ~30M CU/mo free allowance (allowance figure from search snippets only — probable, not confirmed). The right tool is Envio HyperSync (polygon.hypersync.xyz, chain 137 confirmed in docs, ~2000x benchmark for sparse topic-filtered scans — verified; free API token mandatory, free-tier quota absent from docs — unconfirmed, check at signup). warproxxx/poly_data uses HyperSync free tier in production (verified), which is practical evidence the quota suffices for this workload.

Decoder assets: Polymarket/ctf-exchange-v2 repo is the authoritative V2 ABI (verified); polymarket-subgraph's AssemblyScript mappings are ready-made decoders but V2 coverage is unconfirmed — check commits for 0xE111… before relying on them; Whittaker's blog tutorials are pre-fee, pre-V2, wallet-level (directionally useful only); py-clob-client is off-chain-API only — metadata, not mining.

Already-indexed sources

Official Polymarket APIs

Four keyless surfaces, all verified live 2026-07-31:

Rate limits are documented, IP-based, sliding-window, throttled-not-rejected, and generous: Gamma 4,000/10s general, CLOB /book 1,500/10s, /prices-history 1,000/10s, data-api /trades 200/10s (rate-limits.md — verified, every number matched).

What must be recorded live because it can never be reconstructed: order-book state (no historical endpoint — verified), per-trade fee_rate_bps off the wss, holders/oi/liquidity/rewards-config snapshots, umaResolutionStatuses transitions, and new_market/market_resolved/tick_size timing. What need not be collected live: the trade tape itself (data-api /trades reconstructs it later).

Unconfirmed and load-bearing: the semantics of takerBaseFee/makerBaseFee=1000 as inputs to the fee formula were not verified against the fees docs — confirm against docs.polymarket.com/trading/fees and /api-reference/get-fees.md (volume-tiered) before hardcoding fee-aware arb math. See also the refuted fee-rate numbers below.

Cross-venue references

Collector engineering at $0

Build plan for this desk

The desk today: public-repo GitHub Actions crons committing CSV ledgers (desk health ledger, 26-min neg-risk sweep windows, whale tracker). Additions, in order:

1. wss book recorder (build first). Windowed capture, not 24/7: schedule 1–5h capture jobs around known event windows (FOMC days, game windows, Nov 3 2026 midterms) inside single long-lived Actions jobs under the 6h cap. Connect to the CLOB market channel with a rotating watchlist from the existing census (top-N by volume + any market inside an event window); subscribe with custom_feature_enabled for best_bid_ask/market_resolved. Record raw book, price_change, last_trade_price (this is the only public stream of per-trade fee_rate_bps — verified) as zstd JSONL, one file per date/market-group. Liveness = 120s data-inactivity watchdog cross-checked against REST /midpoint; on freeze, reconnect and start a new snapshot epoch — never trust PONG. Optionally hold one RTDS connection (wss://ws-live-data.polymarket.com) in the same job as a free site-wide trade firehose. Commit per-window summary CSVs to the ledger; promote raw JSONL to a monthly GitHub Release; if volumes grow, spill to R2. If the desk later wants continuous coverage, a GCP e2-micro is the free VM of choice (Oracle's idle-reclamation makes a low-CPU recorder a reclamation target — verified).

2. Kalshi REST sweeper. Piggyback on the existing 60s sweep pattern: each cycle pull /markets (filtered to categories overlapping the Polymarket census), orderbooks for matched pairs, and cache /series/{ticker} fee_multiplier/fee_type plus /series/fee_changes for fee-aware edge math (all verified no-auth). Join on a maintained Polymarket↔Kalshi market map; commit a cross-venue spread ledger CSV. This turns the existing single-venue census into genuine cross-venue arb measurement at zero marginal cost. Backfill resolved-market studies from the public /historical endpoints.

3. UMA dispute ledger. Cheap cron: one Gamma sweep per run over recently-closed markets, logging (conditionId, umaResolutionStatuses, umaBond, umaReward, resolvedBy, observed_at) and emitting a transition row whenever the status array changes — this is the only way to get timestamped proposed→disputed→resolved transitions (verified current-state-only). Backfill historical base rates from UMA's voting subgraphs (PriceRequest/CommittedVote/RevealedVote entities — verified) or UMA's on-chain OptimisticOracle events; verify The Graph's free quota at signup (unconfirmed).

4. On-chain post-fee arb miner. HyperSync-based incremental job keyed on a committed block cursor: each run pulls V2 OrderFilled + FeeCharged + NRA PositionsConverted + CTF PositionSplit/PositionsMerge since the cursor, filters to Polymarket conditionIds from Gamma (mandatory — ProphetCTFExchange contamination is proven, verified), decodes fees from the event fields (both regimes carry them on-chain — verified), and commits (i) per-market-hour volume/fee/rebate aggregates and (ii) a split-merge round-trip table (same-address split-then-sell / buy-then-merge within N blocks = executed arb candidate, now net of realized fees) — AFT's definitions on the post-fee regime. Backfill Mar 30→today in chunked jobs under the 6h cap; take the pre-Apr-28 portion from TimeSeventeen/Polymarket-v1 instead of mining it. Track the wrapped-collateral ERC20, not USDC.e, after the cutover (verified). Validate sampled windows against dRPC. Before hardcoding fee math, confirm the formula and current rates against docs.polymarket.com/trading/fees and get-fees.md (volume-tiered) — the secondary-press rate table was refuted (below).

5. Sharp-reference feeds. (a) Daily Atlanta Fed mpt_histdata.xlsx ingest (browser UA required — verified), parsed to a meeting-probability CSV ledger as the Fed-ladder fair-value anchor. (b) the-odds-api snapshotter with a hard credit budgeter (~16 calls/day, concentrated inside event windows; spot-check Pinnacle presence in eu on the free tier first — "most bookmakers" caveat, verified). No backfill exists at $0, so every day not collecting is reference history lost. (c) VoteHub /polls poller (verified) with an in-collector average as the midterms reference; optional scrape of Silver Bulletin's free generic-ballot page. Skip PredictIt (Cloudflare-403 from CI — verified).

Storage architecture as ledgers grow. Keep hot CSV ledgers small: partition by month, rotate before 50MiB, append-only. Monthly compaction job: CSV/JSONL → parquet+zstd, attach to a GitHub Release (no size/bandwidth limits — verified), keep only the current month hot in the repo. R2 (10GB free, zero egress) is the overflow tier for raw WS captures. Never LFS. Artifacts only as intra-run staging. The desk's continuous commits keep the 60-day scheduler-disable timer reset, but add an explicit keepalive commit to any artifact-only workflow.

Refuted / unverifiable