Building a Fault-Tolerant Proxy Rotation Layer for Global Market Data Acquisition

Argus acquires market data across 10,752 daily symbols spanning US equities, European markets, Japan (TSE), Latin America, Southeast Asia, and India (BSE/NSE). The Hetzner server IP is flagged or rate-limited by several critical free-tier data sources. Specific blocks discovered during initial bulk download attempts: FRED's main endpoint accepted TCP connections but delivered zero bytes; CoinGecko's free tier returned HTTP 429 on every request; Stooq.com dropped the TCP connection outright. The Yahoo Finance crumb authentication endpoint worked intermittently but failed for India symbols, affecting 11,706 BSE/NSE tickers.

The design constraint was operational simplicity: the rotation logic needed to live in argus-common, the shared library crate, so every downloader binary got it without duplication. The 23 compiled downloader binaries each link argus-common. Adding proxy support to the common crate meant all 23 binaries gained it in one change.

The proxy pool lives in argus-common/src/proxy.rs and exposes two public types: ProxyEntry and ProxyPool. ProxyPool fetches 100 proxies from the Webshare.io v2 API on initialization, maintains them in a Vec, and rotates through them using an AtomicUsize index. Proxies are selected via fetch_add on the atomic counter, modulo the pool length. This produces round-robin distribution without a mutex on the selection path, which matters for the 50-concurrent-session downloader configurations.

Each ProxyEntry carries an AtomicU32 failure counter. Five consecutive failures mark a proxy as unhealthy; next_proxy() skips it. The failure threshold of 5 was chosen empirically: a single 429 or timeout should not retire a proxy, but a sequence of five indicates either a dead proxy or a source-specific block on that IP range. Healthy proxies are re-assessed at pool refresh time every 3,600 seconds.

The selection path is lock-free using Ordering::Relaxed on the atomic counter. Relaxed ordering is correct here. The atomic counter does not synchronize with any other memory; its only purpose is to produce a different index on each call. Stale reads in a concurrent scenario could cause two tasks to pick the same proxy index, but that is acceptable: the session isolation happens at the TCP level, and two tasks sharing one proxy momentarily does not violate any invariant.

The real complication in production was the Yahoo Finance crumb flow for India symbols. The crumb endpoint at fc.yahoo.com uses a session cookie that ties to the originating IP. With proxy rotation, each request in a download session could come from a different IP, invalidating the crumb. The fix was to acquire the crumb once per downloader task using the designated proxy for that task, then reuse the same proxy for the entire symbol backfill within that task. This required passing the proxy index through to the download loop rather than selecting fresh from the pool on each request.

Before proxy rotation: India BSE/NSE backfill stalled at roughly 40% completion with Yahoo 403 errors. After per-task proxy pinning: completions reached approximately 85% with failures concentrated in a subset of India symbols where Yahoo's crumb validation was stricter.

For US equity batch downloads running 50 concurrent sessions: throughput measured at approximately 180 symbols per minute across the pool versus 40 symbols per minute on direct Hetzner IP and approximately 120 symbols per minute with a single proxy. The 1.5x improvement from 120 to 180 reflects the round-robin distribution reducing per-proxy request density. Healthy pool size in a steady-state session: 83 of 100 proxies after one hour of active downloading.