Use the right source for the time horizon
The Info endpoint serves recent and bounded historical queries for supported request types. WebSocket subscriptions are appropriate for ongoing capture. Public S3 archives provide selected bulk historical records for advanced users.
No single source contains every market, account and chain event forever. Define the required fields, retention horizon and acceptable gaps before choosing an ingestion path.
| Source | Best for | Main limitation |
|---|---|---|
| Info API | Recent snapshots and account queries | Request windows and rate limits |
| WebSocket | Forward real-time collection | Reconnect and gap recovery |
| S3 archives | Bulk books, contexts, fills and blocks | Requester cost and incomplete updates |
What the documented archives contain
The hyperliquid-archive bucket stores L2 book snapshots at market_data/[date]/[hour]/[datatype]/[coin].lz4 and asset contexts at asset_ctxs/[date].csv.lz4. The official example uses date 20230916, hour 9, data type l2Book and coin SOL.
Separate hl-mainnet-node-data paths contain fills by block, older trade formats, explorer blocks, replica commands and miscellaneous events such as transfers, staking and funding. node_fills matches the API format, but node_trades uses a different format.
The archive documentation explicitly says monthly uploads are not guaranteed to be timely and data may be missing. Do not backfill silently with fabricated rows or forward-filled trades.
s3://hyperliquid-archive/market_data/[date]/[hour]/[datatype]/[coin].lz4 s3://hyperliquid-archive/asset_ctxs/[date].csv.lz4
List, download and extract a requester-pays object
Pass --request-payer requester to each S3 command. The requester pays the transfer cost, so list a narrow prefix and estimate the object size before a large download.
The example downloads the official SOL object to /tmp, extracts it with LZ4 and prints the first records. The official extraction command removes the compressed /tmp file. Keep a separate immutable copy when you retain the dataset.
aws s3 ls s3://hyperliquid-archive/market_data/20230916/9/l2Book/ --request-payer requester aws s3 cp s3://hyperliquid-archive/market_data/20230916/9/l2Book/SOL.lz4 /tmp/SOL.lz4 --request-payer requester unlz4 --rm /tmp/SOL.lz4 head /tmp/SOL
Record provenance and schema before transformation
The historical-data page documents paths and archive families, but it does not publish a complete versioned schema for every object. Inspect each raw file, save the observed schema and reject unexpected changes before a transform runs.
- Save the exact S3 URI, retrieval time in UTC, object size, ETag and SHA-256 digest.
- Record the archive family, date, hour, data type, coin, compression and decompression command.
- Keep the downloaded object immutable. Write parsed data to a separate versioned location.
- Record field names, scalar types, numeric units, timestamp unit and timezone, book-side order and asset mapping in a versioned schema.
- Record the row count, first and last timestamp, duplicate count, parse failures and missing intervals.
- Keep node_fills, node_trades and other archive formats separate. Do not merge records until a documented mapping or a tested converter defines each field.
A defensible capture pipeline
For tax, accounting or execution research, verify whether the dataset represents fills, orders, transfers or mark-to-market values. Each record type has a different purpose and cannot replace another record type.
- Store each l2Book WebSocket message as a complete WsBook snapshot. Do not treat the message as an undocumented delta.
- Detect heartbeats, reconnect and resubscribe.
- Record raw numeric strings and event timestamps.
- Deduplicate fills and reconcile gaps after reconnect.
- Label missing periods instead of interpolating silently.