Docs › MTS Lab
Your data
The engine replays one simple file format. Use the synthetic example, record live public data, or convert data from anywhere — the agent can write the converter for you.
Where data lives
In the workspace's data/ folder (subfolders are fine): one file per instrument (or a few) per UTC day, e.g. data/binance-um/BTCUSDT/2024-01-15.cdata. Steps pick files with engine.data_files("binance-um/BTCUSDT/2024-01-*.cdata"); every experiment records each file's hash. A server compute has its own data folder. The home page's Data tab lists files and checks them.
The format (.cdata)
A stream of fixed 128-byte events, little-endian, no header. The engine merges files by time.
| offset | type | field |
|---|---|---|
| 0 | uint64 | ts_recv_ns — the replay time, ns since 1970 UTC |
| 8 | uint64 | ts_exch_ns — the venue's time (0 if unknown) |
| 16 | uint64 | seq — the venue's sequence (0 if none) |
| 24 | uint32 | cid — the instrument's id in instruments.json |
| 28 | uint16 | venue — binance 1, coinbase 2, hyperliquid 3, okx 5, bybit 6, kraken 7, deribit 8, … |
| 30 | uint16 | type — 1 BBO, 2 trade, 9 funding |
| 32 | body (96 bytes): BBO bid_px, bid_qty, ask_px, ask_qty (float64) · trade px, qty (float64), trade_id (uint64), aggressor (uint8: 0 the taker bought, 1 sold) · funding rate, mark_px, index_px, open_interest, next_epoch_ns, interval_ns | |
Python struct: header "<QQQIHH", BBO "<dddd", trade "<ddQB7x", funding "<ddddQQ".
How the engine uses it: an instrument's book is its last BBO, and a BBO equal to the previous one changes nothing. A resting simulated order fills when a trade with the opposite aggressor goes through its price (behind the queue ahead of it), or when the BBO moves through it — so a maker backtest needs trades as well as BBOs.
Instruments
Every cid in a file must be in ~/MTS Lab/instruments.json, keyed <symbol>.<venue>:
"BTCUSDT.binance": {"cid": 1001, "symbol": "BTCUSDT", "ticksize": 0.1, "lotsize": 0.001, "asset_class": "linear_perp"}
cid is unique in the workspace (convention: venue number × 1000 + n). asset_class: spot, linear_perp, inverse_perp, linear_future, inverse_future; contract_size and expiry for contracts. Components and strategies name instruments by port: ["BTCUSDT", "binance"].
Writing files
from mtslab import cdata
cdata.add_instrument("BTCUSDT.binance", cid=1001, ticksize=0.1, lotsize=0.001, asset_class="linear_perp")
cdata.write("binance-um/BTCUSDT/2024-01-15.cdata", bbo=bbo_df, trades=trades_df, port="BTCUSDT.binance")
# bbo_df: ts, bid, bid_qty, ask, ask_qty trades_df: ts, px, qty, side (+1/-1 or buy/sell: the taker)
# ts: UTC ns integers or pandas datetimes; BBOs equal to the previous one are dropped
df = cdata.read("binance-um/BTCUSDT/2024-01-15.cdata")
Rewriting the example's data through cdata backtests identically to the original, fill for fill.
Checking
python -m mtslab data check data/binance-um/BTCUSDT/2024-01-15.cdata
Or Check in the Data tab: events by instrument and type, time range, ordering, gaps, crossed or one-sided books, non-positive prices or sizes, times in the wrong unit, unknown instruments, and a note when a file has no trades. A clean check plus a short backtest is the way to accept new data.
Getting data
- The example: the leadlag template generates its synthetic days (
mtslab.synth); the Data tab makes single-instrument synthetic tapes. - Live public feeds: the install ships feed handlers for public market data and the recorder (
lab/bin/<venue>_md,lab/bin/md_recorder) — your own recordings of public data. - From anywhere: ask the agent in a project — "get Binance USD-M BTCUSDT book tickers and trades for 2024-01-15". It writes a small tool in the project with
mtslab.cdata, adds it as a step, runs it when you ask, and checks the files. Use public data or data you are entitled to; large order-book feeds make large files (several GB per busy day), so start with a day. - Depth and order-by-order data: the format above is top of book (L1). For market-by-price depth (L2) or order-by-order (L3) data, contact me.