MarketLab: Reproducible Market Experiment Platform

Situation

Financial experiments rot into messy notebooks with weak validation and unreproducible results. The subtle killer is look-ahead leakage: training on information that would not have existed at decision time, which silently inflates every amateur backtest into optimism.

Task

A package-first toolkit for reproducible market experiments: installable, config-driven, CI-gated, and published. Framed explicitly as a research tool, not a money-making system; the engineering goal is trustworthiness, not returns.

Action

• Built a validated YAML config layer: experiments are defined as hand-checked dataclass specs where weights must sum to 1.0, caps stay in range, and invalid configs raise before any compute runs.
• Implemented a rolling walk-forward evaluator with a built-in leak guard: the training window is fixed-width and slides forward; any row whose label resolves after the cutoff is excluded, and an executable test proves it cannot peek ahead.
• Added a paper-only Alpaca trading path whose client raises on any non-paper endpoint in code: the refusal to connect to live money is an enforced invariant, not a README disclaimer.
• Packaged the full pipeline behind three CLIs, a FastMCP server, and Docker; shipped to PyPI as marketlab 0.2.0 via OIDC trusted-publisher release.

How it works

Click to zoom

Result

Shipped to PyPI as marketlab; Docker image deployable to Azure, an Alpaca paper-trading bot that runs experiments end-to-end and sends results via Telegram, three CLIs, and a real FastMCP server. Since April 2026, the bot has been running ML models on auto: the portfolio is green, though not yet ahead of buy-and-hold. Results are reproduced from config rather than committed; there is no checked-in P&L number, by design.

Learning

The model is the cheap part; the contract around it is the expensive part. The sklearn estimators are one-line imports with a fixed random_state, but the config validation, the leak-free fold builder, and the paper-trading fence are where the real engineering went. In quant tooling, the work that earns trust is not the model: it is the harness that makes a result trustworthy and a mistake impossible. Walk-forward results reflect historical data only; the Alpaca path is fenced to paper endpoints and has no live-account integration.