Autonomous Alpha Research with LLM Agents — A Design That Doesn’t Fool Itself

Thesis: The hard part of automating alpha research isn’t generating ideas. LLMs are already good at that. The hard part is running a noisy, expensive, easily-overfit search loop without the agent declaring victory on noise. The proposer is easy; the skeptic is the product.

1. The real bottleneck

Quant research is a generate → test → learn loop over an enormous space: features × transformations × horizons × symbols × model forms × gating conditions. An LLM is well-suited to three parts of that loop — proposing hypotheses, writing the code/config to test them, and interpreting results in prose. What it is not natively good at is the part that actually protects you: statistical discipline. Point a tireless agent at a noisy objective and it will find a Sharpe-6 backtest by lunch. It will also be wrong.

So the design question is not “can an LLM propose features” (yes, trivially) but “can you build a loop where the agent’s output is trustworthy enough to act on.” Everything below is in service of that.

2. Why a computation-graph substrate is the enabler

This is feasible in a real stack — and not in a pile of notebooks — for one structural reason: the research pipeline has to be machine-manipulable. In our case the substrate is a graph of typed computation nodes (Variables, PricingModels) wired by name, configured by JSON, fed by a simulator, and fit by a declarative pipeline. On top of that sit three things an agent can read and write:

• a class catalog (every node type, its config keys, units, semantics) — the agent’s API reference;
• a research journal of prior results tagged winner/loser/gotcha — so it knows what’s already been tried;
• runnable research docs that interleave design, code, and results — so an experiment is a reproducible artifact, not a one-off shell command.

The agent doesn’t “do machine learning.” It reads the catalog, emits a config that wires existing nodes (or proposes a new one), launches a sim, and parses the result. The graph is the interface. If your research surface isn’t declarative and introspectable like this, step one isn’t “add an LLM” — it’s “make your pipeline addressable.”

3. The loop

   ┌── PROPOSE ──┐    economic hypothesis + mechanism (not a grid point)
   │             ▼
   │          WIRE        emit Variable/sim_cfg; verify it actually references live data
   │             ▼
   │           RUN        vargen → fit → sim, parallelized, with OOS + regime splits
   │             ▼
   │         EVALUATE     parse trade_stats: Sharpe, edge/trade, turnover, cost
   │             ▼
   │         CRITIQUE     adversarial agent: overfit? regime artifact? wiring stale?
   │             ▼
   └── REMEMBER ──┘    journal the outcome (winner/loser/gotcha) → ground next round

The proposer must justify a mechanism, not just a number — hypothesis-driven, never a blind grid sweep. Grids are how you manufacture false positives at scale; a required economic rationale is a cheap, powerful prior that prunes the search to defensible regions.

4. The skeptic is the product

This is where a serious system lives or dies. Every guardrail below exists because the field (and most practitioners, painfully) has been burned by its absence:

• Multiple-testing control. An agent that runs 10,000 sims is a data-snooping machine. You need a held-out final test set the agent never sees during search, deflated/Bonferroni-style Sharpe expectations as a function of trials, and a hard cap on degrees of freedom per hypothesis. The agent’s superpower — throughput — is also its primary failure mode. Budget your false-discovery rate explicitly.
• OOS + regime validation, always. A single favorable window can show every cell positive simply because the cells share underlying days. The rule: re-validate any candidate on a sample several times larger and across distinct regimes, and watch for Sharpe collapse. A “first positive result” is a hypothesis, not a finding.
• PnL is the objective, not r². Tick-level prediction accuracy is not a proxy for trading PnL — a model can improve correlation and lose money, because fills happen with delay and at specific queue positions. The agent must validate on simulated PnL, never on a prediction metric. Bake this in or the agent will optimize the wrong loss with great enthusiasm.
• Cost and turnover realism. Many “edges” are exactly the half-spread wide and vanish the moment costs are charged honestly. A sign-flip test (does the opposite signal lose by the same magnitude?) is the cheapest way to detect “this signal carries zero forward information and you’re just paying costs.”
• Wiring correctness. The subtlest failures aren’t statistical — they’re plumbing. A config that references a stale/inactive instrument, or a duplicated node that silently doubles an event feed, produces a beautiful, completely fake result. An autonomous agent must verify that its generated config references live data and the intended node graph — not trust that codegen did the right thing. Treat every green result as guilty until the wiring is audited.

If the loop in §3 is the body, this section is the immune system. A system without it doesn’t produce alpha; it produces confident garbage faster than a human could.

5. Multi-agent topology

The reason to split roles is that self-confirmation is the enemy. One agent that both proposes and judges will rationalize its own ideas. So:

• Proposer — generates hypotheses with mechanisms, grounded in the journal (don’t re-test known losers) and the feature-engineering surface.
• Coder — translates a hypothesis into node/config, runs the wiring-correctness checks.
• Runner — fans out sims across cores (OOS + regime cells), collects trade stats.
• Skeptic — adversarially tries to kill each candidate: refit the same idea, demand the sign-flip test, check regime attribution, hunt for stale references. Defaults to “reject” under uncertainty.
• Synthesizer — survives only candidates that beat the skeptic; writes the journal entry and surfaces it for human review.

A panel of independent skeptics with different lenses (statistical, microstructure-cost, wiring, regime) catches failure modes that a single redundant verifier won’t.

6. What the LLM is genuinely good and bad at here

Honesty about this is what makes the design credible:

The division of labor follows directly: the LLM owns generation, translation, and interpretation; the acceptance decision stays with the guardrails and a human gate.

7. Where to stop on the autonomy ladder

• L1 — Suggest: proposes hypotheses for a human to run. Safe, low-leverage.
• L2 — Run & report: runs the loop, hands back raw results. Useful, still trusting.
• L3 — Propose validated candidates: runs the full skeptic gauntlet and surfaces only survivors with their evidence. This is the right target.
• L4 — Auto-promote to live: don’t. Risk capital does not get allocated by an agent that can be fooled by a stale config.

The realistic product is an alpha-research copilot that turns the crank and presents pre-vetted, adversarially-tested candidates for a human’s final judgment — not a fire-and-forget money printer.

8. Measure the researcher, not just the alpha

Treat the research agent as a system with its own metrics: fraction of proposals that survive OOS, dollars of validated edge per compute-hour, and — most important — the realized false-discovery rate (how many “winners” later fail forward). If you can measure the loop, you can improve the loop: tune the proposer’s priors, sharpen the skeptic, prune dead branches of the search.

9. The open problem

The unsolved core is automated overfitting control at scale. Throughput is the agent’s gift and its curse: the same machinery that explores the space exhaustively also snoops the data exhaustively. The teams that win this won’t be the ones with the cleverest proposer — ideas are cheap and getting cheaper. They’ll be the ones with the most ruthless, well-instrumented skeptic. Autonomous alpha research is, in the end, an exercise in building a machine that is very hard to fool — including by itself.

I’ve built this pipeline end to end: feature research, model fitting, bit-identical research-to-production inference, a microstructure-realistic simulator, exchange certification, and live systematic trading. The guardrails above aren’t theoretical — each one is a scar.