Backtesting with AI, honestly: what the validator counts and what it refuses to

Abstract

Language models make it cheap to produce trading rules and expensive to trust them. We describe the validation layer that sits between a compiled model and the live scanner: a bar-by-bar replay over provider history, a strict definition of a trigger, an outcome scorer that requires a fill before it will count a loss, and a dollar simulator that sizes contracts the way a trader does. We list the choices that make the numbers conservative and the two places the method is weakest.

1What a replay is

For every bar in the lookback the evaluator is run as if that bar were the latest, over a window sized from the model's own lookbacks. The question is always the same: would this rule fire here? A rule that is gated to a session is only evaluated inside it, which is what makes sixty days of 5-minute bars replay in about a second.

2What counts as a trigger

Every step passing is not enough. A trigger requires that price traded into the entry zone on that bar. A model that arms constantly but rarely fills is a model with its entry in the wrong place, and the replay should say so rather than credit it with imaginary trades. Each unique setup counts once, however many bars it stays armed.

3Scoring an outcome

Path after the fill	Outcome	R
Stop touched before target one	loss	−1
Target one touched, then target two before price returns to entry	win, runner paid	T1 share + T2 share
Target one touched, then price returns to entry	win, runner flat	T1 share only
Stop touched before any fill	no fill	0, not a trade
Stop and target in one bar	loss	−1, conservative
Nothing within the horizon	undecided	excluded from win rate

The same scorer runs on live fires in the Triggered tab, so a model's validation record and its live record are measured the same way. Earlier versions of the live scorer counted an unfilled setup as a loss and let a later stop touch overwrite a banked target-one win. Both were wrong in the pessimistic direction, which is the acceptable direction to be wrong in, but wrong.

4The gate

A model may go live when nothing crashed, it fired enough times for its timeframe, at least one trade was decided, and the win rate on decided trades is 40% or better. The gate exists to catch models that lose money, not to certify edge. Edge is the trader's judgement, made with every trigger listed and replayable bar by bar.

5From R to dollars

The P&L simulator takes starting capital, a risk per trade in dollars, a contract, a cap on contracts, a split between the two targets, and a minimum stop. Contracts are sized as the floor of risk over stop distance times point value, at least one, at most the cap. That last clause is what stops a trigger with a 1.25-point stop from posting a fantasy 100R trade: with contracts capped, its reward-to-risk is large but its dollar contribution is bounded, and the minimum-stop filter lets the trader remove it altogether and see the result without it. The card reports profit factor, expectancy, payoff, drawdown and recovery, streaks, a month-by-month strip, and a plain-language read that refuses to call a profit factor "strong" on fewer than thirty decided trades.

6Where this is weakest

1. Sixty days. The provider serves about two months of intraday bars. Anything annualised from that is labelled as an extrapolation, not a forecast.
2. Bar-level fills. Intra-bar order is unknown. Ambiguous bars resolve as the stop.
3. Same-history repair. When the improvement loop edits a model, it is judged on the window that produced the edit. See the companion note on self-improving models for what that buys and what it does not.