Backtesting Methodology

A backtest answers one narrow question: had these exact rules been applied to this exact history, under these assumptions about cost and fills, what would have happened? That is genuinely useful. It is also considerably less than most people read into it.

Every backtest engine simplifies. Knowing exactly where is what lets you discount a result by the right amount rather than trusting or dismissing it wholesale.

Two separate requirements, and satisfying one does not satisfy the other: enough trades for the statistics to mean anything, and enough variety for the strategy to have met more than one kind of market.

Variety matters independently. Five hundred trades taken entirely within one sustained bull market is a large sample of a single regime, and it tells you nothing about the range or the crash the strategy has never seen.

Order matters here. Each step is designed to fail cheaply before the next one costs more, and skipping ahead is how a broken strategy reaches real capital.

1. 1
   Write the thesis down first

   One sentence on why this should work, before you look at any result. It is the only defence against rationalising whatever the data happens to show.
2. 2
   Set realistic costs immediately

   Commission and slippage before the first run, not after. A strategy that only works at zero cost should be discarded early, not tuned first.
3. 3
   Run a short range to debug

   One month. You are checking that trades happen at all and in plausible places, not measuring performance.
4. 4
   Inspect the trade log

   Read twenty trades individually. Entry timing, exit reasons, durations. This catches more than any metric.
5. 5
   Run the full in-sample range

   Now measure. Trade count first, then drawdown, then profit factor, then return.
6. 6
   Walk forward

   Validate on data the tuning never touched. This is the step that separates an edge from a memorised history.
7. 7
   Stress the assumptions

   Double the slippage. Shift the date range by two months. Drop the best trade. A robust strategy survives all three; a fitted one does not.
8. 8
   Paper trade

   One to two weeks minimum, for the failures a replay structurally cannot produce.

A backtest is only as good as the series under it, and data problems produce results that look like strategy results.

• Missing bars. A gap in the series shifts every subsequent index. A crossover that should have fired on the missing bar fires on the next one, or not at all.
• Bad ticks. A single erroneous print becomes a spike, and a spike triggers stops, breakouts and volatility filters. One bad tick can create a trade that never existed.
• Unadjusted prices. A split or dividend that is not adjusted for looks like a genuine gap. Any strategy reacting to gaps will react to a corporate action.
• Contract rolls. Futures series stitched across expiries contain a discontinuity at each roll that is an artefact rather than a price move.
• Survivorship. An instrument list drawn from what exists today excludes everything that failed, so the sample is the survivors by construction.