Overfitting & Validation

Overfitting is not carelessness. It is the expected outcome of evaluating many configurations against one dataset and keeping the best. Even with no genuine edge anywhere in the search space, the best of twenty attempts will look good — because you selected it for looking good.

The arithmetic is worth internalising because it is more severe than intuition suggests. Testing n independent configurations and keeping the best gives you roughly the n/(n+1) quantile of the outcome distribution.

And the count is larger than people think, because it includes everything you tried and discarded — every abandoned indicator, every threshold nudged, every timeframe checked. The search does not reset when you change approach.

Three designs, in increasing strength. All of them work by reserving data from the selection process; they differ in how much and how carefully.

Hold-out split

Split once, typically 70/30. Tune on the first part, evaluate once on the second. Simple, and strong provided you evaluate only once — every additional look at the test set converts it into training data.

Walk-forward

Several rolling train/test windows. Each test slice is evaluated once, and agreement across windows is what constitutes evidence. Stronger than a single split because it tests whether the edge persists rather than whether it existed once.

Out-of-sample instruments

Develop on one instrument, validate on a different one with similar characteristics. This is a strong test because it holds the market structure roughly constant while changing the specific price history entirely.

The best defence is not better validation, it is fewer things to validate. Every constraint you impose before looking at results is a constraint you cannot accidentally fit.

• Fix parameters at conventional values. RSI 14, SMA 50/200, ATR 14. These were not chosen by fitting your data, so they cannot be fitted to it.
• Write the thesis first. A stated hypothesis makes the strategy falsifiable and rules out the search-until-something-works loop.
• Cap the condition count. Three or four. Each additional condition multiplies the space of strategies you could have arrived at.
• Prefer round numbers. A 3% stop rather than 2.7%. If the difference matters, the strategy is fitted; if it does not, use the round one.
• Change one thing at a time. And record it. Undisciplined iteration makes the effective search count unknowable, and an unknowable count cannot be corrected for.