Overfitting: when a system learns the past too well
How a system ends up fitted to the noise of the past, why the best of many tests is often luck, the warning signs, and the habits that guard against it.
By Baris Demirci · · Rev00
The idea
Past prices contain two things: patterns that might last, and noise that will not repeat. An overfitted system has been shaped to the noise. Its backtest looks excellent, because the rules have in effect memorised the past, and then it does far worse on anything new.
It is the student who memorised last year’s exam answers. Perfect marks on last year’s paper, lost on this year’s.
How it happens
Overfitting rarely comes from one bad decision. It builds up from many small, reasonable-looking ones, each made while looking at the results:
- Tuning settings. A lookback of 50 days does a little better than 40, so it becomes 50. Then the exit is nudged, then the stop.
- Adding filters to remove past losses. A rule is added because it would have avoided last year’s worst trades. It will not avoid next year’s.
- Choosing the markets and dates. Keeping the markets where the rules worked and the period where they looked best.
- Trying many ideas and keeping the winner. Running dozens of variations and publishing the best one, as if it were the only one tested.
Each choice feels like improvement. Together they produce a system that describes the past precisely and the future poorly.
The best of many tests is often luck
Test enough systems that have no real skill at all, and some of them will still look good by chance. If you then keep only the best one, you have selected luck.
Every test you run spends a little of the evidence in the data. The more you try, the more impressive a result has to be before it means anything, and the more honestly you have to count everything you tried, not just what you kept.
More rules help the backtest, not the future
Adding rules and settings always makes the backtest look better, because each one gives the system more room to fit the past. Results on new data improve at first, while the rules capture something real, then fall away as they start fitting noise.
Warning signs
- Many rules, few trades. Every rule and setting needs evidence behind it. A system with ten settings and forty trades has almost none.
- Very precise settings. If it works at 47 days but not at 45 or 50, the 47 was fitted. The backtesting article shows what a robust plateau looks like instead.
- Rules with a story about one event. “Skip trading in that month” or “exit if this exact pattern appears” usually exist to remove one painful moment from the past.
- A backtest that looks too smooth. Real systems have rough stretches. A test with almost none deserves suspicion, not excitement.
- A sharp drop on unseen data. If results fall apart the moment the rules meet data they were not built on, the earlier results were mostly fit.
Habits that guard against it
- Fewer rules. Every rule should have a reason you could explain before seeing the results.
- Settings chosen by reasoning, then left alone. Pick round, sensible values, and test the neighbours rather than hunting for the best one.
- Keep data back. Build on one part of history, test once on another, as in walk-forward testing.
- Test widely. A rule that works across several markets and periods is less likely to be fitted to one of them.
- Count what you tried. Keep a log of every variation. The longer the log, the higher the bar for believing the winner.
- Watch it before you trust it. Run the system on paper, or with small size, and compare live behaviour with the test before committing capital.
What it asks of you
- Accepting a worse-looking backtest. The honest version of a system usually looks less impressive than the tuned one. It is also the one more likely to hold up.
- Leaving past losses in. A system that lost in a bad year will probably lose in the next one too. Removing that loss from the backtest removes it from the record, not from the future.
The short version
An overfitted system has memorised the past instead of capturing something that lasts. It comes from many small choices made while looking at results, and from keeping the best of many tests. Use few rules, choose settings by reasoning, keep data back, test widely, and count everything you tried. A modest backtest you can trust is worth more than a brilliant one you cannot.
Written for people building their own systematic portfolio management process. Nothing here is a recommendation to buy or sell anything. See the disclaimer.