Backtesting: what a test can and cannot tell you
How a backtest works, what it can and cannot tell you, the ways it flatters a system, and what to check before you trust one.
By Baris Demirci · · Rev00
The idea
A backtest runs a system’s rules over past prices as if they had been traded, and records what would have happened. It is how a mechanical system is checked before any money goes near it.
It is not a look at the future. History ran once, and a backtest replays that one run. What it can do is tell you whether your rules were ever coherent, and how they behaved through conditions that really happened.
How a backtest works
Day by day, the test feeds the rules only the prices that existed at that point, lets them decide, fills the orders at a price a real order could have got, charges the costs, and keeps the account. At the end it reports what happened.
What it can tell you
- That the rules are complete. Every situation has an answer, including “do nothing”.
- That the rules could have been traded, with the orders and costs a real account faces.
- How the system behaves. How often it trades, how long it holds, how often it is right, and the shape of its drawdowns.
- How it went through real episodes, such as a crash, a long sideways market or a sharp rally, and whether that matches what you expected of it.
What it cannot tell you
- What happens next. It is one path through history. A different order of the same events could have looked very different.
- What you will do. A backtest never hesitates, skips a trade or overrides a rule. You might.
- Whether the past conditions will come back. Markets change: costs, rules, participants and behaviour all move over time.
The ways a backtest flatters a system
Most backtests are not wrong on purpose. They are wrong in the same few ways, and each one makes the system look better than it is.
- Looking ahead. The test uses information the rules could not have had at that moment, such as acting at today’s close on a decision that needs today’s close to make. Even a small leak can make a useless system look excellent.
- Survivors only. Testing a stock system on today’s index members leaves out the companies that fell out of the index or disappeared. The test then only ever owns winners.
- Perfect fills. Assuming every order fills at the exact level, with no slippage, no gaps and no missed trades. Breakouts and anything trading at the close are especially sensitive to this.
- Fitting. Every setting you adjust while looking at the result spends a little of the evidence. Adjust enough of them and the test only shows that you are good at adjusting.
- Bad data. Wrong prints, missing days, and splits or dividends handled the wrong way can all create trades that never could have happened.
Keep some history back
The usual defence against fitting is to divide history. Build and adjust the rules on one part, then test them, once, on a part they have never seen. Walk-forward testing repeats that step by step through time.
Look at the neighbours, not just the best setting
A rule that works only at one exact setting is a warning sign. Try the settings around the chosen one. A robust rule shows a broad plateau: its neighbours do nearly as well. A fitted one shows a lone spike: move one step and the result falls away.
Reading a backtest report
Most reports lead with a handful of figures. Each answers a different question:
- Annual growth (CAGR): how fast the account grew, on average, per year.
- Maximum drawdown: the deepest fall from a high. See drawdowns.
- Volatility: how much the account swings from day to day or month to month.
- Sharpe ratio: growth per unit of swing. Higher means a smoother ride for the same growth.
- Number of trades: how much evidence there is. A handful of trades shows very little, whatever the other figures say.
- Time under water: how long the account spent below its last high.
No single figure is enough. A system with fast growth and a fall you could never sit through is not a system you can run.
Before you trust a backtest
- Were all decisions made only on information available at the time?
- Does the data include the companies that disappeared?
- Are costs, slippage and gaps charged realistically?
- Were the settings fixed before the final test, and was that test run on unseen data?
- Do nearby settings give similar results?
- Are there enough trades to mean anything?
- Does the behaviour, including the worst stretches, match what you expected of the system?
The short version
A backtest replays your rules through one run of history. It can show that the rules are complete, tradeable and behave the way you meant. It cannot show what happens next or what you will do. Most tests flatter in the same few ways: looking ahead, survivors only, perfect fills, fitting and bad data. Keep part of history back, check the neighbours of every setting, and read the drawdowns before the growth.
Written for people building their own systematic portfolio management process. Nothing here is a recommendation to buy or sell anything. See the disclaimer.