Backtesting and Forward Testing

The short answer

A backtest answers a narrow question: if I had applied exactly these rules to exactly this history, what would have happened?

It does not answer whether the rules will work. The distance between those two questions is where most strategies that "worked in testing" quietly fail.

Why backtests overstate

Four mechanisms, each of which flatters results independently.

Overfitting

Test enough variations and one will perform well by chance. This happens with random data — it is a property of searching, not of markets.

The tell is fragility. A parameter capturing something real degrades gracefully: 48 and 52 work nearly as well as 50. A sharp peak at exactly one setting, with poor results on either side, is a fit to the sample.

Look-ahead bias

Using information that would not have been available at the moment of the decision. The classic version: a rule that uses the period's close to decide something within that period. You know the close only after the period ends.

This is easy to introduce by accident and it can be very hard to spot, because the resulting equity curve looks plausible rather than absurd.

Survivorship bias

Testing on the assets that still exist today. Delisted, failed and collapsed instruments are missing from most convenient datasets, and they are exactly the cases a method would have handled badly.

In crypto this is severe: the set of tokens available in a current dataset excludes a large population that went to zero.

Omitted costs

Backtests that ignore fees, spread, slippage and funding will show profits that do not exist. For a frequently-trading method this alone can invert the result. Model costs conservatively; the fee calculator will give you a sense of the round-trip magnitude for your own venue.

Testing that means something

Split before you start. Divide your data into in-sample and out-of-sample — 70/30 is a common split. Develop and optimise on the in-sample portion only.

Do not look at the out-of-sample data. Not once. If you peek, adjust, and re-test, it is no longer out-of-sample; it has become part of your development set, and you have lost your only independent check.

Test the final rules on the held-out data, once. If performance collapses, the in-sample result was a fit. That is a useful, cheap finding — much cheaper than finding out live.

Report the whole distribution. Maximum drawdown, longest losing streak, worst month. A profitable strategy with a 45% drawdown is one you will abandon in month three, which makes its expectancy irrelevant.

Forward testing

Once a rule survives out-of-sample testing, run it forward on new data — either on paper or at very small size.

Paper trading has a known weakness: it removes the emotional dimension entirely, so it does not test whether you will follow the rules when money is exposed. Small live size keeps the psychology roughly intact while capping the cost of being wrong. Either way, forward testing supplies things a backtest structurally cannot:

• Real slippage in real conditions
• Whether your entries were actually fillable at the prices assumed
• Whether you follow the rules — which is a property of you, not of the strategy, and not observable in historical data

That last one is the reason to forward test even when the backtest is excellent. The review routine gives you the adherence figure that turns it into a measurement rather than an impression.

A realistic posture

Treat a backtest as a hypothesis generator. It tells you an idea is worth the effort of testing properly. It does not tell you the idea is good.

The sequence that respects that: develop in-sample → validate out-of-sample once → forward test small → scale slowly, with the risk limits unchanged throughout.

Most of the strategies that die in live trading were never tested this way. They were optimised until they looked good, and the optimisation was mistaken for evidence.