How to Trust (or Distrust) a Backtest

A beginner's guide to the biases that fool backtests, and how to read a result with healthy skepticism.

Category: Backtesting

In plain English

A backtest asks: "If I had followed this rule in the past, what would have happened?" You take a strategy, replay history, and see how the imaginary money would have grown. The catch is that it is very easy to build a backtest that looks brilliant on paper but was never real. The whole skill is learning which backtests to trust and which to distrust. This article is a checklist of the common traps, explained with tiny examples. - A backtest is a what-if replay of history, not a record of real trades. - A pretty result can be an accident, a mistake, or a fantasy. - Trusting a backtest means checking how it was built, not just its return.

What it measures

A trustworthy backtest measures how a rule would have performed under honest, realistic conditions. The biggest enemies are biases that quietly leak the future into the past or hide costs and failures. Here are the main ones, each with a plain example. Notice that almost every trap makes the result look better than reality, never worse. - Look-ahead bias: using info you could not have known yet. Example: backtesting a Monday trade that uses an earnings report which was not actually released until later that week. - Survivorship bias: testing only companies that still exist today, so all the ones that went bankrupt vanish and results look too rosy. - Selection bias: cherry-picking a flattering time window, like only testing 2010 to 2020 and skipping 2008. - Data snooping / p-hacking: trying 200 rules and keeping the one that won by luck. Test enough coin-flippers and one gets 10 heads in a row. - Overfitting: a rule so tuned to the past that it memorized noise; it shines on old data and flops on new data. - Survivorship + look-ahead are the two that most often turn a losing idea into a fake winner.

How SurgeFlow uses it

SurgeFlow's Backtesting tab (/backtest/benchmark) puts these honesty controls front-and-center in a "Test Assumptions" panel so you do not have to take a number on faith. It states the universe, the test period, the costs charged, and the known weaknesses out loud. Two distinctions matter most. In-sample means the result was measured on the same history used to design the rule; out-of-sample (OOS) means it was tested on fresh, later data the rule never saw, which is far stronger evidence. SurgeFlow currently labels its factor backtest as in-sample (it shows "0 OOS") rather than pretending otherwise. - Point-in-time inputs: it tries to use only data available on each historical day, reducing look-ahead bias (and admits non-US dates are weaker). - Transaction costs: per-market, asymmetric buy/sell fees are subtracted, so headline returns are net after costs, not gross fantasy. - Survivorship: openly flagged, because its universe is rebuilt from today's surviving names. - Publication gates: a result must clear honesty checks before it is shown, and immature data is labeled as research, not proof.

How to read it on the site

Start with the Test Assumptions panel before you even look at the return. Check whether the result is in-sample or out-of-sample, gross or net, and how much trading it required. Turnover is how often the strategy swaps holdings; high turnover plus real costs can quietly eat the whole edge. A train/validation/test split (designing on one chunk, tuning on a second, and judging only on a third untouched chunk) and walk-forward testing (repeatedly re-fitting on the past and checking the very next period) are the gold standards SurgeFlow points toward as factor history grows. - Find the gross-vs-net label: gross ignores costs, net is what you would actually keep. - Read the turnover number: more trading means more cost drag. - Look for the in-sample / OOS tag near the metrics. - Treat any caveat in the panel as the most important text on the page.

What it cannot tell you

Even a clean backtest cannot promise the future. Markets change, crowds copy good ideas until they stop working, and the next decade rarely rhymes with the last. A backtest is evidence about the past under specific assumptions, not a forecast. Be especially wary of three confusions: past performance is not a prediction; correlation is not causation (two things moving together need not cause each other); and statistical significance is not economic importance (a "real" effect can still be too tiny to matter after costs). None of this is buy or sell advice. - Beginner trust-checklist: Is it out-of-sample, or only in-sample? - Is the return net of realistic transaction costs? - Does it include companies that failed (no survivorship bias)? - Could the rule have used future data by accident (look-ahead)? - How many variations were tried before this winner appeared (p-hacking)? - Does the edge survive higher costs and a different time window?