Backtesting Indian markets: seven ways a backtest lies

1. Using what was not yet known

Look-ahead is any use of information that came after the moment of the decision. It is rarely obvious.

Ranking today's P/E against the full history, including years that had not happened yet. Using a day's close to decide a trade made at that close. Using company results on the quarter-end date instead of the day they were announced.

The cure is to rebuild each day as it stood. In Schema every rank is against earlier days only.

2. Only the survivors

Test a rule on today's Nifty 500 members over ten years and you have tested it on companies that are known to have done well enough to still be there. The failures have been removed.

For a study of breadth or momentum this flatters the past. Proper tests use the index's membership as it was on each date. Where that is not possible, the limit should be stated.

3. Corporate actions

A two-for-one split halves a share price overnight. Unadjusted, it looks like a 50% crash, and every average, band and signal after it is wrong.

Free price histories are less reliable here than they look. In its own data checks Aurevian found a widely used source applying some splits and bonuses only back to the first of January of the year they happened, leaving a false one-day fall on that year's first session: a large Indian retailer showed a 33% drop on a day NSE's own file shows it rose 0.4%.

The check is cheap: list every one-day move beyond about 15% and compare it with the exchange's own record for that day. Demergers need the same care, since the price falls and nothing is lost.

4. The thing being measured changed

• NSE's index P/E moved from standalone to consolidated earnings in April 2021. The published ratio fell by about a sixth overnight.
• Expiry days and lot sizes for index options have changed several times. A test of an expiry-day rule has to know which day was expiry day in each year.
• The closing price itself is set differently since NSE introduced a closing auction.
• Margin rules, the list of weekly contracts and transaction taxes have all changed what a derivatives strategy costs and is allowed to do.

A series that runs across such a break is two series. For options the breaks are many: backtesting Nifty options strategies.

5. Counting the same event many times

This is the least understood. Suppose a condition held on 60 days, and the market was higher six months later after 55 of them. That reads as 92%.

If those 60 days fall in two stretches, there were two events, and the six-month windows after neighbouring days are almost the same window. The evidence is two cases, not sixty.

Count separate stretches, and require them to be at least one holding period apart before calling them separate. A pattern with fewer than five such cases is an anecdote.

6. Costs

Brokerage, securities transaction tax, exchange and regulatory charges, GST and stamp duty are known and should be in every test. Slippage is not known and has to be assumed, conservatively.

For a rule that trades often, they decide everything. Aurevian tested a mean-reversion rule that was right 82% of the time and earned 0.15% a signal before any of them: mean reversion in Indian stocks.

7. Fitting the past

Try enough averages, thresholds and filters and one combination will have made money. It has learned the past, not the market.

The defences are old and they work: decide the rule before the test, keep part of the history unseen until the end, prefer rules with few settings, and distrust any result that falls apart when a setting is nudged.

What an honest backtest reports

• The rule, in full, with every setting, and how the settings were chosen.
• The data, its source, and how splits, bonuses and demergers were handled.
• The number of trades and the number of separate periods they fall in.
• Results after stated costs, with the worst day, the worst month and the deepest fall.
• Results on data the rule was not built on.
• After going live: how far the real results sit from the tested ones, and why.

If you have a rule and want it tested this way, write to Aurevian.

Keep reading