Before I Trust a Backtest, I Check the Data

I trade options, and I build software. Those two parts of my work meet in a question I keep coming back to: what would make me trust this result?

A backtest can produce a smooth curve and precise statistics. Before I take those numbers seriously, I need to know what went into them. Which quotes were available? Which observations are missing? What price does the simulation assume I could have traded?

In Green Machine, my research software, I built a small check for options and intraday datasets. I called it the options honesty study. Its job is to flag basic problems before I move on to testing a trading idea.

The first version checks whether required columns exist, whether there is enough date and row coverage under the configured thresholds, and how much of the required numerical feature data is missing. It also flags negative bids, nonpositive asks, crossed quotes where the ask is below the bid, and a high proportion of wide spreads.

The spread matters because a quoted midpoint is not a guaranteed execution price. Consider a hypothetical option with a $1.00 bid and a $1.20 ask. Its midpoint is $1.10. A simulation that assumes I can buy and sell at that midpoint needs evidence for that assumption. The quote alone does not provide it.

For this prototype, I defined a wide spread as more than 20% of the midpoint. If more than 15% of observations exceed that threshold, the check issues a warning. These are configurable research choices. They are not universal standards or proof that the remaining quotes are tradable.

The output has three states. FAIL identifies a blocking issue under these checks. WARN allows exploratory diagnostics while calling attention to execution assumptions. PASS means only that the dataset cleared this particular screen and can move to a controlled study. It says nothing about profitability.

I tested the implementation with deliberately constructed examples. A 25-date fixture with complete fields and narrow spreads passed. A version with wide spreads in every fifth quote produced a warning. A three-date fixture failed the minimum-history check. Separate tests checked missing columns, crossed quotes, and missing numerical features. All six existing tests passed when rerun for this article.

These are synthetic fixtures. They demonstrate that the tested branches behave as expected. They do not validate a historical options feed, establish realistic fills, or show a trading edge. The dates are generated using weekdays rather than a full exchange calendar.

Reading the implementation also makes its limits concrete. It counts distinct dates across the two input tables; it does not prove that their observations match at the same times. A row count does not establish complete strike or expiration coverage. An average missing-value rate can hide a concentrated gap in one feature. Those distinctions matter before calling a dataset ready.

There is more work to do: explicit timestamp and underlying alignment, exchange-session handling, duplicate and missing-quote checks, quote-age validation, and realistic execution-cost assumptions. This version also does not verify Greeks or whether every input was actually available at the moment a simulated decision would have been made.

The next experiment is to bring a real historical dataset through those checks and inspect the rejected records. After that, I can test whether a defined setup separates future outcomes, using evaluation periods that were not used to choose the rule. Performance claims come later.

This is the kind of work I want XIV to document: the question, the implementation, the result, and the remaining uncertainty. Sometimes the useful result is discovering that I cannot answer the trading question yet.

Neither bull nor bear. I follow the evidence and manage the risk. That starts with checking the evidence itself.