Tharp's Thoughts Weekly Newsletter (View On-Line)
20% Off Sale Going On Now!
For a limited time, all of the Van Tharp Institute's core products are at least 20% off with special discounts of up to 50% off. We don't have sales often, so take advantage of this chance to save!
Home Studies for $636.00
Peak and Systems Home Study Combo for $1,200
eLearning and Audio Courses from $39.00
Books by Van Tharp from $22.40
To see what's else is on sale, click here.
Data Mining Bias
Be Wary of Your Back-Tested Results
Traders generally have a single goal in performing back-testing: to find systems that we can trade profitably going forward. The inherent assumption within the goal, however, presents the following problem: positive system metrics (such as expectancy, SQN, etc.) for back-tested results actually provide very little information in helping you know how the system will perform when traded live. In fact, back-tested results tend to be overly optimistic. Why? One of the main reasons is the data mining bias.
Data mining bias is certainly not the only culprit in poor future performance, but it tends to be a large contributor and one of which many traders aren’t aware. But don’t worry, once you have identified the problem, there are readily available remedies. But first, let’s dive into some background.
Let’s say you’ve done your system development job properly by starting with clear objectives, understanding your beliefs, identifying the system’s edges and as a result, you develop a set of rules and run a back-test. The system generates satisfactory descriptive system metrics and an acceptable equity curve. Everything looks great and you may be convinced that the system is ready to trade live. But are the results from this back-test a good representation of what to expect in the future?
In an ideal scenario, you would want back-tested results that reflect the statistical population of the system’s results. Unfortunately, the ideal scenario is a pure fantasy and back-tested results represent a below average sample from the ultimate population of lifetime trading results. To start, lifetime results would have to include the results for the system from the future. Good luck with that. Second, some combination of biases and/or mistakes inevitably crept into the system development process. Typically, system development is a long, iterative selection process of analyzing a great deal of data and adjusting based on the results. In other words, back testing is data mining. And where there is data mining, the data mining bias pops up most assuredly.
David Aronson talks about this phenomenon in his excellent book, Evidence Based Technical Analysis:
The observed performance of the best-performing rule, amongst a large set of rules, is a positively biased estimate of that best rule’s expected performance because it incorporates a substantial degree of good luck. Data miners who don’t understand this are likely to be disappointed with this rule’s out-of-sample performance.
In reality, you have no way of knowing the most likely future trading results because all you have is one biased sample. The figure below illustrates this situation.
When you simulate a trading system by running a back-test, the result comes from a sample of trades that the system would have taken in the past. Say the back-test has 100 closed trades. Well, going forward into the future, live trading will generate more 100 trade samples. Over a long enough time (you hope to trade for a long time), all of those samples of 100 closed trades will probably generate a distribution similar to the one shown. The question is where does the back-tested sample result fall in this distribution?
Everyone hopes that back-tested results represent the best case (slightly below average, shown by the green arrow) or at least the assumed case (average, shown in blue). Unfortunately, back-tested results probably — and more realistically — fall somewhere between the assumed case (blue) and the worst case (red). In fact, if your back-tested results used optimization with selection based on highest performance, it is almost assured that they are closer to the worst case (red) due to data mining bias.
What exactly is the Data Mining Bias?
In the simplest of terms, data mining bias is the overstatement of the true predictive power and expected system performance based on the level of performance that allowed the system rules to be selected in the optimization process. The data miner mistakenly uses the best back-tested performance to estimate expected performance in the future.
Data mining bias comes from a combination of randomness and a multiple comparison procedure whereby the system rules exhibiting the best performance are chosen from back-tested results. The data mining bias appears when testing combinations of indicators and optimizing combinations of parameters. The probability that a result arose by chance grows with the number of combinations tested.
In Evidence Based Technical Analysis, Aronson mentions five factors that contribute to the data mining bias:
- Number of system rule combinations back-tested: The larger the number of rules tested, the larger the data mining bias.
- The number of trades in the back-test: The larger the number of trades, the smaller the data mining bias.
- Correlation among system rules tested: The lower the correlation, the larger the data mining bias.
- Presence of positive outlier returns: The larger the effect of outliers on total system performance, the larger the data mining bias.
- Variation in expected returns among the tested system rules: The lower the variation, the greater the data mining bias.
These five factors combine in ways specific to the system rules and market data tested and lead to an unknown (but significant) amount of positive bias — the data mining bias.
Most traders are largely unaware of how to minimize this bias even while system development software enables it. Many, many testing software packages arm you with very powerful optimization tools. In fact, the average trading software package likely includes exhaustive optimization, genetic algorithm optimization, neural nets, Monte Carlo simulators, etc. Anyone with reasonable computer skills and limited programming experience will be able to quickly back-test multiple thousand system iterations in a reasonable amount of time using a standard PC.
Most trading software packages include instructions, videos, etc. on how to operate these powerful tools within the package. Unfortunately, the instructions almost always neglect to provide any guidance on how to properly interpret the results of these powerful tools. Valid interpretation requires being well versed in statistics, especially statistical data mining techniques and then being able to apply rigorously the scientific method.
If you followed a robust system development process and factored in the data mining bias, then hopefully your back-tested results would be closer to the assumed (blue arrow) case than to the extreme right (red arrow) on the distribution of past and future trading results shown in the figure. More likely, however, the average system developer uses ad hoc, un-scientific methods such as combining several indicators and using the optimizer to find the best fit parameter values. In cases like this, the system developer is completely blind to data mining bias. Or in other words, there’s only one question about these kinds of back-tested results: How many standard deviations to the right of the mean does the back-tested sample fall?
Several Ways to Deal With Data Mining Bias
First, you can reduce one or more of the required conditions. Since removing randomness from market data is not possible (if you know how to, please tell me), the only other option is to use a system development process that doesn’t make multiple selections based on performance. As an example, rule selection based on stability rather than raw performance might reduce data mining bias.
Additionally, Jaffray Woodriff, founder and CEO of the QIM hedge fund, uses another method. Considering the huge edge his fund enjoys, he has not disclosed the method fully. From the information he has made public, however, some quant bloggers believe they have backwards engineered what Woodriff does. Effectively, QIM's data mining is performed on completely random data. The results of the best system mined from the random data are then used as an estimate of the data mining bias present in back-tested results.
Finally, I propose a stress testing approach that attempts to simulate randomness that could be expected in the future. Specifically randomness is injected into the various system inputs. The randomness injected needs to be “a reasonable amount.” Back-tested data can give some guidance in what amount is “reasonable.” This approach attempts to produce an unbiased sample similar to cross-validation.
What if you already have a system that someone else developed and you want to test it? Fortunately, Aronson provides three statistical techniques to deal with data mining bias: 1) cross validation (Aronson mentions out-of-sample testing which is a subset), 2) a data mining correction factor proposed by Markowitz and Xu, and 3) randomization methods.
Although back-testing and optimization can be powerful tools in the system development arsenal, they can also easily be misused. Many would-be system developers are not even aware of the very real issue of data mining bias which results in poor live trading system performance. Fortunately, resources are available to system developers such that they aren’t fooled by randomness through data mining bias.
Back to Top
All Three Peak Performance Workshops Back-to-Back in January!
This is your chance to attend all three major workshops in only one trip to NC! The next time we have all three together may not be until Winter 2014 (though this is tentative).
If you live in South Africa and would for Van to host a workshop there, please contact us at email@example.com. There is a possibility he will be visiting South Africa in early to mid 2014.
Click here to see the full workshop schedule or to register.
Back to Top
Why is Market Volatility So Low?
“Complacency is a state of mind that exists only in retrospective: it has to be shattered before being ascertained.”
—Vladamir Nabokov, Russia-born U.S. novelist
Volatility Measures at the “Bottom of the Page”
Complacency in the markets is most often measured using the CBOE’s Volatility Index, or VIX. To simplify the concept, VIX really measures how much option premium is being paid for S&P 500 options. If options buyers and sellers think the market will be jumpy (volatile), then they bid up the price of options — market pundits call this a fear premium. On the other hand, if options buyers and sellers foresee smooth markets, then the price paid for options drops and complacency rules.
Over time, markets cycle from higher to lower volatility and back again. But over the past two years, the range of those swings (the amplitude of the cycles) has been severely reduced. In fact, since December of 2011, the VIX has never been above 30 — the traditional “line in the sand” that classically defined an oversold or “volatile” market. Let’s look at a long term chart:
As you can see, we’re in our 23rd month of volatility contraction. Now for all those perma-bears who are screaming that this is a long time and we should expect a massive correction soon — I direct your attention to the period on the chart from April 2003 through July 2007 (a massive 52 month stretch!) when there wasn’t a single VIX reading anywhere close to 30.
Here’s another graph from Lance Roberts of STA Wealth Management that tells the volatility story in a slightly differ manner:
Here we see that Roberts breaks the VIX readings into three categories: Capitulation, Complacency and a Greenspan-esque “Irrational Exuberance”. Again, notice that from 2003 to 2007 the market stayed “irrational”, calling to mind the quote from John Maynard Keynes that most everyone has heard, “The market can remain irrational longer than you can remain solvent”.
So volatility is really low already and then, early this week, it approached its lowest point of the last few months. In fact, VXX and similar volatility exchange traded funds/notes did reach their all-time lows due to contango issues with the instruments they hold.. Note: an all-time volatility low does not point to an immanent market top because this low VIX environment (what I’ve called a grinding bull market) can last for very long periods as recent history shows. There are some trading implications, of course, which we’ll discuss below. But for now, let’s discuss why this period of low volatility persists.
What’s Keeping VIX Depressed?
As we discussed above, low VIX shows a high level of complacency in the market. A simple explanation might be the so called Bernanke Put. The Fed chairman has essentially given investors a put option (the right to buy the market at a particular price at some time in the future), thus backstopping the market. Indeed, after markets had stabilized following the Great Recession of 2007-2009, Operation Twist plus QE’s 3 & 4 seemed to convince market participants that the Fed in particular and central banks in general would keep the liquidity train chugging at full speed. This has led to ongoing and ever-increasing complacency in the markets.
As traders and investors, what are the effects of this low volatility environment? Most obviously, anyone who has a strategy that includes selling options has issues to deal with right now. Covered calls, credit spreads, iron condors and the like are producing much smaller returns for those writing them. Players in this end of the trading world that I know are either using extreme caution or standing aside completely.
For those with a shorter-term time horizon and a small grasp of recent history, you’ll recall that, for the last three years we have had market drops either in the week before or the week of the U.S. Thanksgiving holiday. With VIX at a very low point relative to even the past 22 month’s depressed rates, one might look for a low-risk way to play a potential short term expansion in volatility.
Next week we’ll dig back into our series on monetary policy. Until then, your thoughts and comments are welcome — please send them to drbarton “at” vantharp.com
Matrix Insight Entry
I think the most valuable insight I gained from the book Trading Beyond the Matrix was a deeper understanding that my spiritual and emotional growth, are, in fact, fully interwoven with my ability to be effective in the physical realm.
There is some tendency to think of the spiritual, especially, as being at odds with the physical, each somehow interfering or competing with the other. The stories in this book along with my own experiences are helping me to more fully grasp how my spiritual, mental and emotional growth, clarity and healing are in fact part and parcel of my success in all realms.
This understanding helps me to shift my focus. When challenges arise — or in my day to day life — I am gradually learning to return to square one. To come home and get clear within myself. To find my faith and my relationship with the larger forces which support my path. There I find myself very naturally creating from a space of much greater ease, enjoyment and effectiveness.
It is my fondest belief that these shifts will help me more quickly and joyfully embrace my souls journey, as well as my appropriate place of service in the world.
And my bonus is increased happiness!
Peak Performance Home Study
The ultimate home study course for traders.
Learn how you think when you make and lose money, how to reduce stress,
how to avoid repeating mistakes, and how to eliminate emotions from trading.
Watch the video below for more information on this revolutionary product.
Now, 20% Off! Only $636.00
Click here for more information.
Everything we do here at the Van Tharp Institute is focused on helping you improve as a trader and investor. Consequently, we love to get your feedback, both positive and negative!
Click here to take our quick, 6-question survey.
Also, send comments or ask Van a question by clicking here.
Back to Top
Email us at firstname.lastname@example.org
The Van Tharp Institute does not support spamming in any way, shape or form. This is a subscription based newsletter.
To change your e-mail Address, e-mail us at email@example.com.
To stop your subscription, click on the "unsubscribe" link at the bottom left-hand corner of this email.
How are we doing? Give us your feedback! Click here to take our quick survey.
800-385-4486 * 919-466-0043 * Fax 919-466-0408
SQN® and the System Quality Number® are registered trademarks of the Van Tharp Institute
Be sure to check us out on Facebook and Twitter!
Back to Top