The workflow of the development phase using traditional trading system development platforms is:
- read data
- specify parameters
- compute indicators
- define rules
- generate signals
- produce trades
- evaluate performance
The goal of the model is to determine a set of rules that divide the data points into categories — those days when the rules tell the trader to have a long position and those days to remain flat. Importantly, the indicators are computed before the rules are evaluated and trades produced. In a word — compute indicators, then see what happened later.
While decision tree models can have more than two output classes, I recommend that you begin work with a system that trades a single issue long/flat. Long/flat systems that trade a single issue are easiest to develop and most robust. Short/flat systems are more difficult, in part, because tops are harder to identify than bottoms. Long/flat/short systems can be designed, but I recommend keeping long and short separate. Logic, indicators, and parameters are often different for short trades than long trades. Developing long/flat/short as two systems makes each less complex, easier to validate, easier to manage while trading.
The Indicator article illustrated that we are drawing a boundary perpendicular to the horizontal (indicator) axis, and using the functional relationship to divide data points based on their position relative to the vertical (category) axis. If there are several rules, there are several divisions of the data points. Each rule has its own axis and the model’s space is multidimensional. Each rule follows the same technique of drawing separating boundaries perpendicular to an axis.
Advantages of decision tree models include:
- They are not sensitive to outliers. The data does not need to be standardized or normalized prior to use.
- Models resulting from decision trees are understandable. They are a series of if-then questions.
- The computations are easy and processing is fast.
- Decision tree models are easily overfit.
- Models fit to slightly different data can be significantly different.
- Single decision trees (we will see ensembles of decision trees later) are seldom among the best models for financial data.
Traditional development typically begins with an idea. An article or blog posting might suggest that a particular indicator or chart pattern has predictive value, so the system developer writes the formula for that indicator, selects values for the controlling variables, and begins testing. The model is not general — the idea is integral to the model, defines it, and restricts it. Testing expands as additional rules are added to filter entries and exit trades. Ranges of parameter values are tested in an effort to determine the best fit, where best is measured in terms of the score computed for the objective function.
The process of finding the best parameters is called optimization. It consists of repeating steps 2 through 7 listed above with many variations of indicators, parameters, and rules. Eventually, the results will be good. We do not stop fooling with the system until the results are good.
The trading performance for the set of rules and parameters fit to the data is called a backtest. Backtesting is a necessary step in development, but it is not sufficient. A good backtest score shows only that the model fit the data for the in-sample period. Too often, the result of a backtest of the best set of rules and parameters is taken as an indication of the performance of the system and development stops. That is always premature. Backtest results say nothing about future performance.
The backtest results of the single decision tree can easily lead us astray:
- They are based on in-sample results. That is, testing the same data that was used to fit the model.
- They are the result of a single sequence of trades.
- The single sequence that catches our attention from an optimization run is always better than the average.
- When we say that “the future must resemble the past,” what we require is that the distribution of future results resembles the distribution of past results. Not that the single sequence is repeated.
- The in-sample period is assumed to be consistent throughout. It is assumed to be stationary. Predicting accurately while trading the system relies on the distribution of signals and trades in the future being similar to those in the in-sample data used during fitting. It may be. But we cannot tell from the work done so far.
Validation will help us evaluate the risk and profit potential of the system. We will go there next.