Trading System Development

This is the first in a series of postings describing techniques for developing reliable trading systems.  Additional posts follow in a sequence as they might be in a book. 

The Goal

There is a goal common to all traders:

To have confidence that the signals generated by the trading system precede trades that provide rewards adequate to compensate for the risk.

  • The key word is confidence.
  • The primary limitation is risk.

The Process

The process we will use closely follows the scientific method.

The scientific method consists of techniques for investigating phenomena, acquiring new knowledge, or correcting and integrating previous knowledge. It is based on measurable evidence validated by independent testing, as this flowchart, courtesy of a Wikipedia article illustrates. 

This flowchart shows the scientific method as a flowchart in a form similar to the computer programs we write.

In developing trading systems, we analyze historical price data (and possibly data from other activities).  Then create models to process that data -- make transformations and compute indicators, postulate rules, adjust parameters, generate signals, and record the resulting trades.  The process continues in cycles of adjusting, fitting, and testing the model using the in-sample training data until we are satisfied with the resulting trades.  A separate set of out-of-sample data is tested to validate that the model has learned to recognize genuine signals, and is not merely fit to the noise.

Thinking ahead, if we are successful in developing a system that has low risk and high profit potential, we plan to actually trade it.  For this, we need a separate and distinct model -- one that monitors trades, evaluates risk, adjusts position size, and estimates profit.  It tells us whether the system is working or is broken.  

Managing the trading of the system also follows the scientific method.   The data used by this model is the set of recent trades.  The output of the model is a set of two metrics. 

  •           safe-f is the maximum safe position size for the next trade. 
  •           CAR25 is the estimate of profit.

As trades are accurate and profitable, safe-f remains high and it is safe to use a large portion of the trading account.  As trades are less profitable, safe-f declines.  This indicates that the system has become riskier and position size should be reduced.

CAR25 can be used to compare alternative uses of funds.  When several systems have been developed and are being tracked, the one with the highest CAR25 is the one that should be actively traded.

The trading system that generates signals and the management system that indicates the health of the system are two distinct systems.  This flowchart combines the two shown above and shows their relationship.  

The common element between the two systems is the best estimate set of trades.  The validation phase of the development process produces a set of trades that are the best estimate of future performance of the system.  These result from applying the model (that has been fit during the iterative process of model adjustment and backtesting) to the never-used-before out-of-sample validation data.  A commonly used and excellent validation technique is the walk forward process.  We will discuss validation, including walk forward, in a future post.  The trades that are produced in each of the walk forward steps are gathered together into the best estimate set of trades.  safe-f and CAR25 are computed using these trades and are used to decide whether the system should be moved from development to trading.

When the activity moves from development to trading, the trading system model is frozen, and no further changes are made to it.  The input data for the trading system changes from historical data to real-time data, signals are identified, and trades generated.  Conservative traders will wait for the system to generate several trades before beginning to trade with real money.  These trades are called shadow trades.  They are added to the best estimate set and used to monitor system performance.  Throughout the life of a trading system, all of its trades -- whether taken with real money or as shadow trades -- are added to the best estimate set of trades and are used to compute safe-f and CAR25.  As the flowchart shows, when performance deteriorates, the trading system is taken offline and perhaps sent back to development.    

During the development process, we want a sound model that analyzes data and produces signals.  Described broadly, success of the trading system depends on the quality of the model and its fit to the data.  Quality has several aspects that affect its usefulness, including:

  • Accuracy
  • Profitability
  • Risk
  • Reliability
  • Practicality

Our next posting will discuss trade quality.

Trade Quality — Metrics

During development of a trading system we fit the model to the in-sample training data. Parameters are adjusted, each unique set of parameters defining an associated model. We are searching for the model that best fits the data. Best is defined by a formula -- an objective function computed using the metrics associated with the trades, such as percent of trades that are winners, ratio of average win to average loss, maximum drawdown, etc.

One of the general principles of modeling and forecasting is that the out-of-sample data that will later be processed by the fitted model is similar to the training data.  Specifically to technical analysis, profitable systems depend on the future resembling the past. That is, that when the model is applied to new data the trade results will have the same characteristics as those found in the training data during development. Before going further, we need to understand three important concepts:

  • distribution
  • population versus sample
  • stationarity


The model processes the data, identifying patterns described by the rules of the model that precede profitable trades. While we cannot expect the same patterns and trades to occur in the same order in the future as they did in the training data, we rely on the distribution of signals and trades being the same in the future. We expect that there will be approximately the same number of trades per year, the same winning percentage, the same average gain, etc. We can visualize the distribution in several ways.  One is to sort the trades from worst to best and plot the percentage gain per trade, one bar per trade.

For the charts displayed in this posting, the trades are the out-of-sample trades for the period 1/1/2008 through 12/8/2017 produced by the "RSIExampleForBlog" system described in the post "Why use AmiBroker."  The data series is XLF, using data supplied by Norgate Premium Data.  You can copy that code from the posting listing, paste it into your copy of AmiBroker, and run it yourself.  The system works well for many issues, including US major market ETFs such as SPY, and sector ETFs such as XLF.  

By defining equal-width bins, we can assign trades to the appropriate bin, count the number in each bin, and form a histogram. The histogram forms a probability mass function (pmf) or probability density function (pdf). The distinction between pmf and pdf depends on whether the data is discrete or continuous. Asuming it will cause no confusion, I will refer to them as probability density functions, or pdfs. Note that these are pdfs in the statistical sense, not to be confused with the pdf file format of documents.

In general, the form of the pdf for a set of trades depends on the rules controlling the trades -- particularly the exits. If the trades are close-to-close percentage profit or loss, the pdf is similar to the common bell-shaped curve.  However familiar it looks, it is not safe to assume that the distribution follows the Normal statistical distribution -- financial data and trade results seldom do.

Each pdf can be transformed into its associated cumulative distribution function, CDF, by summing the bin counts from lowest to highest, then normalizing so that the range is 0.0 to 1.0 -- or 0% to 100%. It looks like this.  First as bars, then as a line.  It is the line format we will use most often.

As we progress to the analysis of risk and profit potential, we will be computing CDFs and making estimates from them. The math and interpretation are easier if we form the Inverse CDF. Imagine plotting the CDF on a sheet of paper, holding the CDF by its lower left and upper right corners, rotating front to back, and viewing the graph through the paper. It looks like this.

For the model to be useful in predicting the future, the distribution of trades in the future must be similar to the distribution found in the training and validation using historical data. There are statistical techniques for comparing distributions and estimating whether they are the same. We will be making those comparisons as we go along.

Population versus sample

As we use them, population and sample refer to data.  We can think of them as different views of a set of data -- different in the amount of the data that is visible to us. Population is the entire set of data. Sample is a subset of the population -- usually the specific subset that we are working with. Typically, the population is much larger than the sample and cannot be completely known.

We make estimates of characteristics of the population from measurements of the sample. For example, we might like to know the average height of 25 year old males in the US.  It is impractical to measure the height of every 25 year old male in the US -- the population. But we can measure the height of a subset of that population that we hope will be representative -- the sample. We infer the distribution of the height of the population from the distribution of the height of the sample. As you refer to statistical techniques, estimations of the population based on measurements of the sample are uncertain.  They depend on the techniques used in choosing the sample, and the size of the sample.

In some situations, particularly when models are created to describe a relationship rather than to predict, the population is completely known. An example is an analysis of who survived the sinking Titanic, a well-known data set used in illustrating machine learning. In these cases, there is no other data and the sample and population are the same. In trading, we are seeing only a portion of all trades that could result from our model. We are working with a sample and inferring characteristics of the population.

Given a population, many samples can be chosen from it. There will be variation among the samples.

The important point here is that whatever set of trades we observe from a system, that set is a sample, not the population.


Stationarity is a characteristic of a set or distribution, and it is with respect to some metric. A distribution is said to be stationary if similarly formed subsets have the same characteristic. Consider the price of stock, such as Apple, that has risen in price over time. The set of prices of Apple is not stationary with respect to the mean price over time. Financial data is typically not stationary with respect to price. If a trading system rule relied on price, it would fail, because the future distribution of price does not resemble the past distribution of price.

Price is not stationary, but that might not matter. Whatever signals we identify as preceding profitable trades must be stationary. That is the future that must resemble the past.

Applying techniques that assume the data is stationary when it is not will introduce an unfavorable bias.

I have posted a video to YouTube that might help further understand stationarity and its importance.

Choosing metrics

Putting these three concepts together, we can begin to choose metrics that are representative of the population, are stationary, and will be helpful in selecting good models.

Any single equity curve is formed from a sequence of trades in time order. We are processing price and volume data, using our rules to select the sample of trades associated with that price data, from which we will estimate characteristics of the population of trades, and estimate other possible sets and sequences of trades and equity curves. That is -- we assume the future set of trades will be similar to the past set of trades since they are drawn from the same population. But we cannot assume the same sequence, and we must be prepared for some variation.

Given the set of trades resulting from the training data, they might have occurred in a different order and we cannot assume characteristics of the set of trades that depend on the order in which the trades occurred. While final equity of a set of trades is independent of order, drawdown does depend on order, so drawdown over any single observed equity curve (such as from a backtest) is not a good metric. However, we could use the set of trades from that test, draw many equally likely sequences of trades from it, compute the equity curve of each, measure the drawdown of each, and compute metrics of the distribution of drawdowns. Whether that is possible or not depends on the capabilities of the development platform. For most traditional platforms it is not easily accomplished; for Python it is easy.

We have already seen that trade-by-trade position size belongs in the trading management system rather than the trading system. Metrics will be skewed if the trading system model has a position size parameter, so we will work with equal-sized trades. The percentage gained or lost per trade works well for shares; dollars gained or lost per contract works well for futures.

Some metrics that are useful and independent of sequence include:

  • number of trades 
  • accuracy -- percentage of trades that are winners
  • percentage gained or lost
  • ratio of average win to average loss
  • holding period
  • maximum adverse excursion

Next -- safe-f and CAR25


In the early days of technical analysis -- 1980s and early 90s  -- the markets were much less efficient than they are now.  Historical data was scarce and expensive, bid-ask spreads were wide, commissions were high, holding periods were long, and portfolios were in vogue.

Richard Dennis and William Eckhardt taught neophytes to trade the Turtle method.  Their rules fit on one page -- buy breakouts to new highs and short breakouts to new lows.   Chart readers watched moving averages and trend lines.  These techniques, and others like them, identified profitable trading opportunities visually and with with simple formulas.  

The confluence of affordable personal computers, downloadable historical data, technical analysis and charting software, lower commissions, and narrower spreads taken together lower barriers to entry, enabling many people to take up trading for fun and profit.  By the mid 1990s, the easy techniques had been discovered, exploited, and shared.  The markets became much more efficient, the trends shorter in length and profit.  Signals became weaker. 

Well funded trading houses hired graduate-level mathematicians, equipped trading rooms with powerful high speed work stations, and developed sophisticated software to detect ever fainter signals to trade ever more efficient markets.

As individual traders, beginning with our very first trade, we are immediately in competition with these professionals.  There are no challenger tournaments, no handicaps, no mulligans. 

Much as we we may wish it were otherwise, we must have high quality skills ourselves.  We must be proficient programmers and understand several branches of mathematics.  The Resources pages of this website will help you identify websites, videos, lectures, courses, articles, books, and software.  You will notice there is very little related to traditional technical analysis -- those methods have become less valuable.  There is quite a lot related to mathematics, modeling, simulation, statistical analysis, and computer programming -- these methods are increasingly important.   

In order to compete successfully and profitably, you need the analysis skills equivalent of a minimum of a rigorous four-year university program.  

Why use AmiBroker?

No matter which platform I chose, some people would be disappointed.  Some would prefer an English language description.  Some an artificial pseudo-langauge.  Some the language of only their favorite platform.  Some several of the most widely used languages.

One of the reasons I am publishing is to give system developers an alternative to imprecise trading advice, unrealistic promises, financial astrology, and smoke and mirrors.  The examples must be coded in the language of one of the popular trading system development platforms.  Readers must be able to replicate my results and use the examples as templates for further development and testing on their own.  I chose AmiBroker because:

  1. It is capable of performing everything I wanted to do (I do not know of another reasonably priced platform that can).
  2. It is free to try and inexpensive to buy.  The trial version can do everything I describe in the free Introduction to AmiBroker book.
  3. It has a very clear and concise language that should be easily understood by anyone who is reasonably comfortable programming in a modern computer language or for a different trading system platform.

You can get an idea of AmiBroker’s language:

  • From the sample chapters of my books (see the Books page of this website).
  • Every month, the trader’s journal Technical Analysis of Stocks and Commodities publishes the code corresponding to one of their articles for a dozen of the most popular trading system development platforms.  The AmiBroker code is usually the shortest, clearest, and easiest to understand.

For those people who are not already familiar with AmiBroker and who want to test the techniques I describe:

  1. Download the free trial version of AmiBroker from the AmiBroker website.
  2. Download and read the free Introduction to AmiBroker from that book’s website. It has complete instructions for installation, setting up databases, and using the platform in both charting mode and analysis mode.
  3. Paste in some of the AmiBroker code I provide and try it out.  For example, try this mean reversion system based on RSI.  It is reasonably stationary over a long period of time and works on many broad market and sector ETFs.  (It is for demonstration only, and is Not trading advice


// System settings
OptimizerSetEngine( "cmae" );
SetOption( "Initialequity", 100000 );
MaxPos = 1;
SetOption( "MaxOpenPositions", MaxPos );
SetPositionSize( 10000, spsValue );
SetOption( "CommissionMode", 3 ); // $ per share
SetOption( "CommissionAmount", 0.005 );
SetOption( "ExtraColumnsLocation", 1 );
SetTradeDelays( 0, 0, 0, 0 );
BuyPrice = Close;
SellPrice = Close;

// Parameters
// Good values have been: 2, 37, 5
RSILookback = Optimize( "RSILookback", 2, 2, 10, 1 );
EntryLevel = Optimize( "EntryLevel", 37, 1, 99, 1 );
ExitIncrement = Optimize( "ExitIncrement", 5, 1, 99, 1 );

// Indicator
RSIValue = RSIa( C, RSILookback );

// Rules

// Buy when RSI falls through a critical level
Buy = Cross( EntryLevel, RSIValue );

// Sell when RSI rises
ExitLevel = EntryLevel + ExitIncrement;
Sell = Cross( RSIValue, ExitLevel );

// end //

AmiBroker is used extensively in the Quantitative Trading Systems book and the Mean Reversion book. The code for both those books is available for download, copy, and paste. Whichever platform you decide to use, please use sound modeling and simulation techniques to give realistic estimates of future performance of your system. Use the risk-normalization procedures fully explained in Modeling Trading System Performance to compare alternative uses of your funds.


Blue Owl Press website is being revised

We are in the process of renovating our website. 

The pages for Postings will have a series of entries outlining trading system development using the scientific method.  Several entries have been posted, and more will be added.  The trading system development postings are at the beginning of the list and are in order by recommended reading sequence. 

The pages for Videos are complete.

The pages for Books, including the downloadable material, are complete.  

The pages for Papers has a few entries, and more will be added.

The pages for Resources are very new.  The entries that will appear on these pages will include extensive reference material -- books, videos, courses, articles.