• 
• 
• 

• Performance
• Services
• Company
• Learn About
• My Attain

Synergy / Checkmate / Fusion Developer Dean Hoffman on measuring Start Trade Performance

September 20, 2004

 

What do you do as a developer when two different customers of yours trading the same system are showing drastically different results? When you're Dean Hoffman, developer of Synergy, Checkmate, and Fusion, you tackle the statistics. You explain that performance will vary depending on what date you started trading the system, and show the start trade drawdown and following 12 months performance for hundreds of different starting dates.

Mr. Hoffman is the only developer we work with who goes to such extremes to educate his customers on these vital "start trade" statistics, and we asked him to elaborate on start trade performance in this week's newsletter. The following report was written by Dean Hoffman of Strategic Trading Solutions, www.traderstech.net.

By Dean Hoffman:

One of the most perplexing performance reporting problems facing system researchers involves testing start dates. The date you start testing a system from can have a profound difference on the overall results. For example, assume you chose a historical starting date that immediately preceded a series of winning trades. This could make the system appear as though it needed very little initial capital, as you traded with the markets profits almost immediately. This could make the return on required capital look astronomical.

However, if you tested the same system from a historical date that immediately preceded a series of losing trades - the system might have looked terrible. In other words, you had one system that had 2 completely different results depending on which date you starting testing (or trading) from. This is problematic in that it could allow somebody to "cherry pick" an optimal historical starting date (either accidentally or purposely), making their system appear much better than it actually is.

This is a particular problem with systems that use risk filters and position sizing. Assume you are an investor with $100,000 to invest and are willing to risk up to 2% of your equity on any given trade. This means that a trade with $3,000 in risk would initially be rejected. If your account was up to $150,000 later in the year, however, that same $3,000 trade would now be allowed based on 2% of your account size.

Nevertheless, a new investor that started with $100,000 on the date your account was at $150,000 would still not get the same trade, as the risk remains too high for them. If that trade turned into a huge winner, only one account would actually have the trade in it, eventhough the system is in that position. So during the exact same time period, you can witness two accounts with different percentage profits and losses even though they were trading the same system and portfolio etc. This can be disconcerting to say the least.

The types of scenarios outlined above are endless. The bottom line is that when you are using position sizing with your trading (as you should), the date you start testing (or trading) can have an enormous impact on your results. The question is how can you quantify this and see the impact during testing?

The answer is to run a start trade report. A start trade report repeatedly tests the same system over and over again, from every possible point in history that a new trade could have been taken. In other words, if the system had 1000 trades over the last 10 years, then the system will be tested 1000 times, one test for each possible starting date, i.e the date the system entered each new trade. In addition, the equity is set back to the original amount on each of those runs so that growing or declining equity does not skew your results by assuming you would have taken trades that in reality your position sizing and money management would have rejected.

Let's use an example from one of our systems, Fusion. We ran a start trade report on the system using 21 markets, while assuming $100,000 in starting equity and risking 2% of equity per trade with a permission filter to take a minimum of one contract up to a risk of $2,500. Let's look at some dates and examine some of the extreme outcomes.

Had you started on 1/22/1997 you would have seen your $100,000 account dip to $81,729 at one point and finished your first year with a loss of -$5,189.

However, had you started on 11/20/1989 you would have only seen your $100,000 dip to $99,325 at one point and finished your first year with a profit of $237,604.

The spread in performance here is enormous. What if you had to make a decision based on seeing only one of the 2 results above? By seeing only one start date that's essentially what you're doing. Let's use another example in the same start trade report.

Had you started on 12/06/1989 your test results would have shown first year profits of $196,178.

Had you started on 12//11/1989 your test results would have shown first year profits of (only) $97,782

The testing dates in the above example were only 5 days apart yet there was almost a 100% difference in the results! Can you imagine starting to trade the same system as someone else at about the same time and having a 100% difference in results?

The start trade report allows you to see ALL of the possible start date outcomes, even if you had started on the worst possible date in history. What you end up with after running a start trade report is enough data to form a frequency distribution. This allows you to see the best, worst, and average results as distributed over hundreds or thousands of starting dates (verses just ONE starting date). The test then shows the 12 month performances from each one of those tests.

The chart below shows the frequency distribution of the profit/loss for the 12 month period immediately following over 2000 different starting dates. As you can see, the maximum first year profit was $237,604 and the minimum was -$5189. The average was $54,224.

Next let's look at start trade drawdowns. Start trade drawdowns are the amount you went under your original starting amount. This is different than the maximum equity drawdown. Many people are able to "stomach" giving back a larger percentage of equity if its profit verses if it's their original starting capital. For this reason we separate the two.

The second chart below shows a frequency distribution of over 2000 different start trade drawdowns resulting form different start trade dates. Here you can see that the maximum start trade drawdown was -$24,248 and the average start trade drawdown was -$4,847. There were also start trade drawdowns of $0, this means that your first trade started out profitable and you never went under your starting equity amount. (Wouldn't that be nice)

By looking at the frequency distributions you see that the average first year profit was $54,224 and the average start trade drawdown was -$4,847. This average was generated with over 2000 different starting dates. Having several thousand data points to average over creates a very robust perspective in our opinion.

This is just a sample of what can be done with start trade reports. The full list would be too difficult to include in a brief article such as this. However, feel free to contact myself or the folks at Attain Capital for the complete start trade report.

Sincerely,

Dean Hoffman www.traderstech.net