I'm trying to see if there is any interest in applying scientific
method to trading systems development by comparing predictive
software and architectures in a systematic way.
This is perhaps similar to what Consumer Reports does. The take say
SUVs and drive them, crash them, measure various things and publish
the results. Would this be of value to investors using predictive
software models to predict and trade the market?

If so and you'd like to participate either respond on

http://tech.groups.yahoo.com/group/Neural_Networks_and_Knowledge_Disco
very/

or contact me privately.

Note: some of the vendors of trading tools software seem to really
hate the idea.


Part of the discussion on NNKD follows:

"I've had the idea of creating a test bed data set and standardized
benchmarks for some time.

Here is how the idea would work:

1) Create several price data sets:
a) For example USD/YEN with daily bars and 1 hour bars.
b) The same for a Fortune 100 stock or the DJIA itself
C) The same for a CBOT future contract

2) Create the same Dependant Variables (DV) for the modeling process.
a) Oscillators - there are a bunch we all use
b) Trending indicators - MACD, etc
c) Price pattern indicators - candlestick patterns, etc

3) Have everybody who wants to participate run all sets through their
modeling process. In addition to NN we have folks here using many
other modeling methods.

4) The results would give us a kind of traders' "consumer reports" on
modeling methods, modeling software and model designs.

This could be educational for everyone and perhaps very valuable to
the average trader. Knowing that the same data run through the same
software package, NeuroShell for example, by two different people
gave radically different results, one useless and one tradable would
indicate that differences in internal set up can be the difference as
opposed to the data or software. This may be of value for those
learning complex processes and software configurations.

Differences between software applications would be even more
interesting.

Let's say we have four popular modeling packages:
A) 2 are actively marketed to investors for trading system
development
B) 1 is a general purpose application used by a wide range of data
miners
C) 1 is a GPU package that is free to everyone

It would be a heck of a thing to find that trader package A which
cost say $2000 dollars produces inferior results on all data sets to
trader software B costing say $400. Knowing this could help the
average trader/consumer avoid wasting time and money on a "boat that
will never float".

Among the other possible discoveries: general purpose packages
working better than the trader software, free GPU software working
better than commercial software or visa versa.

What does anyone/everyone think of the idea?

If we did it right, to academic standards we could probably be the
results published in some of the trader magazines....ahh 15 seconds
of fame plus a great service to the consumer and pressure on software
vendors to create better software.

I suggested this idea to the owner of a software company that markets
predictive software designed for traders. As the rather extensive
discussion was private the software company and his specific words
will remain private. The guy is a friend of mine as I've been buying
his software both personally and for corporations I consult with for
over 12 years.

However, the summary of the conversations is that he spent an
incredible amount of effort trying to convince me that this idea is
impossible, unworkable, fraught with peril, a waste of time, too
complex, too time consuming and would provide comfort to the Islamic
terrorists (lol on the last reason).

In the end he just broke off communication. I can understand his
position from a business perspective. He is in the business of making
money from investor by selling them software using their hope to make
money with it and not the business of helping traders actually make
money.

Anyone have any comments?

Anyone willing to participate?

Since in most newsgroups there are many willing to listen and learn
from others posts and discussion while offering nothing of their own,
we would need a mechanism to reward those who would take the time to
participate with the effort of creating standard data sets and
running the tests. Don't know what this is, but a motivator would
increase involvement."