Hi !!

Happy New Yr. !!

I have developed an ANN. I used the RMSE calculated on a test data set

to identify the best architecture. Now I need to do some further

studies on this ANN. Actually I need to compare the performance of

this ANN with another ANN.

Initially I thought of using a t-test for this. But to use a t-test

the error should go in a normal dist. Am I correct?

So I collected my ANNs’ error values for some test data sample.

Unfortunately the error values do not fall in to a Normal Dist.

1. Does this mean that my ANN is not doing well?

2. Is it a problem of the test data set?

3. Are there any other statistical measures I can use for the

comparison of the two ANN?

Thanks.

muditha

Simple.

Before comparing NNs, make sure each is designed as well as possible.

For example, are the errors randomly distributed with zero average?

If not, the NN has failed to extract trend information and more

learning, perhaps with more hidden nodes is indicated. Plot error

vs target value. Consult a statistics handbook and apply a test to

quantify randomness (e.g., runs test).

You indicate below that you are concerned about the possible

nonnormality of the error distribution. My reply is:

You have commanded your NNs to minimize SSE (and nothing else).

If your error distribution is random with zero error, what

difference does it make if it's nonnormal?

Now, let's compare NNs. Aside from training time and number

of weights, common sense leads to:

1. Which NN has the lowest SSE?

2. Is the difference in SSE statistically significant?

3. A statistician can refer you to the appropriate test.

My guess is an F-test.

4. Is the difference in SSE practically significant?

It looks like you are trying to test if the difference in the

average errors is statistically significant not whether the

difference in SSE is. Or

5. Which NN has the lowest bias?

6. Is the difference in bias statistically significant?

7. A statistician can refer you to the appropriate test.

My guess is a t-test.

8. Is the difference in bias practically significant?

Note: A significant bias indicates an inadequate design.

For example, not enough hidden nodes. Reread my

first paragraph.

As indicated above, for a given SSE and zero bias, nonnormality

is a red herring. Besides, even if the biases are nonzero,

the t-test is relatively insensitive to nonnormality.

Probably not.

Who said there is a problem?

As far as comparing the two NNs, you can perform statistical tests

on the distribution of the differences in outputs.

I think the most important comparison test is a plot of SSE1 and

SSE2 vs noise level as zero mean Gaussian noise is added to the

inputs. Well, since the NNs are nonlinear, you might want to plot

biases too.

Hope this helps.

Greg

-----SNIP

Whoops! I meant zero mean.

Hope this helps.

Greg

1. Ant: To dist or not to dist...

2. VB Works in test mode, but Stops when doing normal run.

3. SAS 8.2 (Base) test is available for Beta testing-BETA TESTING IS DONE

4. random number from normal dist

5. Normal Dist area under curve

6. Stixbox and Inverse Normal dist.

7. Doing a "Right Click New Word Document" does not use Normal templa

8. "Matching" column A's values in column B

9. On Open event does not fire when doing DoCmd.OpenReport in normal

10. Finding the Value (of text and numbers) between N/A's

11. Trend test p-value vs. heterogeneity test p-value in PROC MIXED

12. Freq.Dist table...can it be done?

13. Averaging a series of written text grades i.e. Dist*, Dist, Merit, Pass and Fail

14. Undefined variable or function Dist might refer to the function dist.

15. php-4.4.0 and test.wsf = Error: The value for the attribute is not valid: language

3 post • Page:**1** of **1**