Scoring Parameters - Old vs New

[David Relson]

Greetings,

For the past few days I've been working on a "good" set of tests to
compare the following sets of scoring parameters:

    old - what bogofilter's been using approx Dec 2002.
    osa - the parms in use at osagesoftware.com
    new - the results of Greg's recent (very large) bogotune run

I wanted to see how the new parameters compare to the other parameter
sets.  My goal was more to see if they're reasonable.  I'm less
concerned with whether they're a bit better or a bit worse because
I've seen that different sites do very well with very different
parameter sets.

For testing I used the 80,500 ham and 67,500 spam accumulated in the
18 months I've been running bogofilter.  I was curious about the
effects of full training vs. a small training vs. a large training
set:

    small - trained with 10% and scored 90%
    large - trained with 90% and scored 10%
    full  - trained and scored all messages

For each of the 3 tests, I used the 3 parameters sets.  I also tested
using the new parameters with spam_cutoffs of 0.90, 0.80, and 0.70.

Note: the columns in the "Classification Accuracy" parts are:

    ver - config version
    hh  - ham scored as ham
    hu  - ham scored as unsure
    hs  - ham scored as spam -- false positives
    sh  - spam scored as ham -- false negatives
    su  - spam scored as unsure
    ss  - spam scored as spam

### Parameters ###

     robs     robx    min_dev  spam_co  ham_co
old  0.010    0.415    0.100    0.950    0.100
osa  0.0178   0.549138 0.435    0.501    0.376
new  0.0178   0.520    0.375    0.990    0.450
0.90 0.0178   0.520    0.375    0.900    0.450
0.80 0.0178   0.520    0.375    0.800    0.450
0.70 0.0178   0.520    0.375    0.700    0.450    

### small test ###

spam 7502,  good 8938

Classification Accuracy:
ver      hh     hu     hs     sh     su     ss
old   79112   1229     48    141   2690  64587
osa   78880   1405    104    112   1110  66196
new   79193   1180     16    156   4438  62824
0.90  79193   1169     27    156   3088  64174
0.80  79193   1162     34    156   2583  64679
0.70  79193   1156     40    156   2194  65068

### large test ###

spam 67532,  good 80530

Classification Accuracy:
ver      hh     hu     hs     sh     su     ss
old   79982    502     12     50    479  66947
osa   80130    339     27     43    220  67213
new   80176    317      3     48    773  66655
0.90  80176    316      4     48    547  66881
0.80  80176    313      7     48    461  66967
0.70  80176    310     10     48    409  67019

### "full" test ###

spam 74920,  good 89327

Classification Accuracy:
ver      hh     hu     hs     sh     su     ss
old   80017    476      3     40    435  67001
osa   80211    277      8     33    217  67226
new   80234    262      0     39    660  66777
0.90  80234    262      0     39    464  66973
0.80  80234    262      0     39    393  67044
0.70  80234    261      1     39    359  67078