Bug in tuning.sh?

[Greg Louis]

On 20030613 (Fri) at 1223:33 +0200, Boris 'pi' Piwinger wrote:
> Greg Louis wrote:
> 
> > Try this:
> > 
> > --- tuning.sh.orig	2003-06-12 12:45:04.000000000 -0400
> > +++ tuning.sh	2003-06-12 12:54:59.000000000 -0400
> 
> Now here is the result:
> 
> > Top 10 results
> > 06/13 11:24:55 1e-1   0.475 fpos.. 8 at cutoff 0.499000, run0..  21  run1..  21  run2..  25   67
> > 06/13 10:57:34 1      0.450 fpos.. 8 at cutoff 0.500000, run0..  42  run1..  38  run2..  47  127
> > 06/13 10:58:20 1      0.475 fpos.. 8 at cutoff 0.500000, run0..  41  run1..  38  run2..  47  126
> > 06/13 11:11:45 3.2e-1 0.450 fpos.. 8 at cutoff 0.500000, run0..  42  run1..  38  run2..  47  127
> > 06/13 11:12:25 3.2e-1 0.475 fpos.. 8 at cutoff 0.500000, run0..  41  run1..  38  run2..  47  126
> > 06/13 11:24:15 1e-1   0.450 fpos.. 8 at cutoff 0.500000, run0..  42  run1..  38  run2..  47  127
> > 06/13 11:36:44 3.2e-2 0.450 fpos.. 8 at cutoff 0.500000, run0..  42  run1..  38  run2..  47  127
> > 06/13 11:37:23 3.2e-2 0.475 fpos.. 8 at cutoff 0.500000, run0..  41  run1..  38  run2..  47  126
> > 06/13 11:49:15 1e-2   0.450 fpos.. 8 at cutoff 0.500000, run0..  42  run1..  38  run2..  47  127
> > 06/13 11:49:54 1e-2   0.475 fpos.. 8 at cutoff 0.500000, run0..  41  run1..  38  run2..  47  126
> 
> Again, I don't believe those results:
> 
> 1) Why do all have 8 false positives? That is a lot. Where
> is this chosen (getco, right?), why?

Ok, this looks much better.  24 is far too high a target for this data
set, which is why all your cutoffs are 0.5.  Try setting a target of 6
(there's a target= line early in the script).  The new bogotune will
evaluate your data and set an appropriate target; tuning.sh requires
that it be done manually.

One runs tuning.sh to seek the best s and min_dev values, not cutoff. 
You need a false-positive rate higher than would be desirable in
production to ensure that the cutoffs don't bottom or top out over the
range of s and min_dev that's being investigated.  The results are only
valid when the cutoff is exactly the target value, so having them all
the same is desirable.  Once you have the right s and min_dev, you
increase the spam cutoff till you have the desired balance (bogotune will
offer some choices) between fp and fn.

> 2) Again, no significance to robs:

There may really be none for your data, but until we get reasonable
cutoffs (above 0.5 and below 0.999999), I wouldn't hasten to conclude
anything.

> What is BTW the reason for making these three runs all the
> time instead of just one bigger one?

Largely historical, but also, if you do get a small effect of s or
min_dev, you can check its statistical significance with two or three
runs, but not with one.

-- 
| G r e g  L o u i s          | gpg public key: finger     |
|   http://www.bgl.nu/~glouis |   glouis at consultronics.com |
| http://wecanstopspam.org in signatures fights junk email |