[PATCH] experimenting with new parsing rules

[David Relson]

On Tue, 16 Sep 2003 01:41:47 +0200
Matthias Andree <matthias.andree at gmx.de> wrote:

> David Relson <relson at osagesoftware.com> writes:
> 
> > As to tagging of "Return-Path:" vs "Received", I dropped a few lines
> > of code.  I've fixed that.  Attached is the fix (as a patch).  I've
> > also attached the current lexer_v3.l (so you needn't worry about
> > patch conflicts).
> 
> Is this really code that implements Michael's suggestions? I may be
> mistaken or I may be unaware of how to use it, but I don't see "h:"
> prefixed to header tokens:
> 
> $ echo "Testheader: test" | BOGOTEST=2 ./bogofilter -Ph -vvv
> X-Bogosity: No, tests=bogofilter, spamicity=0.038938, version=0.15.3,
> date=2003-09-15T23:41:09Z
>                                      n    pgood     pbad      fw     U
> "test"                              16  0.097403  0.003922  0.038938 +
> "Testheader"                         0  0.000000  0.000000  0.415000 -
> N_P_Q_S_s_x_md                       1  9.61e-01  3.89e-02  3.89e-02
>                                         1.00e-02  4.15e-01  0.100

Matthias,

With the inclusion of the BOGOTEST code to CVS this morning, I changed
BOGOTEST=1 to mean "enable lexer debug mode" (since the debug code is
now present, by default) and BOGOTEST=2 to enable Michael's changes
(which are present for testing purposes).  

The changes have been modified somewhat by me.  Consistent with the rest
of bogofilter, the header prefix is now "head:".  

Remember, too, that bogofilter's current charter is scoring single
tokens to produce a message score.  Combining token sequences (pairs,
triples, etc) may happen in the future, but now is not the time. 

Since special characters (such as spaces, equal sign, quotes, ...) which
are presently excluded from bogofilter's tokens will continue to be
excluded.  I'm not sure if, in all cases, the test rules comply with
this exclusion.  If they don't, I'll modify them.

David