[PATCH] experimenting with new parsing rules

[David Relson]

On Tue, 16 Sep 2003 01:41:47 +0200
Matthias Andree <matthias.andree at gmx.de> wrote:

> David Relson <relson at osagesoftware.com> writes:
> 
> > As to tagging of "Return-Path:" vs "Received", I dropped a few lines
> > of code.  I've fixed that.  Attached is the fix (as a patch).  I've
> > also attached the current lexer_v3.l (so you needn't worry about
> > patch conflicts).
> 
> Is this really code that implements Michael's suggestions? I may be
> mistaken or I may be unaware of how to use it, but I don't see "h:"
> prefixed to header tokens:
> 
> $ echo "Testheader: test" | BOGOTEST=2 ./bogofilter -Ph -vvv
> X-Bogosity: No, tests=bogofilter, spamicity=0.038938, version=0.15.3,
> date=2003-09-15T23:41:09Z
>                                      n    pgood     pbad      fw     U
> "test"                              16  0.097403  0.003922  0.038938 +
> "Testheader"                         0  0.000000  0.000000  0.415000 -
> N_P_Q_S_s_x_md                       1  9.61e-01  3.89e-02  3.89e-02
>                                         1.00e-02  4.15e-01  0.100

As a bit more info, set_tag("header") isn't called until the first '\n'
is encountered.  Hence the first line isn't tagged.

Try the following test:

export BOGOTEST=2
cat | bogolexer -p <<EOF
Head1: word1
Head2: word2
EOF