X-Bogosity header?

[David Relson]

At 09:58 AM 5/5/03, Fletcher Mattox wrote:

>Hi.  Why does the location of the X-Bogosity header have such a dramatic
>impact on bogolexer?  In x1 it is the last header, but in x0 it has been
>swapped with the header before it.  Otherwise the files are identical.
>Yet, bogolexer finds over 3 times the number of tokens in x1.
>
>Thanks
>Fletcher

Fletcher,

Interesting!  Looks like a bug.

To what's different, I ran:

bogolexer -p < x1 > x1.out
bogolexer -p < x2 > x2.out
gtkdiff x1.out x2.out

It looks the lexer is returning the tokens from inside html tags.  0.11.1.5 
has the same problem, while 0.10.3.1 gives the same counts.

I'll take a look and see what's going on.

David