lexer speedup.

[David Relson]

At 09:34 PM 7/19/03, michael at optusnet.com.au wrote:

>While I'm playing with the lexer:
>Variable length look-ahead has a noticable
>speed penalty in flex. This patch removes
>a variable-length look-ahead and speeds
>up discarding loose comments.
>
>No functionality change. Just speedups.
>
>(Good for about 8% lower CPU usage for me).
>
>Michael.

Hi Michael,

I've started the testing process for this patch.  Results are mixed.

Here's the current status:

1 - "make check" works
2 - speed test (with 4608 spam, 3479 ham)
     old - 70.21s user 1.25s system, 1:22.46m elapsed, 86% CPU
     new - 49.18s user 1.01s system, 0:56.87m elapsed, 88% CPU
3 - scoring test
     2934 messages score the same (either 0.000000 or 1.000000)
     5143 messages score differently

Next step:  find out why the scores are different.

David