[patch] small lexer changes

[Mark M. Hoffman]

* Mark M. Hoffman <mhoffman at lightlink.com> [2002-10-07 18:27:45 -0700]:

> Flex builds a table-driven state machine.  Here's how to keep it fast:
> 
> http://www.gnu.org/manual/flex-2.5.4/html_chapter/flex_18.html#SEC18

Following up on my own suggestion, I discovered that bogofilter's scanner
isn't nearly as fast as I thought it was.

> Therefore, moving those html tokens from lexer.l to an ignore list
> *cannot* be faster, even though the resulting scanner (lexer.c) will
> be smaller.

OK strike that... moving the HTML tokens to an ignore list *won't* be
faster, at least not after I'm done with the lexer. ;)

Regarding the size of lexer.c... change "%option full" to "%option ecs" on
line 305.  That reduces lexer.c from 47596 lines to 4894 lines for me.
The scanner isn't ready to take advantage of "%option full" anyway, yet.
If there are no objections, I'll commit that later today.

Regards,

-- 
Mark M. Hoffman
mhoffman at lightlink.com