Minimal rules [was: Test with different lexers]

[David Relson]

Tom and pi,

If you're really intent on simplifying the lexer, reduce the rules to a
single rule which uses only whitespace for delimiters.  That would
indicate what happens without any of the current special rules for
processing html, ignoring binary attachments, tagging header lines.  I'd
be interested in hearing how much smaller it is, how much faster it is,
and what its scoring performance is.

David