lexer speedup.

[David Relson]

Michael,

Good news.  My previous test, in which I reported different spam scores, 
was flawed.  I was comparing the performance of bogofilter using separate 
spam and ham wordlists to bogofilter using a combined spam/ham 
wordlist.  Unfortunately, the combined and separate wordlists didn't have 
comparable content.  Using the same spam and ham wordlist gives the same 
spam scores and the following times:

scoring 4608 spam, 3479 ham
	 old: 140.50s user, 2.02s system, 2:36.17m elapsed,  91% CPU
	 new: 134.92s user, 2.23s system, 2:31.19m elapsed,  90% CPU

So the new code checks out as faster - approx 4% on my machine.

Thank you for the improved code.  I'll be committing it to cvs in the next 
few minutes.

David
--------------------------------------------------------
David Relson                   Osage Software Systems, Inc.
relson at osagesoftware.com       Ann Arbor, MI 48103
www.osagesoftware.com          tel:  734.821.8800