lexer change

[Tom Anderson]

On Tue, 2003-11-11 at 09:12, David Relson wrote:
> The rules determine how the message is parsed and tokens get created. 
> After that, the bayesian technique is applied to the tokens.

When the rules exclude otherwise perfectly good tokens, they are
essentially acting as a pre-filter.  Excluding 10k binary attachments
would be an example of a good rule because it has a major impact on
program/database efficiency.  Excluding a token with a dollar sign is
not, since Bayes can do the filtering on that just fine.

Tom

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://www.bogofilter.org/pipermail/bogofilter/attachments/20031111/9b4e5cd5/attachment.sig>