radical lexer

[David Relson]

On Sun, 26 Nov 2006 20:07:57 +0100
Boris 'pi' Piwinger wrote:

> David Relson <relson at osagesoftware.com> wrote:
> 
> >A quick comparison of bogofilter's lexer_v3.l and your radical lexer
> >was interesting, particularly the following line:
> >
> >TOKENBORDER [^[:blank:][:cntrl:]<>;&%@|/\\{}^"*,[\]=()+?:#$._!'`~-]
> 
> Also we differ on BOGOLEX_TOKEN, where I don't allow =()+
> which you do allow. I really don't know if this is
> important. Can you explain why?

Good question!  BOGOLEX_TOKEN should allow the characters permitted in
a TOKEN _plus_ colon and asterisk (which are used for header tokens and
multi-tokens).

Another patch attached...
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: patch.1126.BOGOLEX_TOKEN.txt
URL: <http://www.bogofilter.org/pipermail/bogofilter/attachments/20061126/27552f67/attachment.txt>