radical lexer

[Boris 'pi' Piwinger]

Boris 'pi' Piwinger <3.14 at piology.org> wrote:

>>A quick comparison of bogofilter's lexer_v3.l and your radical lexer
>>was interesting, particularly the following line:
>>
>>TOKENBORDER [^[:blank:][:cntrl:]<>;&%@|/\\{}^"*,[\]=()+?:#$._!'`~-]
>
>Also we differ on BOGOLEX_TOKEN, where I don't allow =()+
>which you do allow. I really don't know if this is
>important. Can you explain why?

I just created a new version:
http://piology.org/bogofilter/lexer_v3.l.radical.current

This diffs much nicer against David's latest version. The
only real differences are:
1) I adjusted the ID as discussed.
2) I rearranged TOKENCHAR (formerly known as TOKENBORDER;-)
to fit the order of the standard version.

The only open point: What is really required for
BOGOLEX_TOKEN? In the standard version it allows the
following additional characters compared to TOKENMID:
=():$#+?

Why is this? What should it really do in the end?

pi