radical lexer

[Boris 'pi' Piwinger]

David Relson <relson at osagesoftware.com> wrote:

>> Also we differ on BOGOLEX_TOKEN, where I don't allow =()+
>> which you do allow. I really don't know if this is
>> important. Can you explain why?
>
>Good question!  BOGOLEX_TOKEN should allow the characters permitted in
>a TOKEN _plus_ colon and asterisk (which are used for header tokens and
>multi-tokens).
>
>Another patch attached...
>
> TOKENFRONT	[^[:blank:][:cntrl:][:digit:][:punct:]]
> TOKENMID	[^[:blank:][:cntrl:]<>;=():&%$#@+|/\\{}^\"?*,\[\]]*
>-BOGOLEX_TOKEN	[^[:blank:][:cntrl:]<>;    &%  @ |/\\{}^\" *,\[\]]+
>+BOGOLEX_TOKEN	[^[:blank:][:cntrl:]<>;    &%  @ |/\\{}^\"  ,\[\]]*
> TOKENBACK	[^[:blank:][:cntrl:]<>;=():&%$#@+|/\\{}^\"?*,\[\]._~\'\`\-]

So you still do allow =()$#+? which are all not allowed in
TOKEN. Could the also be removed?

pi