Radical lexers

[Boris 'pi' Piwinger]

Tom Anderson wrote:

>> And even more extreme. Tokens are explicitely: [[:alnum:]]+
> 
> How about a d) test where tokens are explicitely [^[:blank:]]+, or in
> other words, anything not a space.

I tried, but it does not work. I am testing a variation now.

>> c) is really mind-blowing. This simply MUST NOT work.
> 
> That is a surprise to some extent, but remember that the current regex
> is pretty restrictive already.  

Not really.

> Making it just a little more restrictive
> probably won't apply to most tokens anyway.

It actually does. I have tons of words with
non-ASCII-characters.

pi