What is a word (lexertest)

[David Relson]

At 06:59 AM 10/22/02, Boris 'pi' Piwinger wrote:

>Hi!
>
>Even though I don't code here, I tested something;-)
>
>[3.14 at pi ~/local/bogolists]$ echo "»cmsg newgroup«"|lexertest
>get_token: 1 '»cmsg'
>get_token: 1 'newgroup«'

I'm guessing that you wanted the two special characters removed???

>[3.14 at pi ~/local/bogolists]$ echo "bla"|lexertest
>
>Both results are not really satisfiying. There might be a reason why
>the second does not return anything, but the first is wrong. Well,
>here we have the problem that we cannot tell without looking at the
>charset.

As said in an earlier message, simple words (only letters and digits, no 
special characters) that are alone on a line are skipped by the current 
lexer.  Hopefully our lexer expert (Clint) can give us a fix.

David