small lexer change

[David Relson]

Just a heads up...

In 0.7.5, the token pattern uses only two control characters, '\n' and '\r' 
as token delimiters.  I have changed the three occurrences of \n\r to 
[:cntrl:] so that all control characters will be token delimiters.  For 
example, my word list used to contain "^T^T^T^T.co.kr" as one of its 
tokens.  Having rebuilt the word lists, I now have "co.kr" which is better.

David
--------------------------------------------------------
David Relson                   Osage Software Systems, Inc.
relson at osagesoftware.com       Ann Arbor, MI 48103
www.osagesoftware.com          tel:  734.821.8800