lexer change

[Boris 'pi' Piwinger]

Tom Anderson wrote:

>> make too much sense. So my question is more if we need the
>> dollar case (not allowing 123.45$ or $100,000 at the same
>> time). Or if we need it why only this special case of a price?
> 
> Why would we be excluding these in the first place?  If it has a
> non-numeric in it, then it shouldn't fit the all-numeric rule,

We require that a token begins with a character from the
following class: [^[:blank:][:cntrl:][:digit:][:punct:]]
So a $ is excluded.

> which probably shouldn't even be there in the first place.  Unless a rule is
> absolutely required, it shouldn't be institued.  The default action
> should be to use Bayesian filtering, not rules.  In short, yes, all
> prices should be evaluated, as they could very well be good indicators.

My test yesterday actually showed it does not help to allow
those tokens.

pi