[PATCH] Better tagging.

[Boris 'pi' Piwinger]

Matthias Andree wrote:

>>> I'd support removing ALL case folding and case blindness from tokens, as
>>> has been pointed out, the naive Bayes stuff needs as much information as
>>> possible, and we've been proven more than once that case matters even if
>>> we then have to plan for 16 variants of "date" and "dAtE" in the data
>>> base...
>>
>> BTW: How about the limitation abour short tokens? Matthias
>> also gave good arguments about that.
> 
> Do you have a Message-ID at hand?

No, but ...

> IIRC, the token size limitation was primarily a measure against base64
> and "rulers" and stuff.

... it was about short tokens like single letters.

pi