[PATCH] Better tagging.

[Boris 'pi' Piwinger]

Matthias Andree wrote:

> I'd support removing ALL case folding and case blindness from tokens, as
> has been pointed out, the naive Bayes stuff needs as much information as
> possible, and we've been proven more than once that case matters even if
> we then have to plan for 16 variants of "date" and "dAtE" in the data
> base...

BTW: How about the limitation abour short tokens? Matthias
also gave good arguments about that.

pi