token pairs [was: Algorithm limitations]

[Boris 'pi' Piwinger]

David Relson <relson at osagesoftware.com> wrote:

>Scanning the message happens as normal.  Each token seen is returned for
>scoring.  Additionally, a token pair is created using each token and its
>predecessor (with a colon separating them).

There is the risk that this creates a token pair which looks
like a tagged entry, since both use a colon.

pi