boundary tokens

[Matthias Andree]

David Relson <relson at osagesoftware.com> writes:

> I was looking at my wordlists and noticed that, right at the beginning
> of each (as dumped by bogoutil) there are a lot of "boundary" symbols
> (as returned from lexer.l).
>
> Approx. 99% of them have counts of 1.  The longest is 94 characters.  My
> spamlist has 1050 of them and goodlist has 2200 of them.
>
> I don't think we need them.  What would happen if lexer stopped returning them?

Your test might break, other than that, it should be safe.

The boundary parser can be reinstated easily later if you make a single
"remove-boundary-return" commit to CVS.

-- 
Matthias Andree