tilde (was: [cvs] bogofilter/src lexer_v3.l,1.125,1.126)

[David Relson]

On Thu, 11 Dec 2003 14:11:52 +0100
Boris 'pi' Piwinger <3.14 at logic.univie.ac.at> wrote:

> David Relson wrote:
> 
> >> >> Exclude tilde from tokens.
> >> 
> >> Everywhere? Or just TOKENBACK?
> > 
> > I changed it in TOKENMID and TOKENBACK.  What did you want?
> 
> Looking at the history, tilde, apostrophe and backtick were
> forgotten when expanding [:punct:] to allow exclamation
> points in TOKENBACK:
> http://cvs.sourceforge.net/viewcvs.py/bogofilter/bogofilter/src/lexer_v3.l?r1=1.23&r2=1.24

Likely so.  The expansion took several iterations to get it right (or so
we thought).

> Tilde was allowed in TOKENMID already in revision 1.1. 

The revision number is somewhat misleading.  When the source files were
moved from bogofilter/ to bogofilter/src, all their revision numbers
became 1.1.  Before that, there was a lexer.l file which underwent major
changes and was renamed to lexer_v3.l.  So there were many earlier
revisions.

> I believe the reason are TOKENs like F~R~E~E. That makes
> sense, but I don't have a strong opinion here.

Unfortunately there are also "F*R*E*E", "F R E E", etc.  Given how few
tokens contain tildes, I doubt it matters whether they're included or
excluded.

> I meant only TOKENBACK where it is a mistake, but we
> certainly can think about TOKENMID.