lexer changes

[Boris 'pi' Piwinger]

David Relson wrote:

> I've tested your patch and "make check" FAILs many of the tests.  The
> patch will not be applied.

Of course it fails, since new tokens are introduced. Without
that it passes (attached).

>> 0) Allowing two-byte-tokens (see my test on the other list)
> 
> Not allowed, for reasons I've previously stated.

I cannot see a reason. Your test did not show the false
positives, where this change brings the major improvement.
Also targets are missed and different targets show one
method ahead or the other. So I did a much more detailled test.

>> 1) Some \ slipped back in. Out again.
> 
> None of them "slipped" in. 

Actually in
<20031104092536.1d799059.relson at osagesoftware.com> you had
some out which are now back in. Example:
+TOKENBACK	[^[:blank:]<>;=():&%$#@+|/\\{}^\"?*,[:cntrl:][\]._+-]

> I'm satisfied with those that are there and
> don't see a need for spending time discussing them or removing them.

I spent the time (three hours for cleanup alone) already.

>> 3) The comment of 09/01/03 does not fit the context. I
>>    *don't* change this one.
> 
> Yes it does. 

"^[\?]" is not part of the pattern.

>> I cannot find the price range rule which is announced in the
>> comment of 12 May 2003.
> 
> Hint: look for the word "dollar"

I only find that a token is returned. That token does not
allow for any -..

>> I am not sure about  MSG_COUNT	^\".MSG_COUNT\" -- are those
>> \ needed?
> 
> Does it matter?  It works.  As they say, "if it ain't broke, don't fix
> it."

It is confusing. And as you said yesterday, there is "code
that _looks_ ok (on casual inspection) but is actually
incorrect", so I try to understand if it is correct, not if
it seems to work.

pi
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: lexer-cleanup2
URL: <https://www.bogofilter.org/pipermail/bogofilter-dev/attachments/20031111/49a17b71/attachment.ksh>