lexer changes

[David Relson]

pi,

Please, when you introduce major new subjects into a message, change the
subject of the message.  Doing that makes it easier to messages on a
given subject.

I've tested your patch and "make check" FAILs many of the tests.  The
patch will not be applied.

On Tue, 11 Nov 2003 13:40:50 +0100
Boris 'pi' Piwinger <3.14 at logic.univie.ac.at> wrote:

> David Relson wrote:
> 
> > Since the latest CVS changes typically take a while to become
> > available, I've attached the latest revision.
> 
> Great improvements in there. Here are some more:
> 
> 0) Allowing two-byte-tokens (see my test on the other list)

Not allowed, for reasons I've previously stated.

> 1) Some \ slipped back in. Out again.

None of them "slipped" in.  I'm satisfied with those that are there and
don't see a need for spending time discussing them or removing them.  If
they break something, then we can discuss fixing what's broken.

> 2) I found some more unneeded \.

Same comment as above.

> 3) The comment of 09/01/03 does not fit the context. I
>    *don't* change this one.

Yes it does.  Both QP and BASE64 use subsets of the printable
characters.  Those subsets could be enumerated in the flex grammar, but
that increases the size a lot.  Instead C code tests for the correct
character sets.

> 4) HTMLTOKEN definition is doing nothing.

It provides an easy way for changing between HTML_WI_COMMENTS and
HTML_WO_COMMENTS.

> 5) By 0) TOKEN_12 reduces to TOKEN_1. Since there was really
>    nothing left to T12 than one class I merge it into
>    TOKEN_1.

And 
> 
> All changes are tested to the best of my knowledge and
> database. Please run a short test anyway.

As stated above, your changes break "make check".  To be specific, 11 of
the 32 tests fail.  I didn't pay attention to which ones, but I'm
assuming that all the tests that involve parsing have been broken.

> I cannot find the price range rule which is announced in the
> comment of 12 May 2003.

Hint: look for the word "dollar"

> Why "NAME lexer_header.l"?

Because lexer_v3.l is descended from lexer_header.l.  I'll change this.

> I am not sure about  MSG_COUNT	^\".MSG_COUNT\" -- are those
> \ needed?

Does it matter?  It works.  As they say, "if it ain't broke, don't fix
it."


> It is still inconsistent if we use [:digit:] or [0-9] and
> similar things.

Right.