Short tokens and numbers

[David Relson]

On Tue, 04 Nov 2003 12:57:17 +0100
Boris 'pi' Piwinger <3.14 at logic.univie.ac.at> wrote:


> Well, if I take them away it still builds and passes all
> tests. I also cannot find a difference with *all* my mails.
> So I assume, my changes are correct. I'm also cleaning up
> the order to make it more easily readable.

pi,

Please send a single patch with the changes you want.  I'm not
interested in evaluating two different, conflicting change sets.

"make check" is a tool to detect regressions.  It does not test all
possible situations, so it's possible that changes to lexer_v3.l could
be wrong and not show up in the test.

As to token length, lexer_v3.l has its maximum of 70 characters.  As
that's very old, I'm guessing it was done to "swallow" a complete base64
line.  In token.c, long tokens are discarded (base on the value of
MAXTOKENLEN).  That's likely what you've noticed.  Off-hand, I don't
know what the effect would be if we change (or remove) the 70 in TOKEN

David