fatal flex scanner internal error--end of buffer missed

[David Relson]

On Sat, 06 Sep 2003 22:33:05 +0200
Matthias Andree <matthias.andree at gmx.de> wrote:

> David Relson <relson at osagesoftware.com> writes:
> 
> > The truth may, of course, be somewhere inbetween.  The problem
> > likely has to do with the lack of a boundary between base64 text and
> > plain text.  Bogofilter is in "decode base64" mode and that likely
> > produces unexpected characters which cause flex to choke.
> 
> Flex mustn't choke on unexpected characters

Easy to say, hard to do.  As you know there are a variety of standards
specifying the format for email.  When a noncompliant message comes in,
all sorts of "interesting" things can happen. 

> > Any thoughts/suggestions???
> 
> Handle MIME decoding outside of the flexer and pass individual MIME
> parts to the lexer only.

And what happens when, say, we have a message body that's base64 encoded
html immediately followed by stuff using acceptable base64 characters
that's really plain text?  Bogofilter will happily decode it (producing
garbage) which may have unpredictable effects when the flex grammar sees
it.

If you've got a solution, I'd love to have it included in bogofilter !

David