effects of lexer changes

[David Relson]

>4 - tokens from mime directives are not being output.  For example, 
>"Content-Type: text/plain" used to return 3 tokens.  Looks like yyredo() 
>isn't working.  I'll fix it.

When processing "Content-Type: text/plain; charset=us-ascii", yyredo() gets 
called twice.  The first time is for the whole message.  While reprocessing 
that line, yyredo() is called a second time for "charset=us-ascii" and this 
caused it to miss "plain".  A partial fix is to check the value of yysave 
(which points to the text being re-processed) and only save the new text if 
yysave is NULL.

>5 - The new code to "ignore anything when not reading text MIME types" may 
>be over zealous.  In msg.2.txt, it causes the text between the message 
>header and the first boundary line to be ignored.  Since there _is_ text 
>there, I think we want bogofilter to see it.  What do you think?
>
>The code to ignore tokens shouldn't take effect until bogofilter is in the 
>body of a mime part.  I think there's a problem of stack level confusion, 
>i.e. the level for setting mime info and the level for using it.

Adding "stackp > 0" to the ignore expression fixes the immediate problem.