lexer header/body

[Boris 'pi' Piwinger]

Matthias Andree wrote:

> I haven't looked at the lexer code in a long time. Anyways, the angle
> brackets contain a "state" - the lexer bogofilter uses is a state
> machine to some extent, the state is changed by the BEGIN(newstate)
> statements. <INITIAL> should be the state where we parse headers, and a
> blank line should switch the lexer to some kind of body mode, away from
> INITIAL.

You seem to be right. I just built a version with this:
<INITIAL>^Status:.*                             /* ignore */

I tested a message with a Status header and the same line in
the body. It was recognized in the body only. Great. So
INITIAL seems to be HEADER (maybe this is what we should
call it then). What I do not understand then ...

token.c defines some function called form lexer_v3.l like
set_tag. This function checks if we have header_line_markup.
This seems unneeded then. Is this overly careful? Or is it a
leftover? If not how and why is it needed?

pi