bogofilter -M drops "From "

[David Relson]

On 16 Sep 2003 15:51:47 +1000
michael at optusnet.com.au wrote:

> David Relson <relson at osagesoftware.com> writes:
> [...] 
> > With 0.15.0, there was a significant reorganization of the input
> > code to better support handling of mbox files and maildir and MH
> > directories. Since the "From " line is added locally to a message,
> > its fields duplicate other header fields.  Thus, scoring its tokens
> > is a duplication.
> 
> No, this is bad. The 'From ' header is indeed added locally, but it's
> the often the only place that the envelope sender is recorded.
> 
> It _may_ duplicate other fields, but it's not guarenteed to
> do so.
> 
> 
> # telnet localhost 25
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> 220 optusnet.com.au ESMTP Sendmail 8.12.8/8.12.8; Tue, 16 Sep 2003
> 15:46:26 +1000 helo fred
> 250 optusnet.com.au Hello localhost.localdomain [127.0.0.1], pleased
> to meet you mail from: <santa at yahoo.com>
> 250 2.1.0 <santa at yahoo.com>... Sender ok
> rcpt to: michael at optusnet.com.au
> 250 2.1.5 michael at optusnet.com.au... Recipient ok
> data
> 354 Enter mail, end with "." on a line by itself
> To: fred
> From: fred
> Subject: hello!
>  
> test
> .
> 250 2.0.0 h8G5kQNr004031 Message accepted for delivery
> quit
> 221 2.0.0 optusnet.com.au closing connection
> Connection closed by foreign host.
> 
> 
> And then the message that's actually delivered is:
> 
> >From santa at yahoo.com  Tue Sep 16 15:47:17 2003
> Received: from fred (localhost.localdomain [127.0.0.1])
>         by optusnet.com.au (8.12.8/8.12.8) with SMTP id h8G5kQNr004031
>         for michael at optusnet.com.au; Tue, 16 Sep 2003 15:47:04 +1000
> Date: Tue, 16 Sep 2003 15:46:26 +1000
> Message-Id: <200309160547.h8G5kQNr004031 at optusnet.com.au>
> To: fred at optusnet.com.au
> From: fred at optusnet.com.au
> Subject: hello!
>  
> test
>  
> 
> 
> Note that the 'From ' line doesn't match anything from the other
> headers...
> 
> Michael.
> 

Michael,

Good work!  You've convinced me of the usefulness of the message
separator line.

Parsing of the message separator will return to bogofilter within the
hour.  Last night I fixed my private copy of bogoreader.c so it doesn't
discard that line, to enable passthrough to work properly.  The plan for
this morning _was_ to modify the lexer rules to discard the tokens. 
That has been scrapped and the task is to correct the reference results
- which is a much easier task.

David