[Bogofilter]UNSURE messages end up in inbox folder

[Nigel Henry]

On Tuesday 12 August 2008 13:01, Daniel Moyne wrote:
> Le Saturday 09 August 2008, Nigel Henry a écrit :
> > On Saturday 09 August 2008 00:55, Daniel Moyne wrote:
> > > I scrupulously followed Nigel's how-to with all these folders :
> > > Spam
> > > NonSpam
> > > nonspamnew
> > > spam
> > > unsure
> > > in KMail (kde-4)
> > >
> > > Apprently everything works fine except that all "unsure" prefixed in
> > > subjet as ???UNSURE??? end-up in my inbox folder though fliter-4 is
> > > upposed to get them dumped into folder unsure !
> > >
> > > Regards.
> >
> > I'll come back on this problem tomorrow, as it's getting late. My version
> > of bogofilter is quite old, but don't think that that is the problem.
> >
> > I'm also using an old version of Kmail on Fedora Core 2 (KDE 3.2.2), but
> > should not affect the filters.
> >
> > Of course I may not be able to help much. I have bogofilter working with
> > Kmail with no problems, but perhaps someone directly working with the
> > bogofilter project may be able to offer better advice.
> >
> > Nigel.
> > _______________________________________________
> > Bogofilter mailing list
> > Bogofilter at bogofilter.org
> > http://www.bogofilter.org/mailman/listinfo/bogofilter
>
> Nigel,
> so far no other UNSURE mails so difficult to say whether it works fine or
> not. Nigel still on you how-to
>
> Can we say that once you have done a run on "Spam" and "NonSpam" folders
> there are no reasons to keep their content that can be dumped ; we still
> have to keep these folders to dump in "Spam" some messages of "unsure"
> after making sure they must be processed as spam, and dumping in "NonSpam"
> some messages of "unsure" after making sure they must be processed as non
> spam.
>
> Once the content of "Spam" and "NonSpam" folders hs been updated we can ru
> you script on them.
>
> I am still wondering abot the use of "nonspamnew" folder.
> Regards.

Bonjour Daniel. Apologies for not replying. I got sidetracked with a KDE4 
problem on my Archlinux install, and forgot all about replying to you.

This is only my way of using the Spam, and NonSpam folders, and I'm sure 
others may do things differently, but this is how it goes. Having created the 
Spam, and NonSpam folders, and before starting to use bogofilter, I started 
to fill the Spam folder with spam that was coming into the inbox. At the same 
time I put the same amount of genuine mail in the NonSpam folder. Now there 
are about 200 emails in both the Spam, and NonSpam folders.

Now having downloaded, and installed bogofilter, and having created 
the .bogofilter directory in my /home/user directory, and also having created 
the bogofilter filter, and the the filters for ham, spam, and unsure, for the 
first, and only time I run bogofilter -sv -B Mail/Spam/cur, and bogofilter 
-nv -B Mail/Nonspam/cur.

Now when you next check the mail, bogofilter now has a bunch of spam, and ham 
to work with, and can decide, based on the spam, and ham in the 
~/.bogofilter/wordlist.db, where to send the incoming mail. Mail that is 
obviously ham will be sent to the inbox. Mail that is obviously spam, will be 
sent to the wastebin, but personally I set up another folder, which I've 
named spamcheck, so that the spam goes there first, and I can make sure that 
no genuine messages havn't been wrongly identified as spam. After that I can 
empty the spamcheck contents into the wastebin.

Now onto the unsure folder. Some spam can look like genuine email, or the 
spammers are trying new ways of getting past the spam filters, and if 
bogofilter isn't sure it will send it to the unsure folder. Of course the 
more you train bogofilter on new spam which turns up in the unsure folder, 
the less errors it makes. Some ham (nonspam) also at times can look a bit 
spammy. For example some genuine emails may have words included, that 
normally turn up in spam emails. Bogofilter again isn't sure, so sends these 
to the unsure folder.

Now I said earlier on that I only run bogofilters training once on the Spam, 
and Nonspam folders, and you could after doing that just send all your spam, 
and ham in these folders to the wastebin. then after doing that you could 
sort the unsure mails out, send the spam to the Spam folder, and the ham to 
the Nonspam folder, and run the training script on both again.

Doing it this way though, if for some reason or other your wordlist.db should 
become corrupted, you have to start all over again, which is why I train just 
once on the 200 emails in the Spam, and Nonspam folders, and leave all the 
ham, and spam in these folders. As I still have all my spam, and ham in these 
2 folders, if the wordlist.db should become corrupted, and I have to delete 
it, all I have to do is rerun the training program for both folders, and the 
wordlist.db will be recreated.

This is why I also created the folders "spam", and "Nonspamnew". I use these 
for the mail that is in the unsure folder. Each day I check the unsure 
folder, and send the spam to the "spam" folder, and the ham to the 
"Nonspamnew" folder. Now I don't run the training program on these 2 folders 
every day, and usually wait until there are about 100 spam mails in the 
"spam" folder, then run the following.
bogofilter -sv -B Mail/spam/cur
bogofilter -nv -B Mail/Nonspamnew/cur

Now I want to move these spam, and ham newly trained emails to the Spam, and 
Nonspam folders, so that if the wordlist.db should become corrupted, all the 
latest ham, and spam will be available to recreate the wordlist.db.

Ctrl +A will highlight all the mails in the spam folder, and right click, and 
move to Spam, will add these latest spam emails to the Spam folder.

Back to your problem with the unsure filter. This is what you said, see below.

<quote>
Apprently everything works fine except that all "unsure" prefixed in
 subjet as ???UNSURE??? end-up in my inbox folder though fliter-4 is
 upposed to get them dumped into folder unsure !
<end quote>

I'm not sure if I understand you here. Could you show how you have this filter 
setup.

The first line should show:
X-Bogosity          contains        Unsure

Depending on how many e-mails you receive each day, you should be getting some 
in the unsure box. the more you train bogofilter, the less mail should be in 
the unsure box, but this takes some time, and when spammers change their 
methods, you may well find more spam in the unsure box again.

Nigel.