<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<title>Bogofilter FAQ</title>
<style type="text/css">
h2 {
margin-top: 1em;
font-size: 125%;
}
h3 {
margin-top: 1em;
font-size: 110%;
}
p {
margin-top : 0.5em;
margin-bottom: 0.5em;
}
ul {
margin-top: 1.5em;
margin-bottom: 0.5em;
}
ul ul {
margin-top: 0.25em;
margin-bottom: 0;
}
li {
margin-top: 0;
margin-bottom: 1em;
}
li li {
margin-bottom: 0.25em;
}
dt {
margin-top: 0.5em;
margin-bottom: 0;
}
hr {
margin-top: 1em;
margin-bottom: 1em;
}
</style>
</head>
<body>
<h1>Bogofilter FAQ</h1>
<p>Official Versions: In
<a href="http://bogofilter.sourceforge.net/bogofilter-faq.html">English</a> or
<a href="http://bogofilter.sourceforge.net/bogofilter-faq-fr.html">French</a><br>
Maintainer: David Relson <relson@osagesoftware.com></p>
<p>This document is intended to answer frequently asked
questions about bogofilter.</p>
<ul>
<li>
General Information
<ul>
<li><a href="#what-is-bogofilter">What is
bogofilter?</a></li>
<li><a href="#bogo-what">Bogo what?</a></li>
<li><a href="#lists">Bogofilter Mailing Lists</a></li>
</ul>
</li>
<li>
Operational Questions
<ul>
<li><a href="#train-on-error">What are "training on error" and
"training to exhaustion"?</a></li>
<li><a href="#training">How do I start my bogofilter training?</a></li>
<li><a href="#production">How can I keep the scoring accuracy high?</a></li>
<li><a href="#spamassassin">How can I use SpamAssassin to train Bogofilter?</a></li>
<li><a href="#vvv">What does bogofilter's verbose output mean?</a></li>
<li><a href="#asian-spam">What can I do about asian spam?</a></li>
</ul>
</li>
<li>
Database Questions
<ul>
<li><a href="#query-database">How do I manually query the
database</a></li>
<li><a href="#rescue">How can I tell if my wordlists are
corrupted?</a></li>
<li><a href="#update">How do I upgrade from separate word
databases to the new combined wordlist format?</a></li>
</ul>
</li>
<li>
Technical problems
<ul>
<li><a href="#nfs">Can I share wordlists over
NFS?</a></li>
<li><a href="#return-codes">Why does bogofilter give
return codes like 0 and 256 when it's run from inside a
program?</a></li>
<li><a href="#changed-options">Now that I've upgraded to
0.11 why are my scripts broken?</a></li>
<li><a href="#changed-exitcodes">Now that I've upgraded to
0.14 why are my scripts broken?</a></li>
<li><a href="#changed-tagging">Now that I've upgraded to
0.15 why is bogofilter working less well?</a></li>
<li><a href="#remove-spam-or-nonspam">Having a combined
wordlist, how can I delete all the spam (or non-spam)
tokens?</a></li>
</ul>
</li>
<li>
Portability Problems
<ul>
<li><a href="#port-notes">How do I get Bogofilter working
on Solaris, BSD, etc.?</a></li>
<li><a href="#make-notes">Can I use the make command on my
operating system?</a></li>
</ul>
</li>
</ul>
<hr>
<h2 id="what-is-bogofilter">What is bogofilter?</h2>
<p>Bogofilter is a fast Bayesian spam filter along the lines
suggested by <a href="http://www.paulgraham.com/">Paul Graham</a>
in his article <a href="http://www.paulgraham.com/spam.html">A
Plan For Spam</a>. Bogofilter uses
<a href="http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.html">Gary
Robinson</a>'s geometric-mean algorithm with the Fisher's
method modification to classify email as spam or non-spam.</p>
<p>The bogofilter
<a href="http://bogofilter.sourceforge.net">home page</a> at
SourceForge is the central clearinghouse for bogofilter
resources.</p>
<p>Bogofilter was started by <a href="http://catb.org/~esr/">Eric
S. Raymond</a> on August 19, 2002. It gained popularity in
September 2002, and a number of other authors have started to
contribute to the project.</p>
<p>The <a href="http://bogofilter.sourceforge.net/NEWS">NEWS
file</a> describes bogofilter's version history.</p>
<hr>
<h2 id="bogo-what">Bogo-what?</h2>
<p>Bogofilter is some kind of a
<a href="http://www.catb.org/~esr/jargon/html/B/bogometer.html">bogometer</a> or
<a href="http://www.catb.org/~esr/jargon/html/B/bogon-filter.html">bogon filter</a>, i.e., it tries to identify
<a href="http://www.catb.org/~esr/jargon/html/B/bogus.html">bogus</a> mail by measuring the
<a href="http://www.catb.org/~esr/jargon/html/B/bogosity.html">bogosity</a>.</p>
<hr>
<h2 id="lists">Mailing Lists</h2>
<p>There are currently four mailing lists for bogofilter:</p>
<table border="1" width="100%">
<tr>
<th>List Address</th>
<th>Links</th>
<th>Description</th>
</tr>
<tr>
<td>bogofilter-announce@aotto.com</td>
<td><a href=
"mailto:bogofilter-announce-subscribe@aotto.com">[subscribe]</a>
<a href="http://news.gmane.org/thread.php?group=gmane.mail.bogofilter.announce">[archive]</a></td>
<td>An announcement-only list where new versions are
announced.</td>
</tr>
<tr>
<td>bogofilter@aotto.com</td>
<td><a href="mailto:bogofilter-subscribe@aotto.com">[subscribe]</a>
<a href="http://news.gmane.org/thread.php?group=gmane.mail.bogofilter.general">[archive]</a></td>
<td>A discussion list where any conversation about
bogofilter may take place.</td>
</tr>
<tr>
<td>bogofilter-dev@aotto.com</td>
<td><a href="mailto:bogofilter-dev-subscribe@aotto.com">[subscribe]</a>
<a href="http://news.gmane.org/thread.php?group=gmane.mail.bogofilter.devel">[archive]</a></td>
<td>A list for sharing patches, development, and technical
discussions.</td>
</tr>
<tr>
<td>bogofilter-cvs@lists.sourceforge.net</td>
<td><a href="http://lists.sourceforge.net/mailman/listinfo/bogofilter-cvs">[subscribe]</a>
<a href="http://sourceforge.net/mailarchive/forum.php?forum=bogofilter-cvs">[archive]</a></td>
<td>Mailing list for announcing code changes to the CVS
archive.</td>
</tr>
</table>
<hr>
<h2 id="train-on-error">What are "training on error" and "training to exhaustion"?</h2>
<p>"Training on error" involves scanning a corpus of known spam
and non-spam messages; only those that are misclassified, or
classed as unsure, get registered in the training database. It's
been found that sampling just messages prone to misclassification
is an effective way to train; if you train bogofilter on the hard
messages, it learns to handle obvious spam and non-spam too.</p>
<p>This method can be enhanced by using a "security margin". By
increasing the spam cutoff value and decreasing the ham cutoff
value, messages which are close to a cutoff will be used for
training. Using security margins was shown to improve results when
training on error.</p>
<p>"Training to exhaustion" is repeating training on error, with
the same message corpus, until no errors remain.</p>
<p>A basic assumption of bayes' theory is that the messages used
for training are a randomly chosen sample of the messages received.
This is violated when choosing messages by analyzing them first.
Though theoretically wrong, in practice "training on error" seems
to work.</p>
<p>Registering messages different numbers of times (as can happen
with training to exhaustion) changes the
distribution of token scores in the training database so that it's
different from the distribution in the input messages. This
violates a basic assumption of Bayesian classification, and may
lead to unpredictable results. Though theoretically wrong, in
practice "training to exhaustion" seems to work.</p>
<p>Note: <code>bogominitrain.pl</code> has a <code>-f</code> option
to do "training to exhaustion". If you choose to use it, you need
to be aware of its possible effects. Using <code>-fn</code> avoids
repeated training for each message.</p>
<hr>
<h2 id="training">How do I start my bogofilter training?</h2>
<p>To classify messages as ham (non-spam) or spam, bogofilter
needs to learn from your mail. To start with it is best to have
collections (that are as large as possible) of messages you know
for sure are ham or spam. (Errors here will cause problems later,
so try hard<code>;-)</code>. Warning: Only use your mail; using other
collections (like a spam collection found on the web), might cause
bogofilter to draw a wrong conclusion — after all you want it to
understand <em>your</em> mail.</p>
<p>Once you have the spam and ham collections, you have basically
four choices. In all cases it works better if your training base
(the above collections) is bigger, rather than smaller. The
smaller your training collection is, the higher the number of
errors bogofilter will make in production. Let's assume your
collection is two mbox files: ham.mbx and spam.mbx.</p>
<ul>
<li><p>Method 1) Full training. Train bogofilter with all your messages. In
our example:</p>
<pre> bogofilter -s < spam.mbx
bogofilter -n < ham.mbx</pre></li>
</ul>
<p>Note: bogofilter's contrib directory includes two scripts that
both use a train-on-error technique. This technique scores each
message and adds to the database only those messages that were
scored incorrectly (messages scored as uncertain, ham scored as
spam, or spam scored as ham). The goal is to build a database of
those words <em>needed</em> to correctly classify messages. The
resulting database is smaller than the one build using full
training.</p>
<ul>
<li><p>Method 2) Use the script bogominitrain.pl (in the contrib
directory). It checks the messages in the same order as your
mailbox files. You can use the <code>-f</code> option which will
repeat this until all messages in your training collection are
classified correctly (you can even adjust the level of
certainty). Since the script makes sure the database understands
your training collection "exactly" (with your chosen
precision), it works very well. You can use <code>-o</code> to
create a security margin around your spam_cutoff. Assuming
spam_cutoff=0.6 you might want to score all ham in your
collection below 0.3 and all spam above 0.8. Our example is:</p>
<pre> bogominitrain.pl -fnv ~/.bogofilter ham.mbx spam.mbx '-o 0.8,0.3'</pre></li>
<li><p>Method 3) Use the script randomtrain (in the contrib
directory). The script generates a list of all the messages in the
mailboxes, randomly shuffles the list, and then scores each
message, with training as needed. In our example:</p>
<pre> randomtrain -s spam.mbx -n ham.mbx</pre>
<p>As with method 4, it works better if you start with full
training using several thousand messages. This will give a
database that is more comprehensive and significantly
bigger.</p></li>
<li><p>Method 4) If you have enough spams and non-spams in your
training collection, separate out some 10,000 spams and 10,000
non-spams into separate mbox files, and train as in method 1. Then
use bogofilter to classify the remaining spams and non-spams. Take
any messages that it classifies as unsure or classifies
incorrectly, and train with those. Here are two little scripts you
can use to classify the train-on-error messages:</p>
<pre> #! /bin/sh
# class3 -- classify one message as bad, good or unsure
cat >msg.$$
bogofilter $* <msg.$$
res=$?
if [ $res = 0 ]; then
cat msg.$$ >>corpus.bad
elif [ $res = 1 ]; then
cat msg.$$ >>corpus.good
elif [ $res = 2 ]; then
cat msg.$$ >>corpus.unsure
fi
rm msg.$$</pre>
<pre> #! /bin/sh
# classify -- put all messages in mbox through class3
src=$1;
shift
formail -s class3 $* <$src</pre>
<p>In our example (after the initial full training):</p>
<pre> classify spam.mbx [bogofilter options]
bogofilter -s < corpus.good
rm -f corpus.*
classify ham.mbx [bogofilter options]
bogofilter -n < corpus.bad
rm -f corpus.*</pre></li>
</ul>
<h3>Comparing these methods</h3>
<p>It is important to understand the consequences of the methods
just described. Doing full training as in methods 1 and 4 produces
a larger database than does training with methods 2 or 3. If your
database size needs to be small (for example due to quota
limitations), use methods 2 or 3.</p>
<p>Full training with method 1 is fastest. Training on error (as
in methods 2, 3 and 4) is effective, but learning is pretty
slow.</p>
<hr>
<h2 id="production">How can I keep the scoring accuracy high?</h2>
<p>Bogofilter will make mistakes once in a while. So ongoing
training is important. There are two main methodologies for doing this.
First, you can train with every incoming message (using the -u
option). Second, you can train on error only.</p>
<p>Since you might want to rebuild your database at some point,
for example when a major new feature is implemented in bogofilter,
it can be very useful to update your training collection
continuously.</p>
<p>Bogofilter always does the best it can with the information
available to it. However, it will make mistakes, i.e., classify
ham as spam (false positives) or spam as ham (false negatives). To
reduce the likelihood of repeating the mistake, it is necessary to
train bogofilter with the errant message. If a message is
incorrectly classified as spam, use switch <code>-n</code> to
train with it as ham. Use switch <code>-s</code> to train with a
spam message.</p>
<p>Bogofilter has a <code>-u</code> switch that automagically
updates the wordlists after scoring each message. As bogofilter
sometimes misclassifies a message, monitoring is necessary to
correct any mistakes. Corrections can be done using
<code>-Sn</code> to change a message's classification from spam to
non-spam and <code>-Ns</code> to change it from non-spam to spam.</p>
<p>Correcting a misclassfied message may affect classification for
other message. The smaller your database is, the higher is the
likelihood that a training error will casue a misclassification.</p>
<p>Using a method like #2 or #3 (above) can compensate for this
effect. Repeat the training with your complete training
collection (including all the new messages added since the earlier
training). This will add messages to the database which show that
adverse effect on both sides until you have a new equilibrium.</p>
<p>An alternative strategy, based on method 4 in the previous
section, is the following: Periodically take blocks of messages
and use the scripts in method 4 above to classify them. Then
manually review the good, bad and unsure files, correct any
errors, and split the unsures into spam and non-spam. Until you
have accumulated some 10,000 spam and 10,000 non-spam in your
training database, train with the good, the bad, and the separated
errors and unsures; thereafter, train with only the separated and
unsures, discarding the messages that bogofilter already
classifies correctly.</p>
<p>Note that you should periodically run:</p>
<pre> bogoutil -d wordlist.db | bogoutil -l wordlist.db.new
mv wordlist.db wordlist.db.prv
mv wordlist.db.new wordlist.db</pre>
<p>or, for spamlist.db and goodlist.db (if using bogofilter with
separate spam and ham wordlists). This will compact the database
so it occupies the minimum of disk space.</p>
<hr>
<h2 id="spamassassin">How can I use SpamAssassin to train
Bogofilter?</h2>
<p>If you have a working SpamAssassin installation (or care to
create one), you can use its return codes to train bogofilter.
The easiest way is to create a script for your MDA that runs
SpamAssassin, tests the spam/non-spam return code, and runs
bogofilter to register the message as spam (or non-spam). The
sample procmail recipe below shows one way to do this:</p>
<pre> BOGOFILTER = "/usr/bin/bogofilter"
BOGOFILTER_DIR = "training"
SPAMASSASSIN = "/usr/bin/spamassassin"
:0 HBc
* ? $SPAMASSASSIN -e
#spam yields non-zero
#non-spam yields zero
| $BOGOFILTER -n -d $BOGOFILTER_DIR
#else (E)
:0Ec
| $BOGOFILTER -s -d $BOGOFILTER_DIR
:0fw
| $BOGOFILTER -p -e
:0:
* ^X-Bogosity:.Yes
spam
:0:
* ^X-Bogosity:.No
non-spam</pre>
<hr>
<h2 id="vvv">What does bogofilter's verbose output mean?</h2>
<p>Bogofilter can instructed to display information on the
scoring of a message by running it with flags "-v", "-vv",
"-vvv", or "-R".</p>
<ul>
<li>
Using "-v" causes bogofilter to generate the "X-Bogosity:"
header line, i.e.
<pre> X-Bogosity: No, tests=bogofilter, spamicity=0.500000</pre></li>
<li>
Using "-vv" causes bogofilter to generate a histogram, i.e.
<pre> X-Bogosity: No, tests=bogofilter, spamicity=0.500000
int cnt prob spamicity histogram
0.00 29 0.000209 0.000052 #############################
0.10 2 0.179065 0.003425 ##
0.20 2 0.276880 0.008870 ##
0.30 18 0.363295 0.069245 ##################
0.40 0 0.000000 0.069245
0.50 0 0.000000 0.069245
0.60 37 0.667823 0.257307 #####################################
0.70 5 0.767436 0.278892 #####
0.80 13 0.836789 0.334980 #############
0.90 32 0.984903 0.499835 ################################</pre>
<p>Each row shows an interval, the count of tokens with
scores in that interval, the average spam probability for
those tokens, the message's spamicity score (for those
tokens and all lesser valued tokens), and a bar graph
corresponding to the token count.</p>
<p>In the above histogram there are a lot of low scoring
tokens and a lot of high scoring tokens. They "balance" one
another to give the spamicity score of 0.5000</p>
</li>
<li>
Using "-vvv" produces a list of <em>all</em> the tokens in
the messages with information on each one, i.e.
<pre> X-Bogosity: No, tests=bogofilter, spamicity=0.500000
n pgood pbad fw U
"which" 10 0.208333 0.000000 0.000041 +
"own" 7 0.145833 0.000000 0.000059 +
"having" 6 0.125000 0.000000 0.000069 +
...
"unsubscribe.asp" 2 0.000000 0.095238 0.999708 +
"million" 4 0.000000 0.190476 0.999854 +
"copy" 5 0.000000 0.238095 0.999883 +
N_P_Q_S_s_x_md 138 0.00e+00 0.00e+00 5.00e-01
1.00e-03 4.15e-01 0.100</pre>
The columns printed contain the following information:
<dl>
<dt>"…"</dt>
<dd>the token in question</dd>
<dt>n</dt>
<dd>number of times this token was encountered in
training</dd>
<dt>pgood</dt>
<dd>proportion of good messages that contained this
token</dd>
<dt>pbad</dt>
<dd>proportion of spam messages that contained this
token</dd>
<dt>fw</dt>
<dd>Robinson's weighted index, which combines pgood and
pbad to give a value that will be close to zero if a
message containing this token is likely to be non-spam
and close to one if it's likely to be spam</dd>
<dt>U</dt>
<dd>'<b>+</b>' if this token contributes to the final
bogosity value, '<b>-</b>' otherwise. A token is excluded
when its score is closer to 0.5 than min_dev.</dd>
</dl>
<p>The final lines show:</p>
<ul style="margin-top:0;margin-bottom:1em;">
<li>The cumulative results of the columns</li>
<li>The values of Robinson's <b>s</b> and <b>x</b>
parameters and of <b>min_dev</b></li>
</ul>
</li>
<li>
Using "-R" produces the "-vvv" output described above plus
two additional columns:
<dl>
<dt>invfwlog</dt>
<dd>logarithm of fw</dd>
<dt>fwlog</dt>
<dd>logarithm of (1-fw)</dd>
</dl>
<p>The "-R" output is formatted for use with the R language
for statistical computing. More information is available at
<a href="http://www.r-project.org/">The R Project for
Statistical Computing</a>.</p>
</li>
</ul>
<hr>
<h2 id="asian-spam">What can I do about asian spam?</h2>
<p>Many people get unsolicited email using asian language
charsets. Since they don't know the languages and don't know
people there, they assume it's spam.</p>
<p>The good news is that bogofilter does detect them quite
successfully. The bad news is that this can be expensive. You
have basically two choices:</p>
<ul>
<li>
<p>You can simply let bogofilter handle it. Just train
bogofilter with the asian language messages identified as
spam. Bogofilter will parse the messages as best it can and
will add tokens to the spam wordlist. The wordlist will
contain many tokens which don't make sense to you (since
the charset cannot be displayed), but bogofilter can work
with them and successfully identify asian spam.</p>
<p>A second method is to use the
"replace_nonascii_characters" config file option. This will
replace high-bit characters, i.e. those between 0x80 and
0xFF, with question marks, '?'. This keeps the database
much smaller. Unfortunately this conflicts with european
language which have many accented vowels and consonant in
the high-bit range.</p>
</li>
<li>
<p>If you are sure you will not receive any legitimate
messages in those languages, you can kill them right away.
This will keep the database smaller. You can do this with
an MDA script.</p>
<p>Here's a procmail recipe that will sideline messages
written with asian charsets:</p>
<pre> ## Silently drop all asian language mail
UNREADABLE='[^?"]*big5|iso-2022-jp|ISO-2022-KR|euc-kr|gb2312|ks_c_5601-1987'
:0:
* 1^0 $ ^Subject:.*=\?($UNREADABLE)
* 1^0 $ ^Content-Type:.*charset="?($UNREADABLE)
spam-unreadable
:0:
* ^Content-Type:.*multipart
* B ?? $ ^Content-Type:.*^?.*charset="?($UNREADABLE)
spam-unreadable</pre>
<p>With the above recipe, bogofilter will <em>never</em>
see the message.</p>
</li>
</ul>
<hr>
<h2 id="query-database">How do I manually query the
database</h2>
<p>To find the spam and ham counts for a token (word) use
bogoutil's '-w' option. For example, "bogoutil -w
$BOGOFILTER_DIR example.com" gives the good and bad counts for
"example.com".</p>
<p>If you want the spam score in addition to the spam and ham
counts for a token (word) use bogoutil's '-p' option. For
example, "bogoutil -p $BOGOFILTER_DIR example.com" gives the
good and bad counts for "example.com".</p>
<p>To find out how many messages are in your wordlists query
the special token .MSG_COUNT, i.e., run command "bogoutil -w
$BOGOFILTER_DIR .MSG_COUNT" to see the counts for the spam and
ham word lists.</p>
<p>To tell how many tokens are in your wordlists pipe the output
of bogoutil's dump command to command "wc", i.e. use "bogoutil -d
$BOGOFILTER_DIR/wordlist.db | wc -l " to display the count. (If
you've got spamlist.db and goodlist.db, run the command for each
of them).</p>
<hr>
<h2 id="rescue">How can I tell if my word lists are
corrupted?</h2>
<p>If you think your word lists are hosed, you can see what
BerkeleyDB thinks by running:</p>
<pre> db_verify wordlist.db</pre>
<p>If there is a problem, you may be able to recover some (or
all) of the tokens and their counts with the following
commands:</p>
<pre> bogoutil -d wordlist.db | bogoutil -l wordlist.db.new</pre>
<p>or with</p>
<pre> db_dump -r wordlist.db > wordlist.txt
db_load wordlist.new < wordlist.txt</pre>
<p>If you've got two wordlists (spamlist.db and goodlist.db), run the above commands for each of them.</p>
<hr>
<h2 id="update">How do I upgrade from separate word
databases to the new combined wordlist format?</h2>
<p>Run script bogoupgrade. For more info, run "bogoupgrade -h" to
see its help message or read its man page.</p>
<hr>
<h2 id="nfs">Can I share word lists over NFS?</h2>
<p>If all you're just reading from them, there are no problems.
When you're updating them, you need to use the correct file
locking to avoid data corruption. When you compile bogofilter, you
will need to verify that the configure script has set "#define
HAVE_FCNTL 1" in your config.h file. Popular UNIX operating
systems will all support this. If you are running an unusual, or
an older version of an operating system, make sure it supports
fcntl(). If "#define HAVE_FCNTL 1" is set, which indicates
fcntl() is supported on your system, then comment out "#define
HAVE_FLOCK 1" so that the locking system uses fcntl() locking
instead of the default of flock() locking. If your system does not
support fcntl(), then you will not be able to share word list
files over NFS without the risk of data corruption.</p>
<p>Next, make sure you have NFS set up properly, with "lockd"
running. Refer to your NFS documentation for more information
about running "lockd" or "rpc.lockd". Most operating systems
with NFS turn this on by default.</p>
<hr>
<h2 id="return-codes">Why does bogofilter give return codes
like 0 and 256 when it's run from inside a program?</h2>
<p>Likely the return codes are being reformatted by waitpid(2).
In C use WEXITSTATUS(status) in sys/wait.h, or comparable macro,
to get the correct value. In Perl you can just use
'system("bogofilter $input") >> 8'. If you want more info, run
<code>"man waitpid"</code>.</p>
<hr>
<h2 id="changed-options">Now that I've upgraded to 0.11 why are
my scripts broken?</h2>
<p>With version 0.11 bogofilter's options for registering mail
as ham or spam have been changed. They now allow registering
(or unregistering) messages in the ham and spam word lists.
Prior to this, there was no way to unregister a message from a
word list (without registering it in the other word list).</p>
<p>Bogofilter has four registration options - '-s', '-n', '-S',
and '-N'. With the release of version 0.11 the meaning of '-S'
and '-N' has been changed to allow unregistering messages from
the word lists. Here's what the four options mean:</p>
<dl>
<dt>-s</dt>
<dd>means to register the current message as spam,
i.e. increment the spam counts of its tokens.</dd>
<dt>-n</dt>
<dd>means to register the current message as ham,
i.e. increment the ham counts of its tokens.</dd>
<dt>-S</dt>
<dd>means to unregister the current message from the spam
word list, i.e. decrement the spam counts.</dd>
<dt>-N</dt>
<dd>means to unregister the current message from the ham
word list, i.e. decrement the ham counts.</dd>
</dl>
<p>Prior to version 0.11, the '-S' option was used to move a
message from the ham word list to the spam word list, i.e. there
were two actions. Now with 0.11 each of the two actions is
invoked by its own option. To get the same effect as the old
'-S', you should use '-N -s' (or '-Ns' which means the same
thing).</p>
<p>Similarly, the old '-N' option is now '-Sn' (or '-S
-n').</p>
<p>MDA scripts typically use '-s' and '-n' and don't need to
change. Other scripts which use '-S' and '-N' for fixing
registration errors <em>do</em> need to be changed.</p>
<hr>
<h2 id="changed-exitcodes">Now that I've upgraded to 0.14 why are
my scripts broken?</h2>
<p>Bogofilter-0.14 introduced an additional exit code to support
its third mail classification (of spam, ham, and unsure). Prior
to 0.14, the exit codes were 0 for spam, 1 for ham, and 2 for
error. They are now 0 for spam, 1 for ham, 2 for unsure, and 3
for error.</p>
<hr>
<h2 id="changed-tagging">Now that I've upgraded to 0.15 why is
bogofilter working less well?</h2>
<p>Prior to version 0.15.4, bogofilter added special prefixes to
tags from "Subject:", "From:", "To:", and "Return-Path:" header
lines. As of version 0.15.4, "Received:" line tokens are also
specially tagged and all other header line tokens are given a
"head:" prefix. Since bogofilter hasn't previously seen "header:"
tokens, these tokens don't contribute to either the ham or spam
scores. This loss of information causes bogofilter to perform
less well.</p>
<p>There are two ways to deal with this problem. The first is
simply to retrain bogofilter with all the old ham and spam. This
will create the needed "head:" entries and all will be fine.</p>
<p>The second is to use the new header-degen ("-H") option. When
classifying a message, header-degen causes bogofilter to look up
both "head:token" and "token" in the wordlist. The two ham and
spam scores are combined to give a cumulative result for scoring.
Registering messages will create "head:token" entries in the
wordlist. After a month or so of using '-H', the wordlist should
contain plenty of the new "head:" entries and using '-H' should no
longer be necessary.</p>
<hr>
<h2 id="remove-spam-or-nonspam">Having a combined wordlist, how can I
delete all the spam (or non-spam) tokens?</h2>
<p>Bogoutil lets you dump a wordlist and load the tokens into a new wordlist. With the added use of awk and grep, counts can be zeroed and tokens with zero counts for both spam and non-spam can be deleted.</p>
<p>The following commands will delete the tokens from spam messages:</p>
<pre> bogoutil -d wordlist.db | \
awk '{print $1 " " $2 " 0"}' | grep -v " 0 0" | \
bogoutil -l wordlist.new.db
</pre>
<p>The following commands will delete the tokens from non-spam messages:</p>
<pre> bogoutil -d wordlist.db | \
awk '{print $1 " 0 " $3}' | grep -v " 0 0" | \
bogoutil -l wordlist.new.db
</pre>
<hr>
<h2 id="port-notes">How do I get bogofilter working on Solaris,
BSD, etc?</h2>
<p>If you don't already have a v3.0+ version of <a href=
"http://www.sleepycat.com">BerkeleyDB</a>, then <a href=
"http://www.sleepycat.com/download.html">download it</a>,
unpack it, and do these commands in the db directory:</p>
<pre> $ cd build_unix
$ sh ../dist/configure
$ make
# make install</pre>
<p>Next, download a <a href=
"http://sourceforge.net/project/showfiles.php?group_id=62265">portable
version</a> of bogofilter.</p>
<h3>On Solaris</h3>
<p>Unpack it, and then do:</p>
<pre> $ ./configure --with-db=/usr/local/BerkeleyDB.4.1
$ make
# make install-strip</pre>
<p>You will either want to put a symlink to libdb.so in
/usr/lib, or use a modified LD_LIBRARY_PATH environment
variable before you start bogofilter.</p>
<pre> $ LD_LIBRARY_PATH=/usr/lib:/usr/local/lib:/usr/local/BerkeleyDB.4.1</pre>
<p>Note that some make versions shipped with Solaris break when
you try to build bogofilter outside of its source directory.
Either build in the source directory (as suggested above) or
use GNU make (gmake).</p>
<h3>On FreeBSD</h3>
<p>The FreeBSD ports and packages carry very recent versions of
bogofilter. This approach uses the highly recommended
portupgrade and cvsup software packages. To install these two
fine pieces, type (you need to do this only once):</p>
<pre> # pkg_add -r portupgrade cvsup</pre>
<p>To install or upgrade bogofilter, just
<a href="http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/cvsup.html">upgrade
your portstree using cvsup</a>, then type:</p>
<pre> # portupgrade -N bogofilter</pre>
<h3>On HP-UX</h3>
<p>See the file
<a href="http://cvs.sourceforge.net/viewcvs.py/*checkout*/bogofilter/bogofilter/doc/programmer/README.hp-ux?rev=HEAD&content-type=text/plain">doc/programmer/README.hp-ux</a>
in the source distribution.</p>
<hr>
<h2 id="make-notes">Can I use the make command on my
operating system?</h2>
<p>Bogofilter has been successfully built on many operating
systems using GNU make and the native make commands. However,
bogofilter's Makefile doesn't work with some make commands.</p>
<p>GNU make is recommended for building bogofilter because we
know it works. We cannot support less capable make commands. If
your non-GNU make command can successfully build bogofilter,
that's great. If you encounter problems, the right thing to do is
install GNU make. If your non-GNU make can't build bogofilter,
we're sorry but you're on your own.</p>
<hr>
<!--
$Id: bogofilter-faq.html,v 1.71 2003/12/15 14:58:29 relson Exp $
-->
</body>
</html>