block_on_subnets

[David Relson]

Greetings,

One of the (perhaps) little known options in bogofilter is 
"block_on_subnets".  When enabled, an IP address such as 1.2.3.4 generates 
4 url tokens - url:1.2.3.4, url:1.2.3, url:1.2, and url:1.  The idea was to 
provide some extra info to help bogofilter classify spam.

I can't say for sure how well it does that, but I have an interesting 
result from a couple of days ago.  Each night, a cron job runs and sends a 
message detailing unusual messages in the system logs.  Given the context, 
these messages are all ham.   However a few days back one arrived 
classified as Unsure, with spamicity of 0.456135

The message itself is pretty innocuous - primarily a few log messages 
generated by postfix/smptd.  The whole message is shown below.  Also shown 
is the histogram (generated by bogofilter -vv) which show a lot of ham 
tokens and 12 tokens with spamicty > 0.6.  The full set of evaluated tokens 
is in the attached file and below I've shown the 12 high scorers - all of 
which are "url:..." tokens.  As you can, bogofilter is recognizing spam 
subnets and using that info in its scoring.  Of course in this case the 
message was ham, but it's interesting to see that the subnet tokens can 
make a significant difference in the spam score.

David

**** MESSAGE ****

  From root at osagesoftware.com  Mon Jan 27 04:02:15 2003
Return-Path: <root at osagesoftware.com>
Delivered-To: root at osagesoftware.com
Received: by osagesoftware.com (Postfix, from userid 0)
	id AA6CF2868B; Mon, 27 Jan 2003 04:02:14 -0500 (EST)
To: root at osagesoftware.com
Subject: nic.osagesoftware.com 01/27/03:04.02 system check
Message-Id: <20030127090214.AA6CF2868B at osagesoftware.com>
Date: Mon, 27 Jan 2003 04:02:14 -0500 (EST)
From: root at osagesoftware.com (root)
X-Bogosity: Unsure, tests=bogofilter-f, spamicity=0.456135, version=0.10.1.1

Security Violations
=-=-=-=-=-=-=-=-=-=
Jan 26 05:08:18 nic postfix/smtpd[32446]: warning: 216.109.73.35: hostname 
om40.yourmailsoure.com verification failed: Host not found
Jan 27 01:33:46 nic postfix/smtpd[6820]: warning: 217.141.203.226: hostname 
host226-203.pool217141.interbusiness.it verification failed: Host not found
Jan 27 01:44:09 nic postfix/smtpd[6866]: warning: 209.236.58.60: hostname 
ms24.mxdat.com verification failed: Host not found
Jan 27 02:38:08 nic postfix/smtpd[7939]: warning: 209.236.58.42: hostname 
ms6.mxdat.com verification failed: Host not found
Unusual System Events
=-=-=-=-=-=-=-=-=-=-=
Jan 26 04:03:05 nic syslogd 1.4.1: restart.
Jan 27 00:01:10 nic msec: changed group of /var/log/procmail from wheel to root

**** HISTOGRAM ****

X-Bogosity: Unsure, tests=bogofilter-f, spamicity=0.456135, version=0.10.1.1
#     int  cnt    prob   spamicity  histogram
#    0.00   17  0.031185  0.015116  #################
#    0.10    5  0.141525  0.034588  #####
#    0.20    2  0.221864  0.046742  ##
#    0.30    2  0.306624  0.063658  ##
#    0.40    0  0.000000  0.063658
#    0.50    0  0.000000  0.063658
#    0.60    2  0.662999  0.109145  ##
#    0.70    2  0.780193  0.162127  ##
#    0.80    3  0.873959  0.244036  ###
#    0.90    5  0.987615  0.425346  #####

**** HIGH SCORING TOKENS ****

                                      n     pgood      pbad        fw 
invfwlog     fwlog U
"url:216"                          867  0.097151  0.181175  0.650945 
-1.05252  -0.42933 +
"url:209.236.58.42"                  2  0.000213  0.000442  0.675053 
-1.12409  -0.39296 +
"url:216.109.73.35"                 41  0.003401  0.011047  0.764581 
-1.44639  -0.26843 +
"url:216.109.73"                    46  0.003401  0.013257  0.795805 
-1.58868  -0.22840 +
"url:209.236.58"                    58  0.003189  0.019001  0.856290 
-1.93996  -0.15515 +
"url:209"                          396  0.020196  0.133009  0.868178 
-2.02630  -0.14136 +
"url:216.109"                      125  0.005102  0.044631  0.897408 
-2.27699  -0.10825 +
"url:209.236"                      232  0.005740  0.090588  0.940411 
-2.82029  -0.06144 +
"url:209.236.58.60"                  1  0.000000  0.000442  0.999416 
-7.44490  -0.00058 +
"url:217.141"                        1  0.000000  0.000442  0.999416 
-7.44490  -0.00058 +
"url:217.141.203"                    1  0.000000  0.000442  0.999416 
-7.44490  -0.00058 +
"url:217.141.203.226"                1  0.000000  0.000442  0.999416 
-7.44490  -0.00058 +
-------------- next part --------------
A non-text attachment was scrubbed...
Name: block_on_subnets.vvv
Type: application/octet-stream
Size: 5086 bytes
Desc: not available
URL: <http://www.bogofilter.org/pipermail/bogofilter/attachments/20030202/aa156d48/attachment.obj>