repeately database corruption after some time

[David Relson]

On Wed, 22 Mar 2006 04:15:58 +0100
Christian Parpart wrote:

> On Wednesday 22 March 2006 03:57, David Relson wrote:
> > On Wed, 22 Mar 2006 03:49:16 +0100
> >
> > Christian Parpart wrote:
> > > Hi all,
> > >
> > > We are running repeatily into the problem, that bogofilter refuses to
> > > filter the mails after some certain time of having been run greatly in
> > > success.
> > >
> > > But we just can't reinitiate the users' database each and every time.
> > >
> > > However, the error  that occurs look like:
> > >
> > > Mar 17 16:30:50 himura postfix/local[29724]:
> > > 9377A79DA3D: to=<trapni at SERVER>, orig_to=<trapni at SERVER>, relay=local,
> > > delay=0, status=deferred
> > > (temporary failure. Command output: bogofilter[810]:
> > > error: the data base file size is only 16 pages bogofilter[810]:
> > >         below the resource limit. Cowardly refusing bogofilter[810]:
> > >         to continue to avoid data base corruption.
> > > procmail: Error while writing to "bogofilter"
> > > procmail: Rescue of unfiltered data succeeded )
> > >
> > > My guess was that the database file might have exceeded some certain
> > > limit of size/whatever.
> > >
> > > My wordlist.db actually is 51M, but can't use it anymore, as I myself get
> > > this error, too. Removing about 500 spam mails by traversing through all
> > > maildirs each day is a pain, so, I really hope to get an answer here...
> > >
> > > Thanks in advance,
> > > Christian Parpart.
> >
> > Hi Christian,
> >
> > Odds are that you're running into a mailbox size limit imposed by
> > postfix.  Try running "postconf | grep mailbox_size_limit" to see what
> > your current setting is.  Additional information is in the FAQ under
> > heading "Why am I getting DB_PAGE_NOTFOUND messages?"
> 
> Hey David,
> 
> thanks for the fast reply. The command you gave me results into:
> 
> mailbox_size_limit = 51200000
> 
> As said, I'm getting an incredable amount of spam each day, and bogofilter 
> frees me of 97% of all of them.
> 
> What I do not understand immediately is that DB_PAGE_NOTFOUND relation to my 
> problem. I mean, I was grepping the FAQ for my problem but didn't really know 
> that it was directly related to postfix itself.
> 
> I increased the value by 4. I suppose it works from now on, lets see.
> 
> Many thanks,
> Christian Parpart.

Christian,

DB_PAGE_NOTFOUND tends to happen when the database maximum size has
been reached, BerkeleyDB tries to go beyond the maximum, and does the
wrong thing.

Bogofilter has had the "cowardly refusing message message for a while"
and that's supposed to provide protection against database corruption.
Likely Matthias will have more to say when he comes online in the
morning (CEST).

What version of bogofilter are you running?  If it's older than 1.0.0,
you're due for an upgrade.  There have been some recent tweaks to
bogofilter's messages when the database nears maximum size as well as
tweaks to disable writing (when near max size).

Two other details:

Have you run bf_compact to compact your wordlist?  That would give you
some room for growth.

It sounds like you're using autoupdate mode, i.e. the "-u" switch.  The
growth rate can be slowed down by using the "update_thresh" option
(either in the config file or on the command line).  I use
"thresh_update=0.01" so that obvious spam (0.99 or above) and obvious
ham (0.01 or below) don't cause database updates.  You could try that
value or a larger one.

Regards,

David