procmail: Timeout, terminating "/usr/bin/bogofilter"

[Matthias Andree]

Daniel Leidert <daniel.leidert.spam at gmx.net> writes:

> Am Sonntag, den 14.01.2007, 18:47 +0100 schrieb Matthias Andree:
>> Daniel Leidert <daniel.leidert.spam at gmx.net> writes:
>> 
>> R> Hello,
>> >
>> > I went away a few days ago and back today I wanted to fetch all my mails
>> > (ca. 10000 at once). My system consists of: fetchmail -> exim ->
>> > procmail -> (several filters, bogofilter, spamassassin, clamassassin) ->
>> > cyrus
>> >
>> > And today I found some weired behaviour. There were more and more
>> > bogofilter processes - the amount increased up to 600-1000 in a really
>> > short time (only 2000-3000 mails were downloaded), _but_ bogofilter
>> >
>> Is that heavy-duty server class hardware, with battery-backed RAID10 or
>> RAID50 or stuff?
>
> No. A simple system, not a high-end one. It's just m private
> mail-server.

Yup. Standard ATA stuff isn't made to handle this kind of workload. Even
SCSI will probably show massive trouble unless you enable battery-backed
write caches. Solid state disks might do though.

>> I'd say avoid running more than a few dozen bogofilter processes at the
>> same time, the more you run, the more contention you'll experience
>
> Logical. Because of this I asked for a build-in possibility to reduce
> the number of running processes.

A local lockfile in procmail should do the job and not execute more than
one bogofilter process at a time. If you have, for instance,

:0fw
| bogofilter -e -p

make that (mind the colon at the end of the :0 line):

:0fw:
|bogofilter -e -p

>> Last time I checked, several years ago, Exim 3.34, it was not at all
>> robust towards increasing load.
>
> There is an queue_only_load option I never used. But for now I reduced
> the amount in smtp_accept_queue_per_connection and removed the
> auto-update switch of bogofilter (-u).

If I recall that option's functionality correctly, it doesn't help in
your problem.

Running not too many delivery processes in parallel is what you're
after, but I don't know how to word this as an exim.conf block.

-- 
Matthias Andree