Large Scale

[Matthias Andree]

Thiago Damas <tdamas at gmail.com> writes:

>    Anyone are using bogofilter in a large scale environment? How can I
> share my wordlists between several machines?

>   I'm using FreeBSD 5.3 and I don't know if the lock mechanism in the
> NFS are stable enough.

While it is likely stable enough, it is insufficient if you use one of
the transactional datastores. Non-transactional Berkeley DB might work,
I'm not sure how well mmap() and NFS mix on FreeBSD 5.
(Linux 2.6 and Solaris 8 or newer are probably fine.)

If you're using bogofilter 0.92.8 (i. e. non-transactional) from FreeBSD
ports, you'll have to try if NFS is sufficient or performs well enough,
but note that the transactional features of newer bogofilter versions
(such as the 0.94.X versions available from bogofilter.org and
sourceforge.net) cannot safely be used with networked file systems of
any kind (AFS, CIFS, Coda, NFS, SMB) to my knowledge (coherency issues),
and you cannot mix in FreeBSD 4 clients as their NFS clients do not
support locking.


If NFS is not an option or you wish to use some transactional database
for robustness, you might consider making updates only to one database
that I'll call "master" and copy that to other machines ("slaves") with
rsync.

Two things:

1. make sure to use the same block size for rsync as is the page size of
   the database file (use db_stat or in newer 0.94 versions bogoutil to
   find out the page size)

2. if you are using the transactional version, copy the *.db files first
   and then the log.* files last, and run recovery on the "slave"
   machine after the copy to make sure the latest updates are merged
   from the log.* files into the *.db files.

   bogofilter cannot detect the need to run recovery in this situation,
   so you must do this manually (only on the slaves).
   Regular recovery is sufficient.

-- 
Matthias Andree
_______________________________________________
Bogofilter mailing list
Bogofilter at bogofilter.org
http://www.bogofilter.org/mailman/listinfo/bogofilter