Memory leak in -l creation (Was: Bogofilter-0.15.9 - New Current Release)

[David Relson]

On Sun, 23 Nov 2003 21:42:43 -0800
Chris Wilkes <cwilkes-bf at ladro.com> wrote:

> On Sun, Nov 23, 2003 at 04:22:15PM -0500, David Relson wrote:
> > Bogofilter 0.15.9 is available on SourceForge.  The download URL is
> > http://sourceforge.net/project/showfiles.php?group_id=62265
> > 
> > * Fixed a memory leak in bogoutil.
> 
> Think I found a source of my memory leak.  When I did a 
> 
>   for i in */.bogofilter/wordlist.db ; do
>     echo $i;
>     bogoutil -c 1 -d $i | bogoutil -l $i-new;
>   done
> 
> on my 100+ mailboxes the free memory dropped from 400MB to 100.  Then
> when I deleted the wordlist.db-new files the vmstat memory usage shot
> up that second from 93MB to 433MB.
>  
> So it appears that the file created with the -l has some sort of lock
> on the memory, as not creating a new file doesn't cause the memory to
> dip at all.
> 
> RedHat 9.0
> db4 4.1.25-3
> 
> Chris

Chris,

I just ran your script with 4 wordlists of 62mb each.  With vmstat and
top running in separate terminal windows, I could see that bogoutil is
using less than 1100 units (kilobytes?) of VIRT and RES memory and that
there is binary i/o (as expected), but no swapping (on my 384M machine).
 

My bet is that RH9 is caching the wordlist.db-new files in ram.  As soon
as you deleted them, the memory usage dropped because it recognized that
the data was no longer needed.  

As a test, try using "strings < $i > $i-new" in your script and see what
happens.

David