Memory leak in -l creation (Was: Bogofilter-0.15.9 - New Current Release)

[David Relson]

On Mon, 24 Nov 2003 08:00:47 -0800
Chris Wilkes <cwilkes-bf at ladro.com> wrote:

> On Mon, Nov 24, 2003 at 08:50:30AM -0500, David Relson wrote:
> > On Sun, 23 Nov 2003 21:42:43 -0800
> > >  
> > > So it appears that the file created with the -l has some sort of
> > > lock on the memory, as not creating a new file doesn't cause the
> > > memory to dip at all.
> > > 
> > 
> > My bet is that RH9 is caching the wordlist.db-new files in ram. As
> > soon as you deleted them, the memory usage dropped because it
> > recognized that the data was no longer needed.
> > 
> > As a test, try using "strings < $i > $i-new" in your script and see
> > what happens.
> 
> You're right!  Same occurances: memory dips down until the
> wordlist.db-new's are deleted.
> 
> So my beef is now with RH9.  Perhaps that's where my memory is going
> -- if anyone has any ideas on this send them to be off-line and I'll
> summarize.
> 
> Chris

Chris,

Linux tends to cache as much as it can.  It will happily use all
available memory for disk caching.  As soon as a task _really_ needs
some memory, the kernel will start discarding cached info and/or
swapping to disk to fill the _real_ needs.  

I've noticed that a lot recently with bogotune.  If my only memory user
is bogotune, with a 40,000 message test I've been using, I'll see its
VIRT and RES memory both go to 128M and stay there.  If I then start
other programs that need memory, I'll see the RES mem drop to 68M.  The
68M size is stable and there's no swapping going on, so I "know" that's
the working set size.  However, without the additional programs, RES
will stay at 128M.

As the say, "it's not a bug, it's a feature".

Cheers,

David