datastore vs. t.lock2

[Matthias Andree]

On Fri, 31 Jan 2003, Shawn Michael wrote:

> Try the following options to mount:
> 
> defaults,noatime,data=writeback
> 
> The data=writeback helped tremendously under heavy IO and the noatime helps 
> to reduce needless IO.  How many people acutally need atime?

Let me first of all point out that some database code changes went into
bogofilter 0.10.1.4 that reduce needless synchronous I/O which is far
more expensive than just writing a single block asynchronously. The
locking test which used to run 10 processes simultaneously went down
from 57 s wallclock time on SCSI + ext3 to 4 s, and from 22 to 3.5 s on
ext2 -- it avoids the unnecessary fsync() calls by passing DB_NOSYNC
when closing a database right after opening it. The test wouldn't even
complete in 4 minutes on a reiserfs-mounted /tmp file system, since the
changes, it completes in less than 4 s. See my benchmark mail in this
list for more detail.

On the tuning side, there is also a finer-grained control for ext2 and
ext3, you can use chattr +A on distinct files to turn atime updates off
for these files, or chattr +A -R ~/.bogofilter for the whole directory.

With high write load to data bases, the chattr +j attribute (ext3 only)
might be worth trying, it forces all data to go through the journal
first, which may improve the write speed because less seeks are needed
before fsync() returns -- the data can be written synchronously and
sequentially to the journal. Of course, tagged command queueing will
also help, currently only available for SCSI on most operating systems
and for some more recent IBM ATA drives (DPTA, DTLA, IC35L...) on
FreeBSD.

And of course, the journal size needs to be tuned for the load, but I
cannot give recommendations here how to analyze if it needs to be bigger
or smaller and by what amount.

-- 
Matthias Andree