[ale] mysql and high load levels

[Joseph Andrew Knapka]

djinn at djinnspace.com wrote:
> 
> This showed up on the mysql list...and it caught my attention given what you guys
> are saying about 2.4 and its wierd memory handling....
> This post (below) describes my situation almost exactly...he's using more
> 'spensive hardware than I am, but scale it down a bit and it's *exactly* what I'm
> seeing.  I'm also seeing 98% RAM usage and even swapping on my web
> server....which has 1GB of RAM and should NEVER need to swap.  Ever.  Even with
> lots of PHP usage...the old server with 256MB ram never swapped.
> 
> So...given this and ya'll's advice...should I drop back to 2.2.19 and forget 2.4
> altogether, or should I disable swap, compile himem into my 2.4.7 kernel, and see
> if it fixes it?

Why did you go to 2.4? If you don't need some feature that's
only in 2.4, you'd probably be best to go back to 2.2.x on
general principles.

I'm not convinced you're seeing exactly the same
problem, though. When the MM system goes nuts in 2.4,
everything just... stops, because every process spends
most of its time waiting for disk I/O to complete on
swapped pages (thrashing). CPU usage would tend to be
low in that kind of scenario. It looks like your mySQL
processes are not having any trouble getting enough RAM
to run their little buns off, so it might be a different
issue.

Incidentally, it's not unusual to have very low amounts
of "free" RAM, even when things are working 100% correctly.
Linux caches pages for as long as possible in order to
make page faults servicable quickly, and those cached pages
show up as cache or buffer space in "top" et al, not as free
RAM. In most cases cache pages are essentially free RAM,
because they can be freed immediately if necessary.

Cheers,

-- Joe
 
> jenn
> 
> <originally posted on lists.mysql.com by  Jeremy Zawodny >
> -------------------------------------------------
> We had a couple of Linux 2.2.19 servers with 2 CPUs (P3-850) and 1GB
> of RAM.  They were solid and they flew.  The never swapped.  They had
> well tuned my.cnf files.  But we needed large files and ReiserFS, so
> we upgraded to 2.4.9.
> 
> After the upgrade, IO performance on the master server began to suck
> at backup/snapshot time.  In fact, Linux claimed that it was running
> out of RAM.  So we added RAM (a 2nd gigabyte).  The problem didn't go
> away.
> 
> So I poked around and watched it really closely for a couple days.
> What I found is that the server would run pretty well until I
> initiated some other type of intensive IO in addition to MySQL.  For
> example, I'd copy a 3GB data file from the main RAID-5 data volume to
> /tmp.  As it copied, Linux decided to swap out most of mysqld (which
> is in the neighborhood of 400MB) to use it for a disk buffer.  This
> not only killed MySQL performance, because it had to swap those pages
> back into physical memory as soon as they were needed, it also made
> the copy take much longer than it should have.
> 
> I could go on, but the pattern was clear.  Linux tried to use as much
> memory as it could for disk buffering--even if it meant "stealing" RAM
> from mysqld.
> 
> The solution?  I disabled swap and it's been humming along just fine
> ever since.  After all, with 2GB of RAM and a good understanding of
> what our RAM requirements really are, the system should never swap.
> -----------------------------------------------------
> 
> Joseph Andrew Knapka wrote:
> 
> > Jeff Hubbs wrote:
> > >
> > > Jenn -
> > >
> > > If it were me - and please understand that I have very, very little
> > > experience with MySQL - but I'd try going back to a 2.2 kernel first.
> >
> > Seconded (including the disclaimer :-). There seem to be a number of
> > ways the 2.4 kernels can snafu WRT memory management. When your load
> > goes up, what process(es) are using all that CPU? I'll place a small
> > wager on kswapd.
> >
> > Cheers,
> >
> > --
> > # Joe Knapka
> > # "You know how many remote castles there are along the
> > #  gorges? You can't MOVE for remote castles!" - Lu Tze re. Uberwald
> > # Linux MM docs:
> > http://home.earthlink.net/~jknapka/linux-mm/vmoutline.html
> > --
> > To unsubscribe: mail majordomo at ale.org with "unsubscribe ale" in message body.
> 
> --
> To unsubscribe: mail majordomo at ale.org with "unsubscribe ale" in message body.

-- 
# Joe Knapka
# "You know how many remote castles there are along the
#  gorges? You can't MOVE for remote castles!" - Lu Tze re. Uberwald
# Linux MM docs:
http://home.earthlink.net/~jknapka/linux-mm/vmoutline.html
--
To unsubscribe: mail majordomo at ale.org with "unsubscribe ale" in message body.