[ale] ATL Colocation and file server suggestions

[Ken Ratliff]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jan 20, 2009, at 2:56 AM, Pat Regan wrote:

> Ken Ratliff wrote:
>>
>> If drives are randomly dropping out of Linux software RAID  
>> something is
> wrong.  I can only recall having two machine that had random drives
> dropping out of MD devices.  Both showed errors when running  
> memtest86.
> One was a bad CPU, I believe the other was bad RAM (IIRC).

Let's just say I have a good idea what the probable cause was, but I  
can't talk about it in public hehe

>> I actually don't like RAID 10 at all. I'd rather toss the 4 drives
>> into a RAID5 and get more space. Sure, a RAID 10 will allow you to
>> survive 2 dead drives, as long as it's the right 2 drives. I've seen
>> both drives of one mirror fail in a RAID 10 a few times, and that has
>> pretty much the same result as 2 dead drives in a RAID 5.
>
> Redundancy isn't the first reason to choose RAID 10 over RAID 5.  If  
> it
> were, everyone would just choose RAID 6 since that would let you lose
> any two drives.
>
> RAID 5 has a terribly write performance problem.  Doing a random
> uncached write to a RAID 5 involves a read and a write to one stripe  
> on
> every drive.

For the most part, most of our customers aren't doing alot of write  
performance, it's all Read, so RAID 5 works fine. I believe we use  
RAID10 with the customers that get a lot of database activity that we  
can't convince to seperate their sql traffic to another server. (For  
those following along at home, don't ever put your database on a RAID5  
if you actually want it to serve at a decent rate).


> Hardware RAID 1 shouldn't have a write performance penalty.  Software
> RAID 1 (or 10) requires double the bus bandwidth for writes.  I can't
> speak for all implementations, but Linux MD RAID 1 spreads reads out
> over all drives in the raid set.

I think you meant software RAID 1 shouldn't have a write performance  
penalty?

But yeah, I was saying that essentially, software RAID1 doesn't have a  
huge impact on the system, to the point where it's negligible. As  
opposed to a software RAID5, which I've seen help a machine come to a  
crawl under load.

> Your slow rebuilds likely had nothing to do with the performance of
> software RAID 5.  I would imagine you needed to tweak
> '/proc/sys/dev/raid/speed_limit_min' up from the default of 1MB/sec.
> There is very little reason for a hardware controller to beat Linux MD
> at RAID 5, especially on modern hardware.  It only requires one more
> drive worth of bus bandwidth than a hardware controller would require.
> Processors have always been able to compute parity faster than current
> hardware cards.  dmesg on my laptop tells me that I can compute RAID 6
> parity at 2870 megabytes per second.

No the slow rebuilds usually have to do with the fact that the server  
is live and doing well over 100mb/s of traffic, when the server is  
under load and the CPU is pegged, the parity calc for an array  
rebuilding doesn't help things. Of course, neither does the iowait.

> Second, have you ever had the annoyance of replacing a failed drive  
> with
> another of the same make/model and the replacement drive is in fact
> smaller than the failed drive?  Whenever I use software RAID I  
> sacrifice
> a few percent off the end of the drive just to keep this from  
> happening.

Yes I have. Western Digital 400 gigger. Same model number, but the  
drive I grabbed was of a lower revision number. The server saw it as 1  
gig smaller than the other WD drive, so obviously, I couldn't use it  
to rebuild the array.

So I said screw that and grabbed a 400 gig barracuda and it stopped  
giving me a problem ;)

> I'm up near 4 TB at home.  That isn't even a big number anymore! :)

> I had 4 TB back in 2001.  That was mighty expensive, though, and it  
> sure
> wasn't a single volume.  I only mention this so you don't just think  
> I'm
> some punk with a RAID 5 on his TV blowing smoke :)

You've got me beat, I think I've got something in the area of 3.5 TB  
total of local storage, with 2 of that being in an array on my file  
server, I didn't build that until '04. Sure was fun walking into work  
and seeing someones eyes bug out when I mentioned I had a 2 TB array  
though. Now, even I'm disdainful of it! 'Only 2 TB? God, I'm such a  
loser!'. As soon as I feel like  I can trust the Seagate 1.5 TB  
drives, I'll probably be upgrading my big array though.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (Darwin)

iEYEARECAAYFAkl2ClYACgkQXzanDlV0VY6w9ACg6ufdvd4yYsoZK6BXA5bTkfcn
wCUAnAvwgYADHWA/zKcqtDOZJNO27f9F
=7uaM
-----END PGP SIGNATURE-----