[ale] Confusing RAID Performance

[Greg Freemyer]

On Wed, Feb 2, 2011 at 7:08 PM, David Tomaschik
<david at systemoverlord.com> wrote:
> On 02/02/2011 06:43 PM, Greg Freemyer wrote:
>> On Wed, Feb 2, 2011 at 6:11 PM, David Tomaschik
>> <david at systemoverlord.com> wrote:
>>> On 02/02/2011 04:23 PM, Jeff Hubbs wrote:
>>>> I sure wouldn't.  For >=~1TB drives, the probability of having an
>>>> unrecoverable read error among all the drives at recover time starts
>>>> becoming significant.  Sure, you can use it - as long as a restore from
>>>> tape, etc. is an acceptable fallback if you can't rebuild after a drive
>>>> replacement.
>>>
>>> Actually, just 3 hours ago, I had a storage consultant telling me about
>>> a whitepaper that indicated a 20%/year chance of failure on a 5-disk
>>> RAID 5 of consumer grade SATA hard drives.  I can't find the paper he
>>> refers to, and the numbers seem high to me, but hey, I'm not going to
>>> chance it.
>>>
>>> David
>>
>> Seems low to me for 1 TB drives.  And I've run the numbers with 2 TB
>> drives.  (I don't recall exactly what the results were off-hand.  But
>> they were scary.)
>>
>> Simply don't use raid 5 with large drives if you don't want to lose
>> your array from time to time.
>>
>> Greg
>>
>
> That seems low?  I'm trying to calculate the numbers now, but figuring
> out bitwise errors, MTBF, sectors, etc, has got me getting weird
> results.  I'd rather just find the whitepaper from the storage experts.
>
> David

I consider myself a storage expert!
===
I'm going from memory, but you have about a 20% chance of a given
drive dieing in a giving year.

Or a 80% chance of it not.

So for a 5-disk array, the odds of no recoveries being needed in a
given year is 80% *  80% * 80% * 80% * 80%

Or about 33%.  (I had to use a calculator for that one!)

So you have a 66% chance of needing to do at least one recovery every year.

Then if I recall right, a 2 TB drive has about a 1 in 3 odds of having
a non-recoverable read error somewhere on it.

If so, a 1 TB drive has a 1 in 6 odds of having a non-recoverable read
error.  (It's a simple linear relationship.  The spec should give you
the error rate.  Just multiply the errors per sector times the number
of sectors on your drives.  But somewhere between 5 and 30% is what I
would expect.  If you're drastically different than that, check your
math.)

So if you have 4 drives from which to recover, the odds of one of them
having a non-recoverable error is 4 times 1 in 6.  That is 4 in 6, or
66%.

If you have both a failed drive and a non-recoverable error, then the
array fails in total because the rebuild fails.

So that's a 66% chance of a recovery being needed * 66% chance of a
failure during the recovery.

That's about a 44% chance of a full array failure every year.  Again
assumes raid 5 with 5x1 TB drives.

The above is from memory, but if you're looking at specs, you can
adjust the above as needed.  Just don't trust the lifetime specs.
Drives don't last as long as the specs say from the studies I've seen.
 A 20% chance of drive failure per year for a given drive is not that
far off of reality.

fyi: Running a raid scrubbing routine can improve these numbers, but
not all raid controllers support that.  Linux MDraid does, but you are
responsible for setting it up in cron.  I would not trust a scrubber
personally.  Meaning, I would set it up, but I would not depend on it.

A raid scrubber reads all the sectors of all the drives on a regular
basis and any non-recoverable errors it finds, it corrects by
calculating the correct data from the other raid members, then writing
the correct data back out.  If you have a hardware raid controller,
scrubbing has to be implemented in the controller itself, not via the
linux kernel or userspace.

Greg