[ale] Seagate 1.5TB drives, bad blocks, md raid, lvm, and hard lock-ups

[Jeff Lightner]

My preference would always be hardware RAID (or near hardware RAID before someone points out that some controllers aren't "real" hardware RAID).

However, software RAID worked for many of my old Linux servers for years.  Also I've used other software RAID technology (HP's MirrorDisk/UX, Veritas Volume Manager) with no real issues.  

In the very least I'd prefer software RAID over no RAID.

-----Original Message-----
From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of Matty
Sent: Thursday, January 07, 2010 12:19 PM
To: Atlanta Linux Enthusiasts - Yes! We run Linux!
Subject: Re: [ale] Seagate 1.5TB drives, bad blocks, md raid, lvm,and hard lock-ups

On Wed, Jan 6, 2010 at 4:38 PM, Greg Freemyer <greg.freemyer at gmail.com> wrote:
> Brian,
>
> If your running raid5 with those drives, you have basically zero fault
> tolerance.
>
> The issue is if one drive totally fails you are almost guaranteed to
> have some bad sectors on the remaining drives.  Those bad sectors will
> prevent mdraid from rebuilding the array fully.  (It may rebuild the
> other stripes that don't have any bad sectors, but definitely the
> stripes that have bad sectors are not rebuildable.).
>
> So at a minimum you need to be running mdraid raid6.  And even then
> you will just achieve raid5 reliability.  (ie. with that size drives
> mdraid raid6 will likely NOT survive a double disk failure.)
>
> And then you need to be running background scans on a routine basis.
> I've forgotten the exact command, but you can tell mdraid to scan the
> entire raid volume and verify the parity info is right.  In theory
> doing that will handle the bad sectors as they pop up.
>
> Unfortunately it sounds like your drives are creating bad sectors
> faster than you can likely force them to be remapped even by routine
> background scans.
>
> Greg
>
> On Wed, Jan 6, 2010 at 3:09 PM, Brian W. Neu <ale at advancedopen.com> wrote:
>> I have a graphic design client with a 2U server running Fedora 11 and now 12
>> which is at a colo handling their backups.  The server has 8 drives with
>> Linux md raids & LVM on top of them.  The primary filesystems are ext4 and
>> there is/was an LVM swap space.
>>
>> I've had an absolutely awful experience with these Seagate 1.5 TB drives,
>> returning 10 out of the original 14 due to the ever increasing SMART
>> "Reallocated_Sector_Ct" due to bad blocks.  The server that the client has
>> at their office has a 3ware 9650(I think) that has done a great job of
>> handling the bad blocks from this same batch of drives and sending email
>> notifications of one of the drives that grew more and more bad blocks.  This
>> 2U though is obviously pure software raid, and it has started locking up.
>>
>> As a stabilizing measure, I've disable the swap space, hoping the lockups
>> were caused by failure to read/write from swap.  I have yet to let the
>> server run over time and assess if this was successful.
>>
>> However, I'm doing a lot of reading today on how md & LVM handle bad blocks
>> and I'm really shocked.  I found this article (which may be outdated) which
>> claimed that md relies heavily on the firmware of the disk to handle these
>> problems and when rebuilding an array there are no "common sense" integrity
>> checks to assure that the right data is reincorporated back into the healthy
>> array.  Then I've read more and more articles about drives that were
>> silently corrupting data.  It's turned my stomach.  Btrfs isn't ready for a
>> this, even though RAID5 was very recently incorporated.  And I don't see
>> btrfs becoming a production stable file system until 2011 at the earliest.
>>
>> Am I totally wrong about suspecting bad blocks for causing the lock-ups?
>> (syslog records nothing)
>> Can md RAID be trusted with flaky drives?
>> If it's the drives, then other than installing OpenSolaris and ZFS, how to I
>> make this server reliable?
>> Any experiences with defeating mysterious lock-ups?

This post on md is somewhat disconcerting, and made me question
whether I should be using MD at all:

http://utcc.utoronto.ca/~cks/space/blog/linux/SoftwareRaidFail

- Ryan
--
http://prefetch.net

_______________________________________________
Ale mailing list
Ale at ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
 
Proud partner. Susan G. Komen for the Cure.
 
Please consider our environment before printing this e-mail or attachments.
----------------------------------
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.
----------------------------------