[ale] RAID mirror boot nightmare

Tue Jul 10 08:41:30 EDT 2012

Can you also send the output from /proc/mdadm ?

On Tue, Jul 10, 2012 at 1:33 AM, Bob Toxen <transam at verysecurelinux.com> wrote:
> Additional details on this miserable problem:
>
> On Boot the kernel complains of:
>
>   Creating root device
>   Mounting root filesystem
>   Mount: Could not find filesystem '/dev/root'
>
> after talking about md0 apparently being created successful and lastly
> panics.
>
> /boot/grub/grub.conf entry  being booted:
> title CentOS-single-md4
>         root (hd0,0)
>         kernel /vmlinuz-2.6.18-308.4.1.el5 ro root=/dev/md4 md=4,/dev/sda6,/dev/sdb6 md=1,/dev/sda2,/dev/sdb2 md-mod.start_dirty_degraded=1 rhgb single noresume
>         initrd /initrd-2.6.18-308.4.1.el5.img
>
> /etc/mdadm.conf (heavily edited by me including switching from uuid to
> devices; I don't presently list swap as that is not critical and it
> fails before even thinking about swap):
> # mdadm.conf written out by anaconda
> DEVICE /dev/sda[26] /dev/sdb[26]
> MAILADDR root
> ARRAY /dev/md4 level=raid1 num-devices=2 devices=/dev/sda6,/dev/sdb6 auto=yes
> ARRAY /dev/md1 level=raid1 num-devices=2 devices=/dev/sda2,/dev/sdb2 auto=yes
>
> fdisk output:
> Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
> 255 heads, 63 sectors/track, 121601 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
>
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sda1   *           1          13      104391   83  Linux
> /dev/sda2   *          14          26      104422+  fd  Linux raid autodetect
> /dev/sda3              27        4200    33527655   82  Linux swap / Solaris
> /dev/sda4            4201      121601   943023532+   f  W95 Ext'd (LBA)
> /dev/sda5            4201       62900   471507718+  83  Linux
> /dev/sda6           62901      121600   471507718+  fd  Linux raid autodetect
>
> /etc/fstab:
> /dev/md4        /                       ext3    defaults        1 2
> /dev/md1        /boot                   ext3    defaults        1 2
>
> #normal /dev/md3        /                       ext3    defaults        1 1
> #normal /dev/md0        /boot                   ext3    defaults        1 2
> #normal /dev/md4        /root2                  ext3    defaults        1 2
> #normal /dev/md1        /boot2                  ext3    defaults        1 2
> tmpfs                   /dev/shm                tmpfs   defaults        0 0
> devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
> sysfs                   /sys                    sysfs   defaults        0 0
> proc                    /proc                   proc    defaults        0 0
> /dev/md2                swap                    swap    defaults        0 0
>
>
> What magic am I missing?  Please help!!!
>
> Bob
>
> On Mon, Jul 09, 2012 at 06:21:43PM -0400, Bob Toxen wrote:
>> I'm having problems getting my RAID mirror booting again in
>> CentOS 5.8 64-bit after GNU's gparted trashed my partition table when
>> one of the new Seagate 1 TB SATA disks failed completely.
>>
>> Booting causes the kernel to complain of an invalid root partition.
>> I can't figure out what is invalid about it.  FSCK is happy.
>>
>> I rebuilt the partition table with "CD Rescue" v2.7.1 bootable CD
>> (wonderful stuff) and reinstalled my existing GRUB to the MBR.
>>
>>
>> Unfortunately, CentOS 5.8 will no longer recognize my md3 root partition
>> (still running in degrated mode since I've not yet replaced the failed
>> disk).  The CentOS kernel does see the RAID partitions but just doesn't
>> want to use one for /!!!
>>
>> I CAN boot the "CD Rescue" disk and tell it to find and boot an
>> existing OS on the hard disk and it will bring up my CentOS 5.8 on
>> md3 (before I did mdadm -S /dev/md3 and used fdisk to change its
>> partition type from RAID to Linux)
>> or will boot up md4 (though apparently with a newer 3.x kernel from the CD).
>>
>> However, no matter what I do I cannot get my CentOS 5.8 to boot the RAID
>> mirror.  I also have the root and /boot file systems duplicated to
>> md4 and md1 respectively.
>>
>> I tried
>>
>>   mdadm -S /dev/md3
>>   mdadm -S /dev/md1
>>   mdadm --create -n 2 /dev/md3 --level=mirror /dev/sda6 missing
>>     "/dev/sd6 appears to be part of an existing RAID; do anyway" y
>>   mdadm --create -n 2 /dev/md1 --level=mirror /dev/sda2 missing
>>     "/dev/sd2 appears to be part of an existing RAID; do anyway" y
>>
>> but when I try to boot CentOS 5.8 with "root-md3" it again fails.
>>
>> However, booting the CD Rescue and selecting "find and boot from hard
>> disk" happily boots either RAID partition.
>>
>>
>> Please PLEASE give me some ideas to fix this.  $50 to the first right
>> answer, REALLY!  I'm desperate.
>>
>> Nothing on the file systems themselves, including  the /boot/initrd*
>> file and kernels, appear to be damaged.  systems themselves)
>>
>>
>> Since FSCK is happy all I can think of is damage to the md superblock at
>> the end of the physical partitions.  I wonder if the CD Rescue kernel
>> (being newer than the CentOS kernel) created to new a version of the
>> mdsuperblock.  However, my recreating the RAID mirror under CentOS 5.8
>> should have fixed it, shouldn't it?
>>
>> HELP!  $50 prize.  Really!
>>
>> Bob Toxen
>> bob at VerySecureLinux.com
>> transam at VerySecureLinux.com
>> 404-216-5100 Cell (except for the next half hour)
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://mail.ale.org/mailman/listinfo/ale
>> See JOBS, ANNOUNCE and SCHOOLS lists at
>> http://mail.ale.org/mailman/listinfo
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo