[ale] New Data - If you own (almost) ANY HDD, repartition with 4k boundaries
Ron Frazier (ALE)
atllinuxenthinfo at techstarship.com
Sat Jan 12 16:38:37 EST 2013
Hi all,
I've been having more painful learning experiences with the new 4K
sector HDD that I bought. It turns out that the partitioning data I
posted previously, in this thread by the same title except for the "New
Data - " part, was just part of the story. It turns out that there are
about 9 things you may have to do to get a 4K drive to work if
transferring an existing operating system, as I was. What I thought
would be a 3 hour install has turned into a week of pain. I want to
document it all so someone else can avoid these potholes.
Here's what happened recently. I originally posted this first part in a
thread on data corruption, but decided to move it here instead.
I just replaced my laptop HDD with a Seagate 750 GB hybrid drive. I
normally run windows 7, but boot into Linux periodically. It ran fine
for a while, but later, after a number of reboots, windows update
started failing. I spent an hour on the phone with a MS tech. We did
all kinds of things and couldn't repair the system. Nothing we did
would repair the errors. Eventually, I reinstalled the OS on top of
itself, which failed and said that setup could not install this OS, the
same one that was running before, on this pc. Before I installed the
new drive, I cloned my original drive to a 500 GB backup drive that I
had. I put the old cloned backup drive in instead, and it works like a
charm. Windows updates work, and everything is fine.
I have now followed the steps I'll outline below and I THINK the drive
is fine, and I THINK everything is working. I'm going to have to test
the drive for a month or so and make sure everything keeps working.
Right now, gparted is repartitioning the drive.
Almost all the concepts I'm going to share are probably relevant to
Linux too. While I was working with Windows, and the steps for Linux
may be different, the concepts are still valid. Once I'm through
getting Windows to work, I'm going to have to get my Ubuntu partition to
work. In googling around, I found one thread on a forum where the
original poster had done some testing with this drive and found data
corruption. It was not immediately obvious. The first boot after
cloning worked fine. Then after about 12 boots into Windows, things got
damaged beyond repair. He did some tests with Linux and found data
corruption after about 60 boots. I'll share what little Linux specific
information that I can but I haven't had much chance to play with that
part of my dual boot systems yet with a new 4K drive.
Note: If ANY bios or software in your system was created prior to 2011,
you may need to update them.
Note: Even if you install a drive and it boots and appears to work, you
could still be getting data corruption
Note: Even if the drive is telling the system it has 512 byte logical
sectors, as they pretty much all do at this point, some functions depend
on being able to determine the actual size of the physical sectors. If
they cannot get this data, it can lead to data corruption.
Step 1) You need an AHCI SATA compatible bios and it needs to be set to
AHCI mode rather than IDE. That was already true on my machine. I'm
not sure of what goes on under the covers with these different settings,
but this does appear to be a requirement. If you're not using SATA, I
can't help you.
Step 2) You need the proper 4K compatible driver for your storage
controller installed BEFORE you clone your OS to the new 4K drive. If
your driver was made before the last half of 2010, you probably need to
update it. Note that, if I just went to device manager and clicked on
the SATA controller and clicked upgrade driver, it said it was up to
date. I had to install an alternative driver. In my case, it was the
Intel Rapid Storage Technology driver, downloaded from the laptop
manufacturer's website. This driver was created in late 2010 in my case.
One key function that the driver must provide to the OS is the ability
to test the actual physical size of the sectors. On a 4K drive in
"emulation" mode, this should show 4096 bytes while the logical sector
size should show up as 512 bytes. I'll show you how to check this later
on Windows 7 and on Linux (if I can find the method). In my case, after
I cloned my old HDD to the new one, the physical sector size showed up
originally as <not supported>. I believe this is what led to the data
corruption that I experienced. Now, when I do the test, it shows up as
4096 bytes, as it should.
Note: I said you have to have the advanced driver in place BEFORE you
clone your OS to the new drive. If you clone, then boot, on an old
driver, you run the risk of corrupting the drive. So, in my case, I
began running on my backup drive which I had before this all started. I
then installed all drivers and patches, THEN, I cloned the OS over to
the new drive. That way, when the system boots up on the new drive, it
has everything it needs.
Note: This can bite you if you're installing an OS from an install
disc. When I tried to do a repair install with the Windows 7 DVD (which
I had bought for upgrading another machine), it hummed and drummed for
an hour then said it couldn't install on my machine. I already had
Windows 7 running on this machine !!! I later learned that it was
likely because the install disc didn't have the driver for this SATA
controller which was 4K compatible. So, you somehow have to interrupt
the install process and provide a driver separately. I haven't done
that, so I can't tell you how.
You must check that the updated driver is in use after cloning the
system and booting the new HDD. In Windows 7, you can run the device
manager and check the properties of your SATA controller. The driver
date should reflect a modern version.
Step 3) You must have an OS / Kernel that is 4K compatible AND that has
all the relevant patches for 4K compatibility BEFORE booting the OS on
the 4K drive. In the case of Windows 7, it means you must have SP1. If
you're installing from an install disc, it should have SP1 already
incorporated into it. In my case, I had to verify that all this was in
place while running on my old backup drive. The links below provide
links to hotfixes for Vista and Windows 7 you can put in place if your
PC doesn't already support it.
You must check that the OS patches for 4K support are in place after you
clone the system and boot the new HDD. In Windows 7, you can try to
install the 4K support hotfixes, and it will hopefully say you've
already done it.
I have not yet tested Ubuntu extensively on the new drive. I think
Ubuntu 11.04 is 4K compliant. I did some googling before writing this
and have not been able to confirm yes or no. I found this link, which
is a bug report that says Ubuntu 11.04 beta was installing on sector 63,
which is not encouraging. I don't know if the final version had this
problem.
http://osdir.com/ml/ubuntu-bugs/2011-06/msg13778.html
Step 4) Your OS must be able to identify how big the physical sectors
are on the HDD. You must check that it has this capability after
cloning to the new 4K HDD and booting it. In Windows 7, start an
Administrator DOS prompt and type the following:
fsutil fsinfo ntfsinfo c:
You should get something like this. Note that, in both the examples
below, I typed in the 4096 in the report, since the drive I'm running on
with this machine that I'm typing on is not a 4K drive.
C:\>fsutil fsinfo ntfsinfo c:
NTFS Volume Serial Number : 0xaa0e0e980e0e5e27
Version : 3.1
Number Sectors : 0x00000000124f3ff0
Total Clusters : 0x000000000249e7fe
Free Clusters : 0x0000000000f72cee
Total Reserved : 0x0000000000000870
Bytes Per Sector : 512
Bytes Per Physical Sector : 4096
Bytes Per Cluster : 4096
Bytes Per FileRecord Segment : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length : 0x0000000011bc0000
Mft Start Lcn : 0x00000000000c0000
Mft2 Start Lcn : 0x000000000124f6bf
Mft Zone Start : 0x00000000010d15e0
Mft Zone End : 0x00000000010d4640
RM Identifier: 53EC96E6-9894-11DF-A4D2-BAEDAA1ED730
Note: The "Bytes Per Sector" is the size of the logical sector. The
"Bytes Per Physical Sector" is the size of the physical sector. If the
latter does not appear, or if it says something weird like <not
supported>, as it did for me originally, you have a problem. You must
resolve this problem before running the drive for any period of time or
you risk data corruption. After I installed the new driver, my printout
looked similar to the one above. On an older style drive, the "Bytes
Per Physical Sector" will also be 512.
I THINK the following will give you the equivalent info in Linux. I
copied this from a forum post and haven't tried it.
|sudo fdisk -l /dev/sda
|where /dev/sda is the hard disk in question
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c1f6b
Look at the line that says "Sector Size (logical/physical)". Again, for
an old style drive, the last number would be 512 bytes.
You Linux gurus can tell me if this last part is correct or if there is
a better way to do this.
Step 5) You must be able to update the OS with patches. In my case, as
far as I can tell. The lack of physical sector size reporting corrupted
the Windows Update patch database. After that, Windows Update refused
to run. Nothing the MS tech had me do on the phone could fix it,
because we were chasing the wrong problem. There are ways to erase the
database and restart, but it's possible it could get corrupted again.
So, I installed all the relevant patches on my old backup drive. After
I finish repartitioning, I expect the system to boot and I expect
Windows Update to run normally. I also expect it to run normally a
month from now, and will be testing that.
Step 6) After cloning your old system to the new 4K HDD, unless your
cloning software automatically aligned the partitions beginning and
ending on 1 MiB boundaries, as described previously in this thread,
repartition the new 4K HDD with a new gparted live boot CD.
Step 7) Using an engineering calculator and gparted as previously
described, check that the starting sector and total number of sectors in
each partition and each gap between partitions is an integer multiple of
2048.
Step 8) If you moved the first partition of the HDD to sector 2048 as
recommended, you probably broke your bootloader. For Windows 7, boot a
Windows install disk and select repair.
If your bootloader is Grub, etc., you'll probably have to fix that. One
of you Linux guys chip in if you like and tell how to fix it.
Step 9) Boot your new HDD and verify that your bootloader works and that
the OS boots properly.
Then, RECHECK the following:
recheck Step 2), that your new SATA controller driver is in place
(if you had to update it)
recheck Step 3), that all the relevant 4K related OS patches are
in place
recheck Step 4), that the OS can identify the number of physical
sectors on the HDD
recheck Step 5), that the OS system updates work properly
recheck Step 6), that the partitions on the HDD are aligned on 1
MiB boundaries
Step 10) DO NOT ASSUME THE NEW DRIVE IS WORKING PROPERLY JUST BECAUSE
THE SYSTEM APPEARS TO BE RUNNING OK. Treat the drive like a criminal
suspect for a while. Be on the lookout for data corruption. Keep
frequent backups of critical files. Do not erase the backup that you
cloned your system from until you've tried out your new drive for a
while and know things are working well. Do not immediately backup your
new drive onto your old backup drive. If you do eventually clone your
new drive to a backup drive, and every clone it back, make sure the
partitions stay aligned.
I've done a bit of research on the Linux compatibility issues with 4K
drives, but haven't had a chance to explore it in depth. You guys that
have more detailed knowledge, feel free to jump in.
I found this:
http://flashdba.com/4k-sector-size/
About 3/4 of the way down, it says this:
"Production Linux kernels prior to 2.6.32 do not have support for
advanced format drives, which means LUNs presented to them must appear
as if they are 512 byte. So the 2.6.18 kernel for example is not able to
support 4k sector drives -- that means all Red Hat Enterprise Linux 5
distributions, all Oracle Enterprise Linux 5 distros using the
RHEL-compatible kernel, SUSE Linux Enterprise Server 10 etc. That's a
lot of versions of Linux, many of which are happily running Oracle
databases today.
The 2.6.32 kernel, as well as later versions, is much happier about 4k
sector sizes and even has a new SYSFS flag to show (or set) whether a
block device relates to rotational media or not (so you would see a zero
for flash):"
I was unable to find further corroboration of this in a short period of
time.
I also found this:
http://ubuntuguide.net/ubuntu-11-04-upgrade-linux-kernel-to-2-6-39-0
Near the top it says:
"Default kernel come with Ubuntu 11.04 is 2.6.38-8"
So, it sounds like Ubuntu 11.04 (which I'm currently running), is OK for
the 4K drive.
The take away from this, for me, is that any Linux distro released prior
to the beginning of 2011 or so may NOT have 4K capability and you could
risk severe performance hits or data corruption using them on a 4K
drive. Again, you experts feel free to jump in.
Well, that's the state of affairs on my new 4K drive at the moment.
Once the repartitioning is done, and I recheck some parameters, I hope
to actually get to use the drive after a week of pain. I never
anticipated a simple HDD upgrade would be so unsimple. I hope this
account will help others working on similar problems. If you've had
experience with the issue, please add it.
Sincerely,
Ron
On 1/12/2013 12:48 AM, Ron Frazier (ALE) wrote:
> I should mention that hotfixes fix most problems in win vista and 7.
>
> Ron
>
>
>
> Justin Goldberg<justgold79 at gmail.com> wrote:
>
>
>> Time to go back to fat32 for drives<2TB.
>> Hi Phil and others,
>>
>> I discovered some really ugly complications regarding using NTFS with
>> 4K
>> drives. It doesn't look good for XP. Anyone who maintains Windows
>> systems
>> needs to see these MS support docs.
>>
>> http://support.microsoft.com/**kb/2510009<http://support.microsoft.com/kb/2510009>
>> http://support.microsoft.com/**kb/982018<http://support.microsoft.com/kb/982018>
>> http://support.microsoft.com/**kb/2553708<http://support.microsoft.com/kb/2553708>
>>
>> Sincerely,
>>
>> Ron
--
(To whom it may concern. My email address has changed. Replying to former
messages prior to 03/31/12 with my personal address will go to the wrong
address. Please send all personal correspondence to the new address.)
(PS - If you email me and don't get a quick response, you might want to
call on the phone. I get about 300 emails per day from alternate energy
mailing lists and such. I don't always see new email messages very quickly.)
Ron Frazier
770-205-9422 (O) Leave a message.
linuxdude AT techstarship.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20130112/96247376/attachment-0001.html>
More information about the Ale
mailing list