[ale] Onboard RAID

[Michael B. Trausch]

On 11/16/2011 06:28 PM, Brian Mathis wrote:
> On Wed, Nov 16, 2011 at 4:53 PM, Michael B. Trausch <mike at trausch.us> wrote:
>> On 11/16/2011 04:12 PM, Brian Mathis wrote:
>>> There is a world of difference between "hardware" BIOS RAID and a real
>>> RAID card like a PERC H700.  Please do not throw both of those things
>>> in the same category.
>>
>> Nobody is doing that.
> 
> 
> It is important to make the distinction between fake-RAID and real
> hardware RAID.  Up to this point in the thread, this had not been
> mentioned.

I mentioned the difference between them in my original reply, when I
mentioned that fakeraid is typically essentially real-mode
firmware-based RAID.  We could go into all sorts of detail: it's
something implemented by the system's firmware, and it follows a set of
conventions that the operating system is expected to be aware of when it
leaves real mode (which means that a driver has to be loaded for it by
the boot loader before protected mode is entered) and so on and so
forth, but it suffices to say that it's really undesirable to use in
many situations.  And heaven help you if a BIOS update comes along that
changes things slightly to fix a bug and the format has to change as a
result.  ("So what, just update the driver," right?  Nah.)

Certainty is key, and certainty comes from software RAID, which puts
control directly in the system administrator's hands and works for any
conceivable mode and layout there is.

The one thing that hardware RAID and BIOS RAID have in common with each
other is that none of them really use a stable or standard on-disk
format.  Even the giant, Adaptec, in its heyday did this.  One format
would be used by one chipset, and the next one would use a completely
different format.  Sometimes they'd even use the same marketing name for
the hardware, and even the same model number with something seemingly
insignificant differing, like a plus symbol or revision number.  This
isn't at all surprising if you've been in the industry for any length of
time, because it is a very common thing across all types of hardware.

The point is that even with contracts in place, you cannot (without
having a cold spare) be certain that you're going to be able to keep it
up and running---and I mean 100% certain, not "should be" or "ought to
be" certain, because I don't make bets on that kind of thing unless I am
forced to.

Software RAID is something that you can control completely.  It's
something that you can reproduce, always.  Add to that the fact that
dedicated RAID controllers are nowadays typically less expedient than
CPUs for processing, and the fact that software RAID is typically a
safer bet, the fact that anyone would advocate something different
astounds me.

>>> Given the choice between software RAID and BIOS
>>> RAID, software RAID is the only real choice.  However, a real RAID
>>> card will almost always be the best option, if you have one available.
>>
>> This is patently false, unless you have knowledge of resources that I do
>> not.
>>
>> Hardware RAID is great---until you no longer have the ability to replace
>> the controller with an exactly identical one.  And then you have to rely
>> on luck.
> 
> Yes, I do have such knowledge.  It's called using a supported,
> enterprise-level RAID card with a warranty.

Or you can make the (just as valid, and less expensive) choice to use a
supported, open, free, production, enterprise-level operating system
kernel that happens to perform *really* well for these sorts of things,
and is capable of being understood by anyone who has the desire to do
so.  Better than a warranty, it has constant support across the world
and bug fixes typically have a turnaround of less than a hardware SLA
can afford (and in a pinch such issues can be fixed by oneself with a
band-aid until a fix is permanently available, too).

So, instead of making *you* rebuild the array automatically when the
vendor is forced to replaced your discontinued equipment, *they* do it.
 Doesn't really matter; the result is effectively the same.  I fail to
see how this is in any way responsible.

For that matter, it's possible these days to build relatively
inexpensive little boxes running Linux that encapsulate a Linux
kernel-based software RAID, and you get the benefits of both.  Or if you
can't even afford that, you can get a SATA port multiplier box and put
multiple HDDs in there and use software RAID on that, if your SATA
chipset supports port multipliers robustly.

> Any server admin worth
> their salt is using equipment from a large manufacturer and not
> cobbling something together from mail-order parts. 

In my world, using systems that are standard, well-understood, and
interchangeable is king.  The value is in as little downtime as is
possible.  I don't experience downtime for swapping out bad disks, nor
do I experience downtime for backups, updates, and so forth.  I do
experience an average of five minutes of downtime per month for reboots
on the individual systems that I build, install and manage.

> All OEMs provide
> warranties and have varying levels of turnaround time for replacement.
>  You select the time you need based on the needs of the server,
> usually down to 4 hours.  If you need it faster than 4 hours, you can
> buy a cold spare at the same time you purchase the server.

I'd already recommend cold spare hardware as SOP, but maybe I'm wrong
there.  That's what I do for everything except for things that I can get
within a one hour window.

> If you don't have the budget for that, software RAID works OK too,
> depending on the application.  I would also only use software RAID at
> levels 0 or 1 -- not on anything that requires a parity calculation
> like RAID5.  You probably shouldn't be using RAID5 these days anyway.

Why must you needlessly cut down software RAID?  Where is software RAID
inferior to hardware RAID?  Software RAID started to become preferable
about a decade ago for many different reasons, and it started because
the buses we are using and the CPUs we are using are fast enough to
perform RAID in software, with "fast enough" meaning comparable to
hardware RAID implementations.

There are businesses that build their entire (massive) infrastructure on
it because it is easy to do, less expensive to do, and quite reliable.

I had a link to a nice set of plans for a box that anyone can build to
create a 100+ terabyte hot-swap array that is essentially an appliance.
 I can't seem to find it at the moment, sadly.  But the whole thing was
that it uses Linux software RAID because it is better and cheaper than
hardware RAID.  It scales well, and you can use commodity hard drives
without worry because you have more than enough redundancy to work
around them.

The nice thing is that you can stack them like this, which is something
that you cannot do in grotesque scale with hardware RAID.  I've seen
some setups where software and hardware RAID modes are blended in order
to create something like that which could be done in pure software RAID,
though I never did see the point: it was simply additional and
unnecessary expense.

>>> I haven't used Windows software RAID recently, but I think it will be
>>> difficult to get a RAID10 working since the drivers required for
>>> accessing the striped data are themselves striped across the disks,
>>> rendering them unreadable to the system as it boots.  Windows may use
>>> a separate boot partition that is not striped to get around this
>>> issue, but you will have to research that (and I'm not sure a Linux
>>> user group mailing list is the place to find the best answer).  I'm
>>> sure you could test it out in a VM.
>>
>> I am pretty sure that Windows supports both RAID 0 and RAID 1.  Just
>> stack them in the appropriate order.  But I haven't tried it.  If it
>> doesn't support those things, then what the heck is it worth anyway?
>>
>> At least, that's how I feel about it.
> 
> My only point was that if it was booting from RAID10, that might be a
> problem.  Since this server will be booting from a mirror and only the
> data is RAID10, there's probably not an issue here.

I'm not sure why there would be a problem booting from RAID 10, as long
as its software RAID and you are using a bootloader that understands the
software RAID format that is in use.  Certainly Microsoft's bootloader
ought to be able to do that; GNU's GRUB certainly can.

>>> As for Windows being completely, horribly sucky sucky, please cut it
>>> out.  A very large portion of the world uses Windows for rather large
>>> file storage on a daily basis, and they don't all constantly crash and
>>> burn.  It may not be your preference, so leave it at that.  Linux has
>>> its own share of problems.
>>
>> I administer both Windows and Linux systems for a living.
>>
>> The Windows boxes are the only ones that constantly need my attention.
>> The Linux boxes are more or less fully automated; I log into each of
>> them once per week using an automatic tool and patch the systems.
>>
>> Just for a single one of my clients, in the last week I have had to
>> spend more than 10 hours working on problems that simply wouldn't exist
>> if Windows weren't in use (or if the stupid person that setup the
>> Windows box hadn't made stupid decisions).
>>
>> If you're going to compare Windows Server and, say, Ubuntu Server,
>> you'll find that Ubuntu Server wins pretty much all over the place.  And
>> it isn't even the best (IMHO) option out there.  I'm partial to Debian
>> for servers, myself.  But I honestly don't care what's running on a
>> server as long as it is running well.
> 
> Actually I have done such comparisons, and Windows wins in areas like
> user authentication (Active Directory) and remote configuration
> management (Group Policy).  I have performed audits on both Linux and

You realize that LDAP and Kerberos are the heart of Active Directory, right?

You realize that there are some really quite excellent configuration
management systems for Linux, right?  In fact, tomorrow's meeting is on
Puppet, which is a common choice.  And from that you can do things very
similar to what you can accomplish with Group Policy.

> Windows servers, and Windows provides a unified way to access all
> configuration data (WMI), while Linux uses a giant pile of
> non-standard text files.  If you think that's easy to grep through
> across multiple distro versions and different software packages, you
> are sorely mistaken.

Really?  I just learn the individual software components and work with
them directly, so I guess it doesn't bother me any.  ISC DHCPd is ISC
DHCPd is ISC DHCPd, regardless if it is running on Ubuntu, Debian,
Gentoo, or something else altogether.  Same goes for BIND or Apache or
what-have-you.  In fact, if you use a halfway decent CMS, I hear it'll
do all that for you.  I don't use one yet, but I do have a set of
scripts and conventions that I use to perform verification and auditing,
and for most things they go to syslog.

What I was originally talking about is a fundamental difference between
the Windows Event Log and syslog (which is what the rest of the world
uses).  Syslog writes the actual message text, whereas Windows Event Log
does not.  In order to decode an event log that is copied off of a
computer system, you must have all the required DLLs to read the
messages.  Otherwise you get meaningless messages like "The description
for the event could not be found" or similar.

It used to be that you could install the NT POSIX subsystem (which has
been known by various names over the years) and get a "syslogd" that
would be able to forward events from the Event Log to a server listening
for syslog messages.  But that's not an option on, for example, Windows
7 unless you have Windows 7 Ultimate.  I've not had a chance to test
other solutions with Windows 7 yet, either (though it's on my TODO list
for the next week or so, actually).

> Linux does of course excel in many other areas, but it's certainly not
> an across-the-board win.

On servers?  It kicks ass.

Personally, I think it kicks ass on desktops, too, but a lot of people
disagree with me there.  That said the biggest argument I actually hear
for Windows workstations is software applications.  That is eventually
going to become a non-issue and I expect that when it does (and no, I'm
not sure when it will be; I certainly won't claim that this year, next
year, or the year after that will be the "year of Linux on the desktop",
because I don't think it'll happen quite like that) that things will
change.  People will eventually start to gravitate to systems that
require less and less maintenance, for example.

>> Also, my experiences have typically been that given a set of
>> requirements and a set of hardware, the solution can almost always be
>> more efficiently implemented with Linux on the server.  With the very
>> real security and costs problems that Windows presents, it (again, IMHO)
>> has no place on a server.  Just because "a very large portion of the
>> world uses Windows for rather large file storage on a daily basis"
>> doesn't mean squat, other than perhaps that there are a lot of very
>> stupid people out there placing a lot of trust in a very historically
>> weak and insecure system.  (And every month, that history simply grows
>> longer.)
>>
>> I'll have rational conversation about Windows all day long, but it
>> sounds to me like you're either a fanatic or a fan boi, and if that's
>> the case, then we haven't anything to discuss.
> 
> If you are unable to effectively administer a Windows server, that
> only speaks to your own inability to do so.  I'm not saying it's the
> greatest thing ever, but it works well enough for many applications.

I'm not sure what you mean here.  I administer Windows servers just
fine.  I don't *like* it, but I can do it.  The only real pain in the
ass is that most people (caveat: that I have personally met) who call
themselves Windows administrators don't spend any time actually learning
how their system works.  I prefer to learn the actual how and why, and
that way I can remember less information and be able to do better things
with it.

My current source of overwhelming joy is a server that some moron
deployed.  He thought it was a good idea to set up SQL server, Microsoft
Exchange, SharePoint, and file sharing all on the same drive on the same
server.  It is maxed out at 4 GB of RAM, and when I took it over it was
constantly digging 2 GB into swap.  Today it uses little to no swap
(yay!) and I've fixed 90% of the problems that it had when I took it
over.  The rest are simply waiting for authorization.

I'll never make the claim that I am a Windows expert: I am not.  I
frequently have to do additional research to figure things out.  But
I'll take nicely organized and compartmentalized text files on a
filesystem than a huge bloody, binary KVS that everything relies on any
day of the week.  It is a lot easier to compare text files, back them
up, make copies of them, and manage them.

Call me nuts, I guess.

> Calling people stupid or a "fan boy" just because they disagree with
> your opinion is a clear sign of a lazy-minded fool.  I guess you only
> agree with this post from your own blog when it suits you:
>     I am absolutely intolerant of ... people with closed minds

I said "it sounds to me like".  I said that because you're here pushing
something that makes little to no sense to push given that we have
better, less expensive and easier to support solutions in today's world.
 Additionally, it's a proved fact that most UNIX-like operating system
kernels are more efficient with handling high server loads (Windows
servers ("services"), much like UNIX servers ("daemons") can be
multithreaded or multiprocess, and on most UNIX-like systems, threads
and processes have very similar (and favorable) performance
characteristics.  This was written about by Microsoft in a paper on
their Singularity operating system, where they compared Windows against
Linux and FreeBSD.

> I'm not here to start an OS war, as you seem to be, but I seem to be
> the only one having an objective discussion.  And you seem to
> emphatically *avoiding* any type if "rational conversation" about it.

I'm not looking for an OS war.  I did suggest, and still suggest, that
the ideal solution (which is sadly unattainable) for the OP would be to
use a small Linux box configured as an appliance that would handle the
storage for the Windows box.  This makes life a lot easier in the
long-run, because the Windows box doesn't have to worry itself about the
storage, it can simply format the whole thing as NTFS and go about its life.

>>> Finally, why do they include BIOS RAID on systems?  Mainly to have a
>>> feature to list on the package.  Incidentally, I don't think I would
>>> buy a board for enterprise usage that has such a feature.  Those are
>>> typically aimed at the enthusiast market.
>>
>> It's safe to just ignore it.
>>
>>> P.S. Please pay attention to whether the replies you receive are top
>>> or bottom posted and use the same method to continue the conversation.
>>>  I'm not one to care, as long as you are consistent within the same
>>> thread.
>>
>> You should take your own advice:
>>    http://news.gmane.org/gmane.org.user-groups.ale
> 
> This thread was already messed up with regards to quoting style.  I
> was specifically talking to Greg, as he started top posting after you
> had already bottom-replied.  Incidentally that link says nothing about
> anything, given that I have only posted 4 times to this list and each
> time tried to remain consistent with whatever method that thread was
> already using.

Whilst doing exactly the same thing in your message?  Sounds
hypocritical to me.

> 
>> Idiot.
>>        --- Mike
> 
> 
> I had no idea I was dealing with such an uncivilized ass.  I have a
> bunch of other words for you too, but I actually have the ability to
> contain myself.

Hypocracy *is* idiocy, IMHO.  I think that your words would have held
merit if you were following them yourself.  I wasn't aware that my
calling you out on that qualifies me as an "uncivilized ass," but if
that's the case, then I will absolutely wear that label.

I will also say that I had thought that this, like the rest of most of
your message, was directed at me.  Therefore, I perceived your words
(and hypocracy) as some form of snark or sarcasm directed at me.  I'll
apologize, though, as I appear to have misinterpreted.

> I'm sorry it bothers you when someone differs with your opinion, and
> the fact is that's all you have: your opinion.  Your anecdotes of
> whatever little servers you've run are irrelevant to the fact that
> many enterprises large and small successfully use both Linux and
> Windows on a day to day basis.  If they did not work, no one would use
> them, simple as that.

That is an illogical and thus flawed argument.  Ink jet printers work
far less frequently and reliably than laser printers, yet they are still
the dominant technology.  They are far more expensive to operate, too,
and yet still dominant.  There are plenty of examples of things that
*don't* work well if at all, all over the place in the world.  Yet
people use them because those are the things that they know.  We could
start with the locks on your front door and work our way to virtually
anywhere, seriously.

Now, if you have Active Directory as a hard requirement, then a Windows
server is (sadly) one's only option.  But there are alternatives that
can be used even with Windows clients that afford the same amount of
control that you advocate with Windows on the server.  And in some
(currently indeterminate) amount of time we'll have native AD support in
Samba, including GP (and believe me, I am looking forward to that,
because it'll make my life easier).

But it can be demonstrated that for almost any set of requirements,
ESPECIALLY server requirements, that virtually any production-quality
free software operating system can do the job just as well, if not
better.  Shoot, we can even run ASP.NET applications on Linux servers
these days, as long as the applications aren't stupid enough to go
trying to access "C:\Windows" or something like that.

To bring this all the way back to where this whole thing started, you said:

>>> As for Windows being completely, horribly sucky sucky, please cut it
>>> out.

Apparently in response to my statement:

>> Windows server sucks, even with only 10 users on it.  The way it
>> operates sucks, the way it treats things on the disks sucks,
>> the overall speed of data access sucks.

I happen to have compared data access times on exact same hardware using
a Linux system (on a live CD) and Windows Server's times.  They really
do stink, even without on-access AV running on the Windows server they
are about twice as slow.  I can't swear that it's the case on all
servers, because I've only had the opportunity to make the comparison on
two servers, but the numbers that I got from both were pretty darn
close: Linux can read and write about twice as fast on the systems that
I have physically had access to.

For that matter, it seems that most of the things that Windows does as
services expect to be on their own systems, more-or-less.  I can run a
network of ten users on a single Linux server running Samba, CUPS, SSH,
Apache, Dovecot, Postfix, SpamAssassin, ClamAV and PostgreSQL without an
issue and in under a GB of RAM.  But it seems that Windows has a problem
doing half of that in 4 GB of RAM.  As an example, the AV program
consumes nearly 1 GB of RAM (and nearly 2 GB of virtual memory overall!)
on Windows.  SQL server likes to use as much memory as it can unless you
configure it to do otherwise, and Exchange seemst o like to use an
utterly astounding amount of RAM as well (contrasted with
Postfix+Dovecot in a mail environment that handles twice as many users
and nearly four times as many messages, and uses almost no memory at all).

And I could go on for days and days about how its overall operations
suck.  There are dozens of points in a Windows system which permit a DLL
to be registered in order to insert itself into the address space of a
certain process, or multiple processes, or even all processes run by the
system.  There is some limited functionality like this on Linux (using
its dynamic linker and some environment variables) but nothing as
massive or layered or complex or easy to compromise as that which ships
in Windows.

They are, however, improving.  They are reducing the requirement for
heavy GUI-oriented applications on their servers, and they now have this
Server Core product that I presume uses PowerShell so that you can
access all aspects of the system.  Though that's nothing new to the
industry (save for the way that their PowerShell utility works in
passing .NET objects instead of text strings around, it's a shift toward
the UNIX model.)  I look forward to the day that 100% of the system can
be managed in that way such that I can do the same sorts of things that
I do today to push out configuration changes on it.  (Heck, it'd be nice
if Windows servers would stop pushing the "remote desktop for
maintenance" model and instead adopt an SSH service so that you can
simply pull up a PowerShell on the server after a successful
authentication, and not have to worry about all that bandwidth waste).

But hey, I'm just a nobody, what do I know.

	--- Mike

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 729 bytes
Desc: OpenPGP digital signature
Url : http://mail.ale.org/pipermail/ale/attachments/20111116/b202f060/attachment-0001.bin