[ale] Little OT: Bad Linux Sysadmin Practices

[Lightner, Jeff]

I worked for a company that had a large data center in Little Rock which had both the telecom systems and the banking/mortgage systems (mainframe and open systems) for multiple telecom, banking and finance companies (at the time 20% of mortgages in the US went through these systems).

On two separate occasions while doing supposedly "planned" maintenance they took down the whole data center accidentally.   In one of these events they had poorly wired one of the breaker cabinets and caused a short which made it fail.  No problem because they had brilliantly set up the power so that failures from one cabinet would shunt the load to the next one.  Oops - there's still a short so now we've fried that next cabinet.  No problem because it fails over to another one etc...  - OH WAIT....!

It is truly gratifying when there is a major production outage that you as a sysadmin could NOT have prevented and do NOT have to be involved in resolving (at least until it comes time to power everything back up).

At that same company a co-worker put together a very good backup policy on our first implementation of NetBackup but when they saw how much it was going to cost for all the tapes required to do the various retention levels management balked.   They nixed his plan and said they'd never need a backup more than 6 months old and "saved money" by not buying as many tapes and adjusting the retention policies.    8 months later when they asked for a backup from the first month due to a critical issue they asked why we didn't have it.  When we told them (and even showed to them in writing) how they had said they would never need more than 6 months did they accept responsibility for poor decision?  They did not - They instead said that the admins were not forceful enough in trying to convince them of the need for the original plan.    Sometimes you just can't win.





-----Original Message-----
From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of Jeff Hubbs
Sent: Thursday, October 11, 2012 7:14 PM
To: simontek at gmail.com; Atlanta Linux Enthusiasts
Subject: Re: [ale] Little OT: Bad Linux Sysadmin Practices

Actually - as I pointed out at the time - there's another reason to not do that.  You don't want to assume that the three conductors going to one power supply are all at the same potential (at the same time) as the three conductors in the other always and forever.  If some yobbo is working on the first distribution point upstream and drops a toolbox onto the connectors, what was once "ground" on one side might momentarily become "hot" - and you really don't want the twain meeting near each other in your servers.  No.  "Redundant power supplies" are for protection against *failure of power supplies*, not failure of mains power; protect against the latter in some other combination of ways.

On 10/11/12 6:28 PM, simontek at gmail.com wrote:
> I usually like the split the load between 2 different circuits, but keeping that if one pdu dies, it can still handle the server load ,so you end up with a ton of empty sockets.
> Sent via BlackBerry from T-Mobile
>
> -----Original Message-----
> From: Jim Kinney <jim.kinney at gmail.com>
> Sender: ale-bounces at ale.org
> Date: Thu, 11 Oct 2012 18:24:16
> To: Atlanta Linux Enthusiasts<ale at ale.org>
> Reply-To: Atlanta Linux Enthusiasts <ale at ale.org>
> Subject: Re: [ale] Little OT: Bad Linux Sysadmin Practices
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
>
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>

_______________________________________________
Ale mailing list
Ale at ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo




Athena(r), Created for the Cause(tm)
Making a Difference in the Fight Against Breast Cancer

---------------------------------
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.
----------------------------------