[ale] I didn't post this but I saw it happening

Lightner, Jeff JLightner at dsservices.com
Tue May 20 16:55:42 EDT 2014


Ouch!

One of our admins ran a script once that accidentally shutdown all of the systems it reached before he killed it.   I forget the details but it was trying to parse /etc/passwd on the remote hosts and when it hit the user named “shutdown” it actually ran the command “shutdown” for some reason.   (Looking at his script after the fact it wasn’t clear to me why that happened but I also didn’t decide to test any variations of it.)   Our director kids him to this day about his script writing abilities.  Luckily impromptu reboots while not good aren’t usually as disastrous as wiping all servers.

A great reminder to everyone that backups ARE critical.   Losing some files is not good but losing them all would be catastrophic.   It flabbergasts me how often I have had to argue with higher ups about backups (and restore testing).   It is especially maddening to hear as I often have the “we’ll back it up once because it never changes” idea – Systems always change even if only a little.   No one ever thinks backups are important until they need a restore at which point it is too late to figure out what you SHOULD have been backing up or whether the backup WILL restore.

From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of Jim Kinney
Sent: Tuesday, May 20, 2014 4:23 PM
To: Atlanta Linux Enthusiasts
Subject: Re: [ale] I didn't post this but I saw it happening



On Tue, May 20, 2014 at 4:11 PM, Beddingfield, Allen <allen at ua.edu<mailto:allen at ua.edu>> wrote:
We were all here bright and early on Sunday morning to do our yearly DR failover exercise (failing to our colo site in Atlanta), and some optimist brought up that "it could be worse, we could be at Emory and actually having a disaster right now..."  Yeah, that didn't make me feel any more awake at 7:00 on a Sunday morning..lol
So, how is the recovery from that going over there?

I am happily ignorant of the scope of the mess. All of my stuff runs Linux and Emory UTS is not allowed to even look in the direction of the cabinets!
Payroll systems were wiped out as was everything in the library and most smaller departments that don't have dedicated support staff.
I saw the notice the following Monday and burst out laughing. I was in the building with payroll staff and was informed they got wiped out. My humour vanished rather quickly. At this point all servers are back and all desktops are working but all files from the day of the wipe are likely gone as the re-image happened before/during backups.
I'm glad I don't work with that bunch.

Allen B.

--
Allen Beddingfield
Systems Engineer
The University of Alabama


________________________________
From: ale-bounces at ale.org<mailto:ale-bounces at ale.org> [ale-bounces at ale.org<mailto:ale-bounces at ale.org>] on behalf of Jim Kinney [jim.kinney at gmail.com<mailto:jim.kinney at gmail.com>]
Sent: Tuesday, May 20, 2014 2:59 PM
To: Atlanta Linux Enthusiasts - Yes! We run Linux!
Subject: [ale] I didn't post this but I saw it happening

http://it.slashdot.org/story/14/05/17/051214/emory-university-sccm-server-accidentally-reformats-all-computers-campus-wide

Um. Yeah. Centralized control is great until it's not.

--
--
James P. Kinney III

Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain

http://heretothereideas.blogspot.com/
_______________________________________________
Ale mailing list
Ale at ale.org<mailto:Ale at ale.org>
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo



--
--
James P. Kinney III

Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain

http://heretothereideas.blogspot.com/

Athena®, Created for the Cause™

Making a Difference in the Fight Against Breast Cancer

____________________________________________________

CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20140520/e90165a8/attachment-0001.html>


More information about the Ale mailing list