[ale] Godaddy outage

Wed Sep 12 12:41:51 EDT 2012

On 09/12/2012 11:58 AM, Scott Plante wrote:
> I was at a then Fortune 50 company and their Sys Admin "consultant"
> from CapGemini deleted their running ERP database. Luckily he realized
> what he did and asked for help. I was there and convinced them to just
> hard power off the machine. When it booted, we were able to get the
> database segments out of lost+found, inspect the binary header at the
> top of each file to determine the order, rename the files and bring up
> the database with no data loss or corruption. Poor guy was let go,
> which I thought was too bad because they'd have been screwed if he'd
> kept his mouth shut. A proper shutdown or even database stop and the
> files would have been lost, or at least harder to recover. We were
> back up and running in <20 minutes.

The next time somebody does something dumb they will not be so lucky. No
one will take the blame and they will not know what really happened.

>
> ------------------------------------------------------------------------
> *From: *"Jeff Lightner" <JLightner at water.com>
> *To: *"Atlanta Linux Enthusiasts" <ale at ale.org>
> *Sent: *Wednesday, September 12, 2012 10:29:50 AM
> *Subject: *Re: [ale] Godaddy outage
>
> I was at a Fortune 500 once where the main ERP DB went belly up.  
> That caused us to:
> 1) Declare a disaster and invoke our DR plan which included sending
> folks and tapes to Philadelphia.
> 2)  Spend time on trying to recover the original Production.
> 3)  Build in house systems to try to figure out what caused the issue.
> 4) Jump through dozens of other hoops.
>
> In the end we really couldn't pinpoint what had caused the issue
> though I suspected fat finger somewhere.   The powers that be wouldn't
> accept this and they kept a team on it for over a year.   That team
> eventually did put out an RCA but I always had my doubts.
>
> Later one of the DBAs did a fat finger causing another outage and
> immediately did a mea culpa.  He asked if they wanted him to resign
> and they said no.  We were all just so relieved that he had admitted
> the mistake and saved us all the extra headaches.  (As if recovering a
> major product DB all by itself isn't headache enough - it takes time
> even with good backups and logs.)
>
>
>
>
>
> -----Original Message-----
> From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of JD
> Sent: Wednesday, September 12, 2012 9:56 AM
> To: Atlanta Linux Enthusiasts
> Subject: Re: [ale] Godaddy outage
>
> --
> I saw the JSC mission control primary NFS server for every workstation
> in the building failover during ascent making all flight monitoring
> applications static for 45 seconds. This happened while SRBs were
> firing just after the shuttle was launched. The console cabling from
> the primary server was so tight that when the main engineer pulled the
> console 3 inches closer to get a better view, the cable disconnect
> caused a failover event.
> I didn't cause this, but was sitting a few yards away monitoring a
> different server during critical flight phases.
> --
> Spaces are critical. A month of work was destroyed by a single space
> error.
> * "rm -rf directory*" vs "rm -rf directory *"
> --
> Elsewhere, I have seen a few $2M/hr outages due to corrupted Oracle
> tables after a software vendor told a support guy to type a specific
> command on the running, production, dispatching system.  That took
> about 7 hours to recover and months to convince upper management that
> it was a fluke and not the fault of the guy doing the typing.
> --
> The problems that I've causes were usually due to bad scripting.
> * find+rm is dangerous.
> * rsync can destroy a system - especially if this is a backup just
> prior to an upgrade
> --
>
> I suspect we all have seen some pretty interesting outages over the years.
>
>
> On 09/11/2012 08:19 PM, simontek at gmail.com wrote:
> > Worked at a data center in LA, netzero looped their switch, took out
> > the whole network. I assumed, they were an isp, and knew better. They
> > called to complain, I had the fun of telling them, I took them offline
> > til they fixed their issues. Not fun. Also babysat them, every time
> > they came in to work on stuff. Sent via BlackBerry from T-Mobile
> >
> > -----Original Message----- From: Jeff Hubbs <jhubbslist at att.net> Sender:
> > ale-bounces at ale.org Date: Tue, 11 Sep 2012 20:11:24 To:
> > <stephen.r.blevins at gmail.com>; Atlanta Linux Enthusiasts<ale at ale.org>
> > Reply-To: Atlanta Linux Enthusiasts <ale at ale.org> Subject: Re: [ale]
> > Godaddy outage
> >
> > I once worked at a place where there was a guy who, not meaning to
> > cause trouble, created subdirectory after subdirectory on a Mac until
> > the OS wouldn't function anymore.  We named it the "Copeland Worm"
> in his honor.
> >
> >
> > On 9/11/12 7:59 PM, Stephen R. Blevins wrote:
> >> Early in my IT career (early 1980's), I learned that "No malevolent
> >> cracker, no matter how malicious, can even begin to do the damage an
> >> authorized and well-meaning but incompetent user can do."
> >>
> >> QED
> >>
> >>
> >> Stephen R. Blevins stephen.r.blevins at gmail.com
> >>
> >> On 09/11/2012 02:30 PM, Michael H. Warfield wrote:
> >>> On Tue, 2012-09-11 at 13:53 -0400, Matt Hessel wrote:
> >>>> Well anonymous is claiming they took it down, I don't know if
> >>>> anyone at godaddy broke it. :)
> >>> NO!
> >>>
> >>> First and foremost...  "Anonymous" has not claimed any action.  One
> >>> individual down in Brazil using a handle that has been associated
> >>> with Anonymous has claimed to have done this but stated they were
> >>> acting independently.  The collective has not claimed this and it
> >>> remains unconfirmed.
> >>>
> >>> Second...  GoDaddy itself now claims it was not hackers and not a
> >>> DoS attack but a royal screwup in their routers that resulted in
> >>> corrupted routing tables.  I'm not totally sure how much credibility
> >>> I will lend to that idea but, if true, this is one of the grandest
> >>> screwups since Microsoft dicked up their DNS years and years ago
> >>> with all their public name servers on a single network segment and
> >>> then cut them off from the private master name server with a
> firewall update.
> >>>
> >>> I'm not sure which is worse.  Being hammered by a collective of
> >>> malicious individuals out to get you or displaying a level of
> >>> technical incompetence and inability to follow RFCs and BCPs that
> >>> would put a technotard to shame!  How did they manage to put all
> >>> their (DNS) eggs in one basket so that a single point of failure
> >>> could have such wide spread consequences???  Well, I guess they
> are on good company.  MS has done it.
> >>> AT&T has done it.  Others have done it.  You would think they would
> >>> know better but they obviously do not.
> >>>
> >>> Regards, Mike
> >>>
> >>>> On Sep 11, 2012 1:41 PM, "Scott Plante" <splante at insightsys.com>
> >>>> wrote:
> >>>>> Yes, we use GoDaddy for registration but not DNS nor hosting and
> >>>>> we were unaffected. Our one client who was affected used them for
> >>>>> registration and DNS, but not hosting and they were affected. It
> >>>>> was just name resolution though, you could still access their
> >>>>> externally hosted site by IP of course. I don't know anyone who
> >>>>> was hosting with GoDaddy. You couldn't get to godaddy.com but I
> >>>>> didn't know their IP to try that.
> >>>>>
> >>>>> I imagine someone's in big trouble, if not fired, over that one.
> >>>>>
> >>>>> Scott
> >>>>>
> >>>>> ------------------------------ *From: *"Brian Stanaland"
> >>>>> <brian at stanaland.org> *To: *"Atlanta Linux Enthusiasts"
> >>>>> <ale at ale.org> *Sent: *Monday, September 10, 2012 5:51:13 PM
> *Subject:
> >>>>> *Re: [ale] Godaddy outage
> >>>>>
> >>>>> I know one group with DNS by GoDaddy but hosting elsewhere has
> >>>>> been affected. All machines are still reachable via IP address, of
> >>>>> course. Speaking of which, anyone know if GoDaddy hosted sites can
> >>>>> be reached by IP?
> >>>>>
> >>>>> --Brian
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Sep 10, 2012 at 5:18 PM, Michael H. Warfield
> >>>>> <mhw at wittsend.com>wrote:
> >>>>>
> >>>>>> On Mon, 2012-09-10 at 15:49 -0400, Scott Plante wrote:
> >>>>>>> You guys notice the Godaddy DNS outage? I have a customer' s
> >>>>>>> website
> >>>>>> down.
> >>>>>> http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millio
> >>>>>> ns-of-sites/
> >>>>>>
> >>>>>>
> >>>>>>
> Been following this...  Their DNS servers are impacted.  Hosting servers
> >>>>>> indeterminate.  Claims are made that #Anonymous3 down in Brazil
> >>>>>> is behind this for one reason or another but no one else from
> >>>>>> Anonymous has stepped up to the plate and claimed responsibility.
> >>>>>> Looks to be a loose cannon with a wild hair at this point...
> >>>>>>
> >>>>>> If you are using them as a registrar but are managing your own
> >>>>>> DNS then you do not seem to be impacted at this time.
> >>>>>>
> >>>>>> If you are using their DNS servers then you are probably impacted
> >>>>>> whether you are hosting with them or not.
> >>>>>>
> >>>>>> If you are using their hosting services but managing your own
> >>>>>> DNS, please let us know.  I have no data points on this curve.
> >>>>>>
> >>>>>>> Scott
> >>>>>> Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 |
> >>>>>> mhw at WittsEnd.com /\/\|=mhw=|\/\/          | (678) 463-0932 |
> >>>>>> http://www.wittsend.com/mhw/ NIC whois: MHW9          | An optimist
> >>>>>> believes we live in the best of all PGP Key: 0x674627FF        |
> >>>>>> possible worlds.  A pessimist is sure of it!
> >>>>>>
> >>>>>> _______________________________________________ Ale mailing list
> >>>>>> Ale at ale.org http://mail.ale.org/mailman/listinfo/ale See JOBS,
> >>>>>> ANNOUNCE and SCHOOLS lists at
> >>>>>> http://mail.ale.org/mailman/listinfo
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> -- Government's view of the economy could be summed up in a few
> >>>>> short
> >>>>> phrases: If it moves, tax it. If it keeps moving, regulate it. And
> >>>>> if it stops moving, subsidize it. - *Ronald Reagan (1986)
> >>>>>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
>
>
>
> Athena(r), Created for the Cause(tm)
> Making a Difference in the Fight Against Breast Cancer
>
> ---------------------------------
> CONFIDENTIALITY NOTICE: This e-mail may contain privileged or
> confidential information and is for the sole use of the intended
> recipient(s). If you are not the intended recipient, any disclosure,
> copying, distribution, or use of the contents of this information is
> prohibited and may be unlawful. If you have received this electronic
> transmission in error, please reply immediately to the sender that you
> have received the message in error, and delete it. Thank you.
> ----------------------------------
>
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
>
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo

-- 
Jay Lozier
jslozier at gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.ale.org/pipermail/ale/attachments/20120912/82b35056/attachment-0001.html 