[ale] [OT] Bad ISP Experience (kinda long rant) - was "Inexpensivebroadband recommendations"

[Jonathan Glass (IBB)]

I installed a firewall (linux based, of course) on my previous employer
network, b/w the 'Net and the network.  Nice machine, although a bit of
overkill for a mere T1 firewall running squid.  This box was, and is,
rock-solid.

After I left, I was called back to help with the migration from one ISP
to another.  The new ISP brought fiber to our telcom closet, and hooked
it to some little netscreen box, and told us we could cut-over on a
certain date.  I showed up that day, and the "Network Engineer" was in
Vegas at a "conference", so they sent one of their $5.15/hour
front-office staff guys to help us cut-over.  

So, I setup the firewall with two sets of scripts, one for ISP A and the
other for ISP B.  We cutover, and I roll to firewall scripts for ISP B. 
I can connect to their little netscreen, and it can't see me.
Wonderful!  What's the ISP's reponse?  Must be the Linux firewall. 
Great!  I show Mr. $5.15/hour that my firewall can ping his box's
internal interface, and explain that there must be something wrong with
the firewall settings on his box, b/c he cannot ping me from his outside
interface.  

He relents, calls his backup, and they walk him through clearing the
firewall settings on the netscreen.  Lo and behold, the netscreen was
not allowing inbound traffic!

Next, we have the boxes talking to each other.  I can ping parts of
their backbone, but noone can find my network.  Their response:  Must be
the linux firewall!  Woohoo, I knew that was coming!  So, after more
troubleshooting, I determine that his backbone routers don't know how to
route to my network segment.  He gives me a blank stare only an intern
can manage, and says he'll call his backup.  The backup guy calls the
"network engineer" in Vegas.  HE says "oops!"  He apparently forgot to
add the route to our network!  He and the backup guy fix this problem.

Now, I've been at this for a good 4-5 hours, and getting really ill. 
While waiting for the phone tag to die down, I automate the cutover and
roll-back procedures with a script so I can leave.  Then we discover the
error of our ways.  They have screwed up the IP address assignments, and
basically the little netscreen goes ape-crazy.  It can't handle the
little 4 IP subnet, and the 16 IP subnet that I need, so they have to
call a tech from 45 miles south to bring us a Cisco 2600 series router.

I'm really ill, I know the firewall is rock solid, my wife is yelling,
my kids are yelling, and I've missed my class for the evening.  Now I
have to wait 90 more minutes for the guy to arrive with the Cisco.  He
gets it configured, we get it plugged in, I run my automated script, and
lo-and-behold, everything works like a charm.  I ask him, did you save
those settings on the Cisco?  He says yes.

One week later, the power goes out for an extended period of time (big
100+ year old oak tree takes on small power pole - no contest), and when
the network equipment comes back online, the two networks no longer
communicate.  What's the first response from the ISP and network admin? 
Must be the Linux firewall! So they call me.  I tell them the firewall
is set to automatically setup and connect to the ISP's network, and that
is not their problem.  I tell them to have the ISP check the Cisco's
configuration.  Yep!  He forgot to save the settings, and the thing had
reverted back to its last setup.

Geesh!  I hope to NEVER have to go through that again!  But, through it
all, I was able to point to the Linux firewall and let them know that IT
wasn't the problem!  Long live the penguin!

Jonathan Glass


_______________________________________________
Ale mailing list
Ale at ale.org
http://www.ale.org/mailman/listinfo/ale