[ale] somewhat OT: sysadmin must-knows?

[Jeff Hubbs]

On Fri, 2004-11-12 at 00:13, jay wrote:
> For everyone who doesn't know me, name is Jay Loden, I'm a student and 
> assistant systems administrator at Elon University (NC), and I need your help 
> and advice.

Well, well - a new recruit! :)

> I just kind of "fell into" my job with systems administration, much the way  I 
> got involved with Linux, and it turns out I like both.  I am graduating in 
> May and I want to work in systems administration (or indeed anything where I 
> can use Linux daily).  Here's the problem: I'm graduating with a degree in 
> Corporate Communications.
> 
> Basically, I need to teach myself every scrap of useful information I can cram 
> into my brain before May about being a sysadmin, because I'm sure haven't 
> gotten it from my classes!  What I'm looking for is helpful suggestions, i.e. 
> books I need to read, skills I need to have, experience that's critical, etc.

I'd say that the most germane experience you could get would be to set
up server-type computing resources in pursuit of a real-world goal or
reasonable facsimile thereof.  It can be a file, Web, print, database,
mail, or fax server.  Take that process from beginning to end and you'll
see that there is much to be learned in hardware selection and config,
OS installation, RAID, software version control, backup/restore,
performance maximization, and all sorts of stuff.  

> I am working on honing my Python scripting skills (with the hope of being a 
> competent Python programmer by May), 

I like this.  I've programmed since 1977 but haven't done it in a
dozen-odd years until just recently and I'm doing it in Python.

> and I am rapidly learning as much as 
> possible about Linux in general, but I know that there are bound to be things 
> I'm neglecting or unaware of that are essential sysadmin know-how.  I would 
> love to hear from systems administrators out there on the list...how did you 
> get into your field?  

My first job was as a civilian working for the Air Force, and I did
software engineering for automatic test equipment.  I found myself
gravitating more toward working with the VAXcluster that the development
and simulation work was done on and the two guys who ran it.  They both
had a bit of the BOFH (Google that) about them but what really got to me
was that they were usually having a lot of fun.  For one thing, they
didn't like to be bothered and they didn't want anything interfering
with their "recreational" computing activities, so they put an awful lot
of effort into automating every admin task that could be automated and
setting everything up so as to enable them to actually have to do work
as infrequently as possible, and so at any given moment they were as 
likely to be hitting a BBS somewhere or piddling with spare disk drives
(which were bigger than a breadbox back in those days) as anything
else.  This may sound like the antithesis of professionalism, and it
certainly had that appearance, but the fact of the matter was, how they
managed that cluster and the network surrounding it affected the working
lives of a few hundred people and the throughput of a significant
software production operation.

But they would also do stuff like this:  There was a new guy named Doyle
who was geeky.  Real geeky.  He had the first Amiga I ever saw.  Anyway,
John and Jack promised Doyle a steak dinner if he would figure out a way
to change a process' process name on the fly, i.e., while the process
was still running (ordinarily in VMS a process name is set at its
initiation and can be changed only by that process after that).  They
gave him a pile of really, really arcane manuals (assembler, VMS
internals, etc.) and they even let him come in on a weekend so he could
work on it using a privileged account and it wouldn't harm anything if
he crashed a VAX (and he did, several times).  The "fun" aspect of this
was that once Doyle had his program working, John and Jack could sit
there and change people's process names so that when anyone did a SHOW
USERS command, they could tell little stories or otherwise mess with
people in the process name column.  Some Alabama fan might have "Go
Tide!" as a process name and John, an Auburn grad, would change it to
"No Tide!" (i don't remember if that actually happened but that's the
sort of sort of thing they'd do).

I saw what happens if you don't take care of the air conditioning in a
data center or, more correctly and to the point, put its care into the
hands of people who don't report to you.  These guys could tell by
"feel" if someone was doing something crazy that slowed everybody else
down and they could figure out what it was without the perp's assistance
or knowledge. 

> What do you think I need to know? 

See above.

> How can I learn it 
> best?  

See above.

> Is getting involved with OSS projects helpful (e.g. helping with a 
> distro)? 

Yes, but probably not in the sense you're thinking.  Back in da day, if
your operation depended on some software, whether it was something
relatively arcane like an off-the-shelf warehouse management app or an
operating system like VMS, and you found a problem with it or needed
help with it, you could ring up the company you paid good money to and
talk to a person who knew more about the product than you did, and if
your findings led toward IDing a bug or suggesting a documentation
change, that was pretty much what would happen.  I once caught a pretty
serious security flaw in VMS (when people would enter the wrong
password, VMS would log the incident including what they entered, which
was not good because if you saw in the log that someone had typed
"GO+BRABES" as a bad password, you could be pretty sure that the real
password was "GO+BRAVES" - under VMS, no one, not even an admin, was
supposed to be able to determine a user's password) but it turned out
that I wasn't the first person to find it and it was already being
corrected.  

Thanks in large part to Microsoft, all this has changed.  If you were
calling DEC tech support in the mid/late 1980s, they assumed that you
were an IT professional who had been working in the field for quite some
time and had a technical bachelor's degree at the very least.  Now, when
you call Dell about your Win2K server (who, by the way, is NOT the OS
vendor), Dell assumes you were selling magazines door-to-door two years
before and still have wet ink on your vo-tech diploma, and you're
talking to someone with about the same amount of experience, if that.  

To me, working with Linux and OSS brings back the *good* aspects of the
Good Old Days.  It restores the meritocracy and its associated value
system.  My point - and I do have one - is that if there is something
about a distro or an app that you don't like, you can do something about
it up to your maximum capability to do so.  If you are dependent on
xsane to run your scanning operation and you need xsane to behave
differently, you can either a) make the changes yourself - it is, after
all, Open Source or b) get on the developers' mailing list, make a
cogent argument why it should be so, rally people to your cause, and
maybe shoot them some cash.  So, from that standpoint, it might be a
good thing to keep a finger, even if only a passive one, in the OSS
projects that seem the most key to your operation.

> What options are there for a guy with only a small amount of 
> "official" sysadmin experience who's willing to learn fast and hard? 

Become the flunky of someone a lot better than you!

> 
> Thanks in advance and I apologize for the lengthy mail! 
> 
> -Jay
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://www.ale.org/mailman/listinfo/ale