[ale] Compromised System

[attriel]

> We have a system at work that has been compromised.  It looks like
> they got in and used several different executable files, I've got the
> command history however I don't think it is complete.  For example I
> see that directories were created, but I never saw that they were
> removed and I can't find them.  It looks like about 5 ftp sites were
> hit and there was about 3 wget commands to pull down files.  Also
> apache was downloaded and installed, even though it was already
> running on the system.  So here's my question, I know that rebuilding
> the system is the only way to be sure that there is nothing else
> hidden on it, but that's not an option at this point.  Are there any
> good HowTo's or books out there that can give me some direction on how
> to check they system for irregularities?  This is the first time I've
> dealt with this so I would like to learn as much as I can about it,
> I've already determined how they got in.  A user made their password
> the same as their login name, which obviously is no longer allowed.
> BTW the system is running Red Hat 7.3.

(1) You REALLY want to do the scorched earth route.  It's the only way to
be sure

(2) If you can get static compiles of ls, ps and chkrootkit (or however
that tool is spelled), built on a SEPERATE machine, you can try to look. 
BUT!  That won't garauntee, it'll just help you find things.

Common rootkits, last I looked, put in hacked versions of ls (to not show
their secret dirs), ps (to not show their secret listeners), netstat (to
not show their open ports), iptables (to not tell you it's open), etc. 
more, less are modified to show archived versions of files rather than the
new (hacked) versions, etc, etc, etc.

Some of the newer ones/active attackers put in silent kernel modules
(which won't show up on the hacked ls, and won't show up on the hacked
lsmod, depmod, or rmmod).  No way to be sure about the kernel mod, really.
 My vital servers have started running without loadable modules, now, to
tighten that up.

Your user should not be allowed back on the server.  You WANT to rebuild
it, or you ARE still compromised.

Run crack (or whatever the latest password checkers are), see if anyone
else has bad passwords.  Run them regularly, to see if someone MAKES one.

If you got hit by a scriptkiddie, you might be able to recover and be OK
until you can do a rebuild next week.  If you got hit by a real attack,
and they're competent, you're unlikely to find all the bits.

--attriel