[ale] RHEL 5 will not allow login from Console

[Bob Toxen]

On Thu, Sep 01, 2011 at 04:06:04PM -0400, Michael H. Warfield wrote:
> On Thu, 2011-09-01 at 15:34 -0400, John Temple wrote: 
> > I have a RHEL 5 VM system that will not allow us to login from the console.
> > We have tried to use both a valid user and root, for both of them after
> > entering the username "Invalid Username" (or something like that) flashes
> > and then we are returned to the login prompt. We have also tried booting
> > into single user mode by editing the grub command line. No dice there
> > either. Any suggestions on how to get the system back up?

> You say it did NOT prompt you for a password and failed immediately?
> That sounds like a corrupted binary or something serious pretty deep in
> the system.  Are you able to get in from other locations or are you just
> flat out locked out?
Check /bin/login for corruption or bad permissins (755 owned by root is
normal) and /sbin/mingetty.  Also, check /etc/securetty.

> > A couple of things that we have noticed:
> > 1. When the VM boots the system displays a couple of failures most noteably
> > iptables and xinetd.

> Ewww...

> > 2. A few weeks ago a co-worker said that he had trouble with the system
> > saying that it was in read only mode.

> That is generally indicative of file system corruption.
Yup, it sounds like your system is seriously screwed up, clearly with
some file system damage that could explain the lack of being able to
log in.
You could
compare to backup with "tar -d" to diff against backup or reinstall.
I assume you only can log in via ssh, which doesn't use /sbin/mingetty or
/bin/login.

> You say it's a VM?  I take it, it must be one of the paravirtualized
> VM's?  VMware, VirtualBox, XEN, or KVM?

> What I would suggest is laying hands on a good run-live forensic CD,
> like the Network Secuirty Toolkit, NST, here:

> http://www.networkseckuritytoolkit.org

> They just came out with one based on Fedora 15.  The previous one was
> based on Fedora 13 and is what I've been using the most.

> Boot your VM from the CD Image.  I think both VMware and VirtualBox
> default to the hard drive, rather than the CD and you'll have to
> interrup the BIOS and select the boot device.

> Get it up and running and then try running an fsck on the partitions
> that it sees on the hard drive.  NST does start up LVM and you can fsck
> LVM partitions too.

> If you have no errors, mount the partitions over a mount point in the
> correct relative hierarchy (tedious, I know).  You can then chroot into
> that mount point and you'll see your machine as if you had logged into
> it (just that nothing is running) and you can poke around and check logs
> and even manually start up run-time services and see how they behave.
> You can run an rpm -V and do some verifying in there as well and see if
> it finds anything to piss'n'moan about.

> > -- 
> > John Temple
> > cjtemple at gmail.com

> Regards,
> Mike
> -- 
> Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
>    /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
>    NIC whois: MHW9          | An optimist believes we live in the best of all
>  PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!

Bob Toxen
bob at verysecurelinux.com               [Please use for email to me]
http://www.verysecurelinux.com        [Network&Linux security consulting]
http://www.realworldlinuxsecurity.com [My book:"Real World Linux Security 2/e"]
Quality Linux & UNIX security and SysAdmin & software consulting since 1990.
Quality spam and virus filters.

"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond where
the shadows lie...and the Eye is everwatching"
-- The Silicon Valley Tarot Henrique Holschuh with ... by Bob