[ale] RHEL 5 will not allow login from Console

[Michael Trausch]

On 09/01/2011 03:34 PM, John Temple wrote:
> I have a RHEL 5 VM system that will not allow us to login from the
> console. We have tried to use both a valid user and root, for both of
> them after entering the username "Invalid Username" (or something like
> that) flashes and then we are returned to the login prompt. We have also
> tried booting into single user mode by editing the grub command line. No
> dice there either. Any suggestions on how to get the system back up?

I had to come to the computer to reply to this...

Assuming that RHEL uses the same underlying base as most other
distributions, the login process on the console should be:

 1.  Some form of getty (this is what issues the "login: " prompt),
     your system may use agetty or another implementation.  Check its
     man page for how it works.
 2.  You enter a username.  The getty implementation should then spawn
     /bin/login $LOGIN (which is what prompts for the password).  If
     the correct password is specified, then login will setup the user
     environment and spawn the user's shell. (see "man login")
 3.  If the initially provided password is incorrect (or if the login
     is invalid), /bin/login will then ITSELF prompt for a password.
     It will do this usually 3 to 6 times (my system is configured for
     3, and the limit is configured in /etc/login.defs).

Allow me to demonstrate; the first session is me logging in as a user
that does _not_ exist on my workstation; the second session is me
logging in as a user that _does_ exist on my workstation:

================= Logging in as invalid user =========================
mbt at aloe ~ % sudo -i
aloe ~ # exec /bin/login afas
Password:

Login incorrect
aloe login: afas
Password:

Login incorrect
aloe login: afas
Password:
Maximum number of tries exceeded (3)
mbt at aloe ~ %
================= Logging in as invalid user =========================

Note that it returned to my normal user's shell prompt because the "exec
/bin/login" line replaced the root shell that I spawned via sudo with
the /bin/login process; once /bin/login terminated, the sudo session was
over and returned back to the parent shell which spawned sudo.

Here is a session with a valid user:

================== Logging in as valid user ==========================
mbt at aloe ~ % sudo -i
aloe ~ # exec /bin/login mbt
Password:
Last login: Fri Sep  2 22:23:12 EDT 2011 on pts/4
mbt at aloe ~ % exit
mbt at aloe ~ %
================== Logging in as valid user ==========================

Here, I entered the correct password and login behaved as it should
have, logging me in.  Here is a third session, valid user, bad passwords:

================== Logging in w/ bad passwd ==========================
mbt at aloe ~ % sudo -i
Password:
aloe ~ # exec /bin/login mbt
Password:

Login incorrect
aloe login: mbt
Password:

Login incorrect
aloe login: mbt
Password:
Maximum number of tries exceeded (3)
================== Logging in w/ bad passwd ==========================

So, you can see here, the correct message (and the one that I have seen
on virtually every system I've ever mistyped my username, password, or
both on at the console practically since I started using Linux-based
systems over 15 years ago) is "Login incorrect".

Something is not right.  I would not be so quick to blame something
innocuous for your problems, either; see this:

mbt at aloe ~ % sudo strings /bin/login|grep -i invalid
Invalid root directory '%s'
invalid root `%s' for user `%s'
mbt at aloe ~ % sudo strings /bin/login|grep -i user
ruserok
((void *)0) != ptr_pam_user
((void *)0) == username
cannot find user %s
((void *)0) != username
get_pam_user
unable to change owner or mode of tty stdin for user `%s': %s
bad user ID `%d' for user `%s': %s
bad group ID `%d' for user `%s': %s
initgroups failed for user `%s': %s
USER
unable to cd to `%s' for user `%s'
invalid root `%s' for user `%s'
no subsystem root `%s' for user `%s'
USERDEL_CMD
USERGROUPS_ENAB
mbt at aloe ~ % sudo strings /bin/login|grep -i ncor
Login incorrect

There is a "cannot find user %s" string in there, but it is only ever
logged to the system logger:

mbt at aloe shadow-4.1.4.3 % find . -name '*.c' -o -name '*.h' | xargs grep
-n 'cannot find user'
./src/login.c:886:		SYSLOG ((LOG_ERR, "cannot find user %s", failent_user));
./src/login.c:1166:				         "cannot find user %s after update of
expired password",

I'd copy and paste the function it was in, src/login.c has only one
function (main) and it is over 1,000 lines long (sheesh!) so I'll skip that.

> A couple of things that we have noticed:
> 1. When the VM boots the system displays a couple of failures most
> noteably iptables and xinetd.

What are the failures, exactly?  Do they come from a script expecting to
load firewall rules, or do they indicate the lack of loaded kernel modules?

> 2. A few weeks ago a co-worker said that he had trouble with the system
> saying that it was in read only mode.

This could be anything.  Bad filesystem, failure to remount read-write
in the boot process (e.g., modified, missing, or [unlikely!] corrupt
init scripts), an explicit remount read-only shortly enough after being
mounted read-write that it is allowed to take effect... anything.

I realize that relative to those who have answered your thread, I have
no credentials.  But I have cleaned up after several a break in, and I'm
telling you, this smells like one to me.  (I would hope that I am
wrong... but it doesn't seem likely.)  It is either that, or RHEL
doesn't use standard /bin/login, and I'm not sure which would frighten
me more.  (Note that I'm not saying I like the coding style of the
ubiquitous one from the shadow package... but I've used it for years and
it's on every box I am aware of.  It's standard.  It's known.
Therefore, it's tested and likely to be good.)

I'd at the very least like to hear back when you figure out what's up
with the box, for no other reason than I am curious.

	--- Mike

-- 
A man who reasons deliberately, manages it better after studying Logic
than he could before, if he is sincere about it and has common sense.
                                   --- Carveth Read, “Logic”