As part of troubleshooting bug 598631 we got a workaround into util-linux-ng that makes the symptoms of that bug go away.  We should make sure we revert that patch once the root cause is found, since the patch could potentially have more far reaching implications than immediately apparent.

Relevant context below:

+++ This bug was initially created as a clone of Bug #598631 +++

Description of problem:

None of shutdown, reboot, halt and Ctrl-Alt-Delete actually achieve anything useful.

If I do any of them, I see a broadcast message:

[root@andromeda linux-2.6.32.i686]# shutdown now

Broadcast message from root.org.uk
        (/dev/pts/0) at 19:24 ...

The system is going down for maintenance NOW!

The terminals on /dev/tty# and /dev/ttyS# are killed, but that's all.  Nothing else happens.  My ssh login is still there, and I can still do things.

Looking in ps, I see:

 1591 ?        Ss     0:00 /usr/sbin/atd
 1710 ?        Ss     0:00 /bin/sh -e /dev/fd/10
 1714 ?        S      0:00  \_ initctl emit splash-request IMMEDIATE=1 MODE=shutdown MESSAGE=Shutting down...
 1715 ?        Ss     0:00 /bin/sh -e /dev/fd/11
 1716 ?        S      0:00  \_ /sbin/plymouthd --mode=shutdown
 1717 ?        S      0:00      \_ /sbin/plymouthd --mode=shutdown


Version-Release number of selected component (if applicable):

upstart-0.6.5-5.el6.i686

How reproducible:

100%

Steps to Reproduce:
1. Do any of shutdown, reboot, halt or C-A-D.
  
Expected results:

The machine should shut down.

Additional info:

May need to be running an X session.


--- Additional comment from chrisw on 2010-06-18 19:02:35 EDT ---

I have the same problem with plymouth-0.8.3-5.el6.x86_64

It simply hangs here:

# ps -ef | grep plymouth
root     25120 25119  0 14:55 ?        00:00:00 /sbin/plymouthd --mode=shutdown
root     25121 25120  0 14:55 ?        00:00:00 /sbin/plymouthd --mode=shutdown
# strace -p 25121
Process 25121 attached - interrupt to quit
open("/dev/ttyS0", O_RDWR|O_APPEND

killing that child plymouthd allows reboot to complete.


--- Additional comment from rstrode on 2010-07-09 18:30:42 EDT ---

Thanks Marian.

I've looked into this a bit today, but still need to investigate more.

It looks like the CLOCAL flag on the serial console tty is somehow getting cleared.

If I run

stty clocal --file=/dev/ttyS0

then shutdown works fine.

I've also added a

stty -a --file=/dev/ttyS0

call to /etc/rc.local and indeed clocal is properly set at that point (right before agetty is run). 

Adding -L to /etc/init/serial.conf also makes reboot work properly.  This means that CLOCAL is getting cleared some time after rc.local runs and some time before agetty calls termio_init().

it could upstart, plymouth, the kernel, or something else clearing CLOCAL. 

I've got to run now, but I'll investigate more on monday.

--- Additional comment from rstrode on 2010-07-09 23:42:30 EDT ---

So it looks like this is agetty itself dropping CLOCAL.

In termio_init, it does:

tp->c_cflag = CS8 | HUPCL | CREAD;

which wipes out all existing control flags from the tty (including CLOCAL).  The only reason it doesn't hang itself when trying to open the tty is that it uses O_NONBLOCK.  I think we need to do one of:

1) Fix the assignment above to:

 tp->c_cflag |= CS8 | HUPCL | CREAD;

2) Fix the assignment above to:

 tp->c_cflag = CS8 | HUPCL | CREAD | (tp->c_cflag & CLOCAL);

3) change initscripts to pass -L to agetty when the tty is initialized with CLOCAL by the kernel

4) change plymouth to open it's tty with O_NONBLOCK

I'm not sure what the "right" answer is out of these 4 (or if some 5th possibility is the right answer), but since 2 of the 4 involve changing agetty, i'm moving this to util-linux-ng

--- Additional comment from rstrode on 2010-07-12 13:33:41 EDT ---

So, I've been reading up a bit more about serial protocol, and I think the CLOCAL "fix" may just be a workaround.  I'd appreciate if someone who knows more about this stuff would chime in, though.  Here's an explanation on how I think it works:

As I understand it, CLOCAL means "don't block waiting for carrier detect and dataset ready signals".  These are two different bits that deal with hardware flow control. I think they mean something like this:

- When carrier detect is asserted it means there is a connection between the remote serial console and the running machine.
- When dataset ready is asserted it means the serial console is reporting that its ready to receive data from the running machine.

stty -a --file=/dev/ttyS0

on an affected machine outputs:

speed 115200 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>;
eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread -clocal -crtscts cdtrdsr
-ignbrk brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl -ixon -ixoff
-iuclc -ixany imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke

The "cdtrdsr" there suggests the tty is configured for hardware flow control.  Despite this, I'm guessing we're not getting a "dataset ready" signal from the remote serial console which is what's causing the open() call to block.  When CLOCAL is set, the kernel doesn't bother waiting for the "dataset ready" signal, so no blocking will happen.  Note, I don't believe the remote end will assert "dataset ready" until the kernel asserts "data terminal ready" (a different control line) in response to an open() call from the application.  So the problem could be the kernel isn't ever asserting "data terminal ready" and so the remote end isn't ever asserting "dataset ready" in response. Or it could be something else.

Anyway, this stuff is a bit outside my knowledge domain, so I'd appreciate feedback from someone more knowledgeable here.

Also, I just noticed Karel committed a patch that does suggestion 2) from comment 25 in util-linux-ng-2.17.2-5.el6.  I'll test that and report its results.

--- Additional comment from rstrode on 2010-07-12 13:52:52 EDT ---

Just to follow up, I can confirm util-linux-ng-2.17.2-5.el6 makes the problem go away.

--- Additional comment from rstrode on 2010-07-12 14:23:29 EDT ---

Created an attachment (id=431237)
dmesg output after echo t > /proc/sysrq-trigger while cat /usr/share/pango*/HELLO.txt > /dev/ttyS0 is blocked

I've talked to aris about this issue, and he believe this is a kernel bug.  He's asked for the above output.

--- Additional comment from kzak on 2010-07-12 14:50:04 EDT ---

Note for QA & PM: it seems that the change in agetty is a workaround for kernel bug #613756. I'm going to keep the patch in util-linux-ng until we found a better solution.