Bug 681167 - Can't log in over serial console
Summary: Can't log in over serial console
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: plymouth
Version: 15
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Ray Strode [halfline]
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker https://fedoraproject...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-03-01 11:12 UTC by Kamil Páral
Modified: 2011-03-08 12:18 UTC (History)
12 users (show)

Fixed In Version: plymouth-0.8.4-0.20110304.1.fc15
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-03-08 02:19:13 UTC


Attachments (Terms of Use)
rpm -qa output (5.46 KB, text/plain)
2011-03-01 11:12 UTC, Kamil Páral
no flags Details
/var/log/boot.log (3.14 KB, text/plain)
2011-03-02 00:07 UTC, James Laska
no flags Details
plymouth-debug.log (85.45 KB, text/plain)
2011-03-02 12:34 UTC, Kamil Páral
no flags Details
plymouth-debug.log (plymouth-0.8.4-0.20110419.1.fc16) (85.52 KB, text/plain)
2011-03-02 14:54 UTC, Kamil Páral
no flags Details

Description Kamil Páral 2011-03-01 11:12:02 UTC
Description of problem:
I installed F15 Alpha RC2 into a VM over serial console. The installation went fine. After reboot it correctly presented login prompt over serial device. But I can write only the username. As soon as I write username and I hit enter, "Password:" prompt appears. *Any* key pressed causes password field to be submitted and "login incorrect" is printed out. I can't simply fill in my password.

Logging in over tty1 in VM works fine.

Version-Release number of selected component (if applicable):
systemd-17-5.fc15.i686

How reproducible:
seems always

Steps to Reproduce:
1. Follow https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_serial_console, install to VM
2. Try to log in over serial console

Comment 1 Kamil Páral 2011-03-01 11:12:28 UTC
Created attachment 481588 [details]
rpm -qa output

Comment 2 Michal Schmidt 2011-03-01 11:52:11 UTC
Looks like bug 663380. Should be fixed in systemd-19. Can you retest with that version? Get it from Koji if you don't see it in updates-testing yet.

Comment 3 James Laska 2011-03-01 12:45:36 UTC
I was randomly seeing this on some kvm guests with serial console enabled.  I couldn't figure out the pattern why some guests exhibit the problem, while others didn't.  I'll be sure to test the updated systemd-19-1.fc15 (https://admin.fedoraproject.org/updates/systemd-19-1.fc15).

Adding CommonBugs so we ensure this gets documented for the Alpha

Comment 4 Kamil Páral 2011-03-01 13:31:46 UTC
No, systemd-19-1.fc15 doesn't fix the problem, still the same problem. Also, if I boot without console=ttyS0, the login credentials are echoed (the password is in cleartext). IIRC there was some bugreport about this already.

Comment 5 Lennart Poettering 2011-03-01 14:20:28 UTC
Kamil, can you paste the output of the following command when this haoppens on systemd 19?

systemctl show plymouth-quit-wait.service

Comment 6 Lennart Poettering 2011-03-01 14:48:48 UTC
Kamil, are you using Plymouth? or are you using rd_NO_PLYMOUTH on the kernel cmdline? Or maybe you removed rhgb or quiet from the cmdline?

Comment 7 Kamil Páral 2011-03-01 15:38:04 UTC
(In reply to comment #5)
> Kamil, can you paste the output of the following command when this haoppens on
> systemd 19?
> 
> systemctl show plymouth-quit-wait.service

"This happens" meaning "login over serial device fails"? This is the output then:

$ systemctl show plymouth-quit-wait.service

Id=plymouth-quit-wait.service
Names=plymouth-quit-wait.service
Requires=basic.target
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target serial-getty@ttyS0.service getty@tty1.service getty@tty2.service getty@tty5.service getty@tty4.service getty@tty3.service getty@tty6.service
After=rc-local.service plymouth-start.service basic.target
Description=Wait for Plymouth Boot Screen to Quit
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/lib/systemd/system/plymouth-quit-wait.service
InactiveExitTimestamp=Tue, 01 Mar 2011 10:31:23 -0500
InactiveEnterTimestamp=Tue, 01 Mar 2011 10:31:23 -0500
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
DefaultControlGroup=name=systemd:/system/plymouth-quit-wait.service
ControlGroup=cpu:/system/plymouth-quit-wait.service name=systemd:/system/plymouth-quit-wait.service
NeedDaemonReload=no
JobTimeoutUSec=0
Type=oneshot
Restart=no
NotifyAccess=none
RestartUSec=100ms
TimeoutUSec=20s
ExecStart={ path=/bin/plymouth ; argv[]=/bin/plymouth --wait ; ignore=yes ; start_time=[n/a] ; stop_time=[Tue, 01 Mar 2011 10:31:23 -0500] ; pid=632 ; code=exited ; status=0 }
UMask=0002
LimitCPU=4294967295
LimitFSIZE=4294967295
LimitDATA=4294967295
LimitSTACK=4294967295
LimitCORE=4294967295
LimitRSS=4294967295
LimitNOFILE=1024
LimitAS=4294967295
LimitNPROC=7921
LimitMEMLOCK=65536
LimitLOCKS=4294967295
LimitSIGPENDING=7921
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=4294967295
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=inherit
StandardError=inherit
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSetDrop=0
MountFlags=1048576
PrivateTmp=no
SameProcessGroup=no
KillMode=control-group
KillSignal=15
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
ExecMainPID=632
ExecMainCode=1
ExecMainStatus=0
MainPID=0
ControlPID=0
SysVStartPriority=-1
FsckPassNo=0


(In reply to comment #6)
> Kamil, are you using Plymouth? or are you using rd_NO_PLYMOUTH on the kernel
> cmdline? Or maybe you removed rhgb or quiet from the cmdline?

plymouth is installed. "rhgb" nor "quiet" nor "rd_NO_PLYMOUTH" are present on the kernel cmdline. When I remove "console=ttyS0" from kernel cmdline, I don't see plymouth graphics, I see standard kernel boot output.

Comment 8 Lennart Poettering 2011-03-01 18:48:03 UTC
Hmm, if you add "rhgb quiet", does the problem go away?

Comment 9 James Laska 2011-03-01 19:01:25 UTC
(In reply to comment #8)
> Hmm, if you add "rhgb quiet", does the problem go away?

In my testing 

Adding "rhgb quiet" to the boot arguments allows me to proceed and login on the serial console (ttyS0).  

# cat /proc/cmdline 
ro root=/dev/mapper/vg_test189-lv_root rd_LVM_LV=vg_test189/lv_root rd_LVM_LV=vg_test189/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0 rhgb quiet

While booting without "rhgb quiet" (which is the default), I'm not able to login.

ro root=/dev/mapper/vg_test189-lv_root rd_LVM_LV=vg_test189/lv_root rd_LVM_LV=vg_test189/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0

Comment 10 Lennart Poettering 2011-03-01 19:22:19 UTC
Hmm, so we do call "plymouth --wait" and synchronize properly to it, but somehow the tty is still locked by plymouth when the getty's open it. There's something wrong here, possibly that "plymouth --wait" doesn't correctly synchronously wait for the ply exit when "rhgb" is not passed.

Comment 11 James Laska 2011-03-01 19:25:16 UTC
(In reply to comment #8)
> Hmm, if you add "rhgb quiet", does the problem go away?

I just successfully logged in without "rhgb quiet" ... so I guess my test results differ :(

# cat /proc/cmdline 
ro root=/dev/mapper/vg_test189-lv_root rd_LVM_LV=vg_test189/lv_root rd_LVM_LV=vg_test189/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0

Comment 12 Lennart Poettering 2011-03-01 19:43:35 UTC
Which plymouth version is this?

Comment 13 Lennart Poettering 2011-03-01 19:44:19 UTC
Hmm, if you do "dracut -f" and reboot, does this fix the issue?

the fact that sometimes things work and sometimes they don't suggests this is a race.

Comment 14 Ray Strode [halfline] 2011-03-01 19:55:29 UTC
/var/log/plymouth-debug.log with plymouth.debug on kernel command line would be useful as well.

Comment 15 James Laska 2011-03-01 20:37:33 UTC
(In reply to comment #12)
> Which plymouth version is this?

 * plymouth-0.8.4-0.20110209.2.fc15.x86_64
 * kernel-2.6.38-0.rc5.git1.1.fc15.x86_64
 * systemd-17-5.fc15.x86_64

(In reply to comment #13)
> Hmm, if you do "dracut -f" and reboot, does this fix the issue?

After running $(dracut -f), I repeated the following test at least 15 times.

 1. Boot system
 2. Attempt to login from ttyS0 as root (with valid password), then logout
 3. Attempt to login from ttyS0 as root (with *invalid* password)
 4. Attempt to login again from ttyS0 as root (with valid password)
 5. Shutdown system

I'm still trying to figure out the failure case.  Sometimes the test passes, sometimes it fails.  It doesn't seem like the `dracut -f` made a difference.  Your impression about timing/race seems likely.

I next updated my system to the following packages, and repeated the tests.  

 * kernel-2.6.38-0.rc6.git6.1.fc15.x86_64
 * systemd-19-1.fc15.x86_64
 * plymouth-0.8.4-0.20110209.2.fc15.x86_64

The test results with the updated packages were not noticeably different.

Will try to reproduce the failure with 'plymouth.debug' enabled ...

Comment 16 Lennart Poettering 2011-03-01 22:24:46 UTC
Can you upgrade to plymouth-0.8.4-0.20110419.1 and see if that makes a difference? (get it from koji) You need to run "dracut -f" after upgrading to make sure it is properly copied into the initrd.

Comment 17 James Laska 2011-03-02 00:05:33 UTC
(In reply to comment #16)
> Can you upgrade to plymouth-0.8.4-0.20110419.1 and see if that makes a
> difference? (get it from koji) You need to run "dracut -f" after upgrading to
> make sure it is properly copied into the initrd.

I installed plymouth-0.8.4-0.20110419.1.fc16 (http://koji.fedoraproject.org/koji/buildinfo?buildID=229463) and rebuilt the initrd (using `dracut -f`).  Following the test procedure outlined in comment#15 ... I *still* see the problem.

Comment 18 James Laska 2011-03-02 00:07:29 UTC
Created attachment 481748 [details]
/var/log/boot.log

At Ray's request, and for completeness, I am attaching /var/log/boot.log from a failed login attempt.  After encountering the failure, I gracefully shutdown the system, and accessed /var/log/boot.log from rescue-mode.

Comment 19 Lennart Poettering 2011-03-02 02:37:39 UTC
Hmm, given that the boot.log ends where it ends we at least know that the gettys are indeed started only after plymouth has exited. So I am really tempted to claim that ply is at fault here, not systemd...

That said, James, Ray was actually asking for /var/log/plymouth-debug.log. Can you attach that too please?

Comment 20 Kamil Páral 2011-03-02 12:34:22 UTC
Created attachment 481855 [details]
plymouth-debug.log

(In reply to comment #8)
> Hmm, if you add "rhgb quiet", does the problem go away?

No. (But it stopped echoing the password on tty1.)

(In reply to comment #13)
> Hmm, if you do "dracut -f" and reboot, does this fix the issue?

No.
 
(In reply to comment #12)
> Which plymouth version is this?

plymouth-scripts-0.8.4-0.20110209.2.fc15.i686
plymouth-0.8.4-0.20110209.2.fc15.i686
plymouth-core-libs-0.8.4-0.20110209.2.fc15.i686

(In reply to comment #14)
> /var/log/plymouth-debug.log with plymouth.debug on kernel command line would be
> useful as well.

Attached.

Comment 21 Lennart Poettering 2011-03-02 12:44:24 UTC
There we are:

[ply-terminal.c]               ply_terminal_set_buffered_input:couldn't unlock terminal settings: Bad file descriptor

Reassigning to Plymouth, as Plymouth apparently doesn't unlock the TTY settings properly.

Comment 22 Lennart Poettering 2011-03-02 12:46:38 UTC
*** Bug 678720 has been marked as a duplicate of this bug. ***

Comment 23 James Laska 2011-03-02 13:21:25 UTC
(In reply to comment #19)
> That said, James, Ray was actually asking for /var/log/plymouth-debug.log. Can
> you attach that too please?

Sorry, I discussed with Ray on IRC that I haven't managed to reproduce the failure with plymouth.debug enabled.  Thankfully, Kamil managed to.

Comment 24 Lennart Poettering 2011-03-02 13:40:05 UTC
Hmm, but then again Kamil didn't use plymouth-0.8.4-0.20110419.1.fc16 to reproduce this issue.

Kamil, any chance you can try to reproduce the issue with a newer plymouth (steal it from koji) and attach the plymouth debug log for us?

Comment 25 Kamil Páral 2011-03-02 14:54:28 UTC
Created attachment 481878 [details]
plymouth-debug.log (plymouth-0.8.4-0.20110419.1.fc16)

Here you go.

The first time I booted after upgrade the system froze just before starting gettys (it happened a few times before already). It is probably a non-relevant bug. But some of that may be left in the logs, so just mentioning.

The second time I booted everything went as expected (login failed over ttyS0), log is attached.

Comment 26 Ray Strode [halfline] 2011-03-02 15:21:19 UTC
So this looks like it may be a problem that Lennart coincidentally warned me about recently.

In the log there is this:

[ply-terminal.c]                           on_tty_disconnected:tty disconnected (fd 9)
[ply-terminal.c]                           on_tty_disconnected:trying to reopen terminal '/dev/ttyS0'
[ply-terminal.c]                      ply_terminal_open_device:Unable to open terminal device '/dev/ttyS0': Input/output error

The kernel now returns EIO in some cases to mean "try again"

See

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/554172/comments/245

for more details.  I'll try post update packages soon to retry on EIO.

Comment 27 Toshio Ernie Kuratomi 2011-03-02 18:46:08 UTC
Regarding relationship of bug #678720, I'll paste the symptoms of that here so we can remember to test it when we get a fix for this.  I've tested with the following versions:

Rawhide i686 VM (kvm setup by virt-manager)
systemd-19-1.fc15.i686
plymouth-0.8.4-0.20110419.1.jlaskatest.fc16.i686

Note that I've discovered that setting plymouth.debug in grub.conf makes those symptoms go away.

= adamw's reproducer =

So, here's the ins and outs. This happens when you manage to get boot messages
displayed. There's a few ways to achieve this.

You can boot and hit 'esc' during boot, but *only* if you have the 'fixed'
systemd - systemd-18-1 . This systemd is not in the Alpha at present. You can
get this systemd, though, if you install Alpha and enable the updates-testing
repo, and that's the *default* configuration for a net install (which I think
may be a bug too).

You can also hit this with any version of systemd by booting with 'rhgb quiet'
parameters removed. That will cause you to get system messages spewed to
console, and hit this bug if you try to log in at the console.

So, to review - you don't hit this at all if you don't interact with boot at
all. You hit it just by pressing 'esc' during boot if you have the updated
systemd. You hit it by dropping 'rhgb quiet' from boot parameters with any
systemd.

= symptoms description =

When logging in, the username and password are echoed to the screen like this:
"""
Username: badger
badger
Password: fake_password_that's_displayed

Last Login: Wed Mar  2
"""

After logging in, the console is in a bad state.  tab and arrow keys are not expanded by the shell (the shell apparently gets the proper input but it doesn't display the right thing on the screen.)  vim and less don't display the buffer correctly, commands are echoed to the top line of the screen.  Control-L doesn't redraw the screen.

Comment 28 Adam Williamson 2011-03-02 22:15:51 UTC
Discussed at the go/no-go meeting of 2011-03-02. We agreed this shouldn't be a blocker - there's no criterion for serial installs (any more) and we agree this is fine, we don't care about them certainly at Alpha. The test priority in the install matrix should be changed.

Comment 29 Fedora Update System 2011-03-05 01:31:52 UTC
plymouth-0.8.4-0.20110304.1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/plymouth-0.8.4-0.20110304.1.fc15

Comment 30 Fedora Update System 2011-03-05 19:23:10 UTC
plymouth-0.8.4-0.20110304.1.fc15 has been pushed to the Fedora 15 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update plymouth'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/plymouth-0.8.4-0.20110304.1.fc15

Comment 31 Toshio Ernie Kuratomi 2011-03-07 02:43:16 UTC
The update does not fix the symptoms I reported in comment 27.  Kamil, does  this update fix your problems?

Comment 32 Michal Schmidt 2011-03-07 09:39:24 UTC
Toshio, did you run "dracut -f" after updating the plymouth package? I was able to reproduce the bug (as in the original description by Kamil) before doing that, but not afterwards.
Maybe your issue has a different cause after all.

Comment 33 Kamil Páral 2011-03-07 11:45:55 UTC
(In reply to comment #31)
> The update does not fix the symptoms I reported in comment 27.  Kamil, does 
> this update fix your problems?

Yes, it does fix this problem. (First I updated only plymouth and the problem remained, but after updating plymouth* it finally worked).

Comment 34 Toshio Ernie Kuratomi 2011-03-07 16:17:52 UTC
(In reply to comment #33)
> Yes, it does fix this problem. (First I updated only plymouth and the problem
> remained, but after updating plymouth* it finally worked).

Thanks Kamil!  I had also only updated plymouth.  With an update of plymouth-scripts and plymouth-core-libs as well, my symptoms are gone as well.  Thanks Ray!

Comment 35 Fedora Update System 2011-03-08 02:19:08 UTC
plymouth-0.8.4-0.20110304.1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.