Bug 691374

Summary: No way to login into a VM's serial console.
Product: [Fedora] Fedora Reporter: Gilboa Davara <gilboad>
Component: plymouthAssignee: Ray Strode [halfline] <rstrode>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 15CC: bugzilla, fedora, gavdav, hdegoede, johannbg, john, jvillalo, kchamart, lpoetter, marchenko, metherid, mfuruta, mschmidt, notting, ondrejj, paolini, plautrba, rstrode
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-07 18:18:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gilboa Davara 2011-03-28 11:40:41 UTC
I've installed F15/x86_64 VM using the GNOME LiveCD (F14/x86_64 host).
I'm updated the machine to latest and enabled serial console by adding "console=tty console=ttyS0,57600n8". [1]
I've also added the required serial console configuration into grub.conf [2].

During the boot, the serial console seems to be working just fine; however, once I get the login prompt, trying to type anything into the login or password prompt also seems to send spontaneous new-lines which prevents me from logging in. (I occasionally managed to type the user name, but never managed to type the password).

I should add that the same configuration seems to be working just fine under RHEL/CentOS 5.x and Fedora 14 guests.

[0] VM configuration:
$ cat /proc/9346/cmdline 
qemu-kvm -cdrom /dev/sr0
  -serial telnet::XXXX,server,nowait -soundhw ac97
  -net nic,vlan=1,macaddr=00:E0:81:B0:FB:10,model=e1000
  -net tap,vlan=1,ifname=tap100
  -net nic,vlan=2,macaddr=00:E0:81:B0:FB:1A,model=e1000
  -net tap,vlan=2,ifname=tap101
  -m 2048
  -name Fedora-Next
  -hda /usr/drives/kvm/Fedora-Next/Fedora-Next.img
  -smp 4 -usb -usbdevice tablet
  -vga qxl -spiceport=XXXX,disable-ticketing
  -boot c

[1] Kernel command line:
$ cat /proc/cmdline
ro root=/dev/mapper/VolRoot-LogRoot rd_LVM_LV=VolRoot/LogRoot rd_LVM_LV=VolRoot/LogSwap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rhgb quiet console=tty console=ttyS0,57600n8 vga=0x37f

[2] Grub configuration:
serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console

Comment 1 Gilboa Davara 2011-03-28 11:41:18 UTC
Forgot to add:
$ rpm -qa | grep systemd
systemd-units-20-1.fc15.x86_64
systemd-20-1.fc15.x86_64

Comment 2 Michal Schmidt 2011-03-28 13:02:47 UTC
There was a very similar bug before: bug 681167

What happens if you drop the "console=tty" parameter?

Comment 3 Gilboa Davara 2011-04-02 08:39:01 UTC
Yes.
Dropping the console=tty seems to solve the problem.

- Gilboa

Comment 4 Michal Schmidt 2011-04-02 15:25:48 UTC
I can reproduce this. Indeed "console=tty console=ttyS0" is necessary for the bug to happen. When "console=ttyS0" is used alone, it works fine. In case anyone wonders - passing "console=..." twice is perfectly legitimate. It tells the kernel to output its messages on both the virtual terminal and the serial console. And the last device (ttyS0 in this case) is to be used for /dev/console.

The terminal settings of /dev/ttyS0 must be locked, because "stty -F /dev/ttyS0 sane" complains: "unable to perform all requested operations"

Passing "rd_NO_PLYMOUTH" and masking plymouth-start.service using "ln -s /dev/null /etc/systemd/system/plymouth-start.service" works as a workaround.

This indicates plymouth is to blame for the broken terminal settings. Reassigning.

Comment 5 Michal Schmidt 2011-05-09 15:02:28 UTC
*** Bug 683729 has been marked as a duplicate of this bug. ***

Comment 6 John Villalovos 2011-06-02 17:27:08 UTC
I see this bug in a NON-VM setting.

Comment 7 John Villalovos 2011-06-02 18:42:47 UTC
My issue actually only occurs when I only have "console=ttyS4,115200n81".  If I use "console=tty0 console=ttyS4,115200n81" then it works.

Comment 8 Maurizio Paolini 2011-06-18 10:45:23 UTC
I have a very similar problem, but concerning /dev/tty1 after upgrading to
fedora 15:
After boot (ex runlevel 3) the user password is not hidden.  And indeed
the command "stty -F /dev/tty1 sane" gives "unable to perform all requested operations".

The problem is not systematic, but shows up with (say) 50% probability
after each reboot.

Following the suggestions of Michal Schmidt above (rd_NO_PLYMOUTH and
disallowing plymouth-start.service as described) *solves* the problem.

We have a lab with approx 20 very old compaq PCs that were upgraded
subsequently with "yum upgrade" from previous fedora versions.

Comment 9 Jason 2011-07-22 12:54:04 UTC
I'm having this same problem with RHEL 6.1 on all my servers: virtual (VMware ESXi 4.1) and physical (Dell Poweredge R510s, R610s).  I suggest updating the subject of the bug to reflect that.

$ sudo stty --file=/dev/ttyS0 -clocal
stty: /dev/ttyS0: unable to perform all requested operations

Once echo is disabled for typing my password, each of my characters turn into newlines until agetty is restarted.

cassdevb01 login: root
Password: <newline>
Login incorrect

login:  <newline>
Password:  <newline>
Login incorrect

login:  <newline>


I believe Bug 711411 is related.

Comment 10 Gilboa Davara 2011-11-02 15:00:27 UTC
I can confirm that Michal Schmidt work around seem to work just fine.
Any chance of fixing it before F16 GA?

- Gilboa

Comment 11 Michal Schmidt 2011-11-02 16:12:01 UTC
No, F16 is practically done and this is not a blocker.

Comment 12 Michal Schmidt 2012-01-02 13:50:47 UTC
*** Bug 770995 has been marked as a duplicate of this bug. ***

Comment 13 Michal Schmidt 2012-01-02 14:00:03 UTC
*** Bug 755003 has been marked as a duplicate of this bug. ***

Comment 14 John Florian 2012-01-06 17:03:58 UTC
(In reply to comment #4)

> Passing "rd_NO_PLYMOUTH" and masking plymouth-start.service using "ln -s
> /dev/null /etc/systemd/system/plymouth-start.service" works as a workaround.
> 
> This indicates plymouth is to blame for the broken terminal settings.

This work-around does allow me to successfully login via the serial console, but it has undesirable side effects, specifically the messages that would appear during boot are now completely gone with plymouth disabled.

I need console messages to appear on both tty1 and ttyS1 and I need working logins on both for hundreds of embedded systems.  Presently, I don't see how this can be achieved, with or without work-arounds.

I have zero need for any of plymouth's "features", but could live with it if it didn't present such issues.  I'd attempt my builds without installing plymouth, but dracut has a dependency on it and my builds do require dracut.

Any advice on how to meet my needs would be greatly appreciated.

Comment 15 Jan ONDREJ 2012-01-06 17:57:01 UTC
(In reply to comment #14)
> I need console messages to appear on both tty1 and ttyS1 and I need working
> logins on both for hundreds of embedded systems.  Presently, I don't see how
> this can be achieved, with or without work-arounds.

+1, same needs for me on kvm virtual machines and servers with serial remote console.

> I have zero need for any of plymouth's "features", but could live with it if it
> didn't present such issues.  I'd attempt my builds without installing plymouth,
> but dracut has a dependency on it and my builds do require dracut.

One thing, which I think plymouth goes is, that it has better logging into /var/log/boot.log, which works well with plymouth.
But for me it's only secondary feature, primarily I need serial boot console and login.

Comment 16 John Florian 2012-01-06 18:25:50 UTC
(In reply to comment #15)

> One thing, which I think plymouth goes is, that it has better logging into
> /var/log/boot.log, which works well with plymouth.
> But for me it's only secondary feature, primarily I need serial boot console
> and login.

Yes boot.log is useful ... if the system gets that far.  I'm hesitant to deploy systems that cannot show their boot progress because if any get stuck prior to login/ssh startup, I'll have zero clues from which to debug.  Serial console access is my last resort when bad luck visits and it's hard to choose between boot messages and login.

In the past few Fedora releases (for this problem has been around since at least F13), my work-around was to have a local, non-root account with a single character password from which I could then su.  Horribly insecure, but it's the most workable solution I've yet found.  With that setup, you must get the user name correct on the first try and press the key corresponding to your password.  You won't need to press Enter on the password.  Term IO is funky thereafter, but sufficient in a pinch.  If you botch the id or password, you need a new agetty instance, so press Enter a few times until the login reappears at the *top* of the screen. (This matches what's described by Jason in comment 9.)

Comment 17 Michal Schmidt 2012-01-09 13:23:42 UTC
(In reply to comment #14)
> the messages that would appear during boot are now completely gone with
> plymouth disabled.

What messages are you talking about exactly?
I get the kernel's boot messages and systemd messages on both the serial console and the VGA console here with plymouth disabled.
Are you passing the "quiet" option?

Comment 18 John Florian 2012-01-09 15:17:06 UTC
(In reply to comment #17)

> What messages are you talking about exactly?

The usual messages from each service (e.g., SSH) as they're started up.  I suspect it would be everything handled by the systemd-stdout-syslog-bridge process, but that's just a guess.  Let's just call them the init messages as opposed to the kernel messages.

> Are you passing the "quiet" option?

Yes, because it's forced on all images produced by livecd-tools.  I can tell livecd-tools to append more options, but a few like "quiet" are there by default.  The last time I investigated that behavior, there was nothing that could be done about it short of hacking up python-imgcreate.  If I boot one of these images and manually remove "quiet" ala grub's editor, I do get the expect messages on both consoles although /var/log/boot.log is nearly empty having only "NET[850]: /sbin/dhclient-script : updated /etc/resolv.conf".

Comment 19 Gavin Davenport 2012-06-27 18:47:36 UTC
Is this going to get attention ?

FWIW I have a 32bit FC15 and a 64bit FC15 (both upgrades from about FC12).

It only happens on the 64bit instance.

I have removed plymouth and dracut as I have no need for them.

Comment 20 Fedora End Of Life 2012-08-07 18:18:24 UTC
This message is a notice that Fedora 15 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 15. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '15' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 15 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 21 Hans de Goede 2013-01-02 14:26:38 UTC
Note for others coming over here after a google search like me :)

It seems that this bug is fixed in F-18 (don't know about F-16 / F-17), as long as the console=ttyS0,... is the last console= on the kernel cmdline, iow the serial console is the default console, then systemd will automatically start a serial getty on it.

Comment 22 John Florian 2013-01-02 19:29:29 UTC
(In reply to comment #21)

> It seems that this bug is fixed in F-18 (don't know about F-16 / F-17), as
> long as the console=ttyS0,... is the last console= on the kernel cmdline,
> iow the serial console is the default console, then systemd will
> automatically start a serial getty on it.

That matches my observation as well.  In my F18 tests, I have "console=ttyS1,115200 console=tty0" on the kernel cmdline and both of these consoles work very well now for both kernel messages and logins.  I no longer need any of the silly kludges -- it just works as it should.

The last time I tried with F16, it was still broken though and I don't know at all how well F17 works in this regard.