Description of problem: As part of our kernel testing, I need verify that RHEL7 x86_64 XEN Guests install on a RHEL5 XEN DOM0. The kickstart serial console configurations for the XEN Guests are incorrect. Version-Release number of selected component (if applicable): Beaker 0.18.1 How reproducible: consistently Actual results: https://beaker.engineering.redhat.com/jobs/763143 https://beaker.engineering.redhat.com/kickstart/978324 -----<-snip->----- # System bootloader configuration bootloader --location=mbr --append="console=ttyS0,115200 console=ttyS1,115200" text -----<-snip->----- Expected results: Correct serial console configs for Guests Additional info:
Okay, so currently a RHEL6 guest on RHEL5 will have this in its kernel command line after installation: console=ttyS0,115200 console=ttyS1,115200 ... console=ttyS0,115200 which is why it works. /dev/ttyS1 is indeed non-functional but /dev/console is pointing at ttyS0 and so we get kernel messages and beah output going to it (which the host sends to the console log file). I don't know where the second console=ttyS0 is coming from, I think it must be set by Anaconda. With this bug, a RHEL7 guest on RHEL5 will have this in its kernel command line after installation: console=ttyS1,115200 ... console=ttyS0,115200 which looks like it should work, for the same reason... but actually it doesn't. On RHEL7 /dev/console is pointing at ttyS1 and it breaks. # cat /sys/devices/virtual/tty/console/active ttyS1 The kernel docs say this: Note that you can only define one console per device type (serial, video). which suggests that maybe the RHEL7 kernel is just ignoring the second console=ttyS0 because we have already defined one serial console.
So I think the right solution here is to just drop --append="console=ttyS0 console=ttyS1" from the guest kickstarts completely. On RHEL6 we will still configure upstart to run agetty on ttyS0 and ttyS1, and it seems to happily ignore ttyS1 if it doesn't exist. On RHEL7 systemd takes care of running agetty on all consoles for us automatically. So we will still always have a login prompt everywhere possible. So the guests will always have their kernel output and /dev/console output going to ttyS0, which is always the guest console log. And on KVM where ttyS1 is connected to a pty for virsh console, we will always get a login prompt. I just need to test all the various combinations to make sure that that really works...
Patch to remove --append="console=ttyS0 ttyS1" from kickstarts: http://gerrit.beaker-project.org/3369 I noticed that /distribution/virt/install is checking for "ttyS1" in the kickstart before enabling the second serial port. We can just do that unconditionally: http://gerrit.beaker-project.org/3370 I've tested all the relevant combinations and I think the results are good. === RHEL7 on RHEL5 === https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/704 Install cmdline has: console=tty0 console=ttyS0,115200 Post-install cmdline has: console=ttyS0,115200 The VM has a serial port redirected to a pty. zrhel5_write_consolelogs.py runs virsh console to capture the output. Install console log is captured in /distribution/virt/install results. Post-install console log is uploaded as console.log for the guest recipe. Running 'virsh console' on the host leads to garbled output, due to the other running 'virsh console' process. It is not possible to log in using virsh console. (I think this must be the expected behaviour, due to not having a second serial port on RHEL5 hosts.) Graphical console shows a login prompt but no kernel messages or beah output. === RHEL6 on RHEL5 === https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/705 As above for RHEL7 on RHEL5, except that I can't get a login prompt on the graphical console. I think this must be an existing, unrelated problem. === RHEL5 on RHEL5 === https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/706 As above for RHEL6 on RHEL5. === RHEL7 on RHEL6 === https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/708 Install cmdline has: console=tty0 console=ttyS0,115200 Post-install cmdline has: console=ttyS0,115200 The VM has two serial ports, first one is redirected to a log file, second one is redirected to a pty. Install console log is captured in /distribution/virt/install results. Guest recipe console.log has both install log and post-install log. (This is expected because QEMU is writing the logs, and so everything is logged to the same file during install and afterwards.) Running 'virsh console <guestname>' on the host gives an error: error: internal error character device (null) is not using a PTY This is expected because the serial port is redirected to a file, not a pty. Running 'virsh console <guestname> serial1' on the host works, but there is no login prompt. That's because systemd in the guest has not been told to listen on ttyS1. I don't think this is a regression, but it would be a useful RFE. === RHEL6 on RHEL6 === https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/709 As above for RHEL7 on RHEL6, except that 'virsh console <guestname> serial1' does show a login prompt because we configure upstart to run a getty on ttyS1. === RHEL5 on RHEL6 === https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/707 As above for RHEL7 on RHEL6. No login prompt on ttyS1 because we never configure it (and I don't even know how, with the old initscripts...)
Paul, Jan, can you please have a look over my results in comment 4 and make sure they are what you expect? Can you see anything I have missed?
(In reply to comment #1) > > If you need a desperate > > workaround you could try something like this (untested): > > > > <ks_append> > > %post > > sed -i -e 's/console=\S*//g' /boot/grub2/grub.cfg > > %end > > </ks_append> I tested this workaround, and it does get the RHEL7 guest recipes running, but it removes all consoles logs. I should have realised that would happen. A better option might be this (haven't tested this one yet): <ks_append> %post sed -i -e 's/console=ttyS1\S*//g' /boot/grub2/grub.cfg %end </ks_append> I forgot to mention originally, this would go inside <ks_appends/> in the *guest* recipe.
(In reply to Dan Callaghan from comment #4) > === RHEL7 on RHEL5 === > https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/704 fullvirt xen guest > > Install cmdline has: console=tty0 console=ttyS0,115200 > Post-install cmdline has: console=ttyS0,115200 > > The VM has a serial port redirected to a pty. zrhel5_write_consolelogs.py > runs virsh console to capture the output. > Install console log is captured in /distribution/virt/install results. > Post-install console log is uploaded as console.log for the guest recipe. > > Running 'virsh console' on the host leads to garbled output, due to the > other running 'virsh console' process. It is not possible to log in using > virsh console. (I think this must be the expected behaviour, due to not > having a second serial port on RHEL5 hosts.) I think this never worked and extra console was only for RHEL6+: Bug 978419 - RFE: Change in rhel6 guest kickstarts I'm not sure if Gurhan's intention was also to have kernel messages printed on both consoles - I'm happy with 2nd console just being available for login, there are other ways to get to kernel messages. > > Graphical console shows a login prompt but no kernel messages or beah output. > > === RHEL6 on RHEL5 === > https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/705 fullvirt xen guest > > As above for RHEL7 on RHEL5, except that I can't get a login prompt on the > graphical console. I think this must be an existing, unrelated problem. Doesn't ring any bells. > > === RHEL5 on RHEL5 === > https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/706 fullvirt xen guest > > As above for RHEL6 on RHEL5. > > === RHEL7 on RHEL6 === > https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/708 KVM guest > > Install cmdline has: console=tty0 console=ttyS0,115200 > Post-install cmdline has: console=ttyS0,115200 > > The VM has two serial ports, first one is redirected to a log file, second > one is redirected to a pty. > Install console log is captured in /distribution/virt/install results. > Guest recipe console.log has both install log and post-install log. (This is > expected because QEMU is writing the logs, and so everything is logged to > the same file during install and afterwards.) > > Running 'virsh console <guestname>' on the host gives an error: > error: internal error character device (null) is not using a PTY > This is expected because the serial port is redirected to a file, not a pty. > > Running 'virsh console <guestname> serial1' on the host works, but there is > no login prompt. That's because systemd in the guest has not been told to > listen on ttyS1. I don't think this is a regression, but it would be a > useful RFE. I'm not 100% sure, but I think this worked. Isn't systemd generating getty unit for each "console="? http://www.freedesktop.org/software/systemd/man/systemd-getty-generator.html > > === RHEL6 on RHEL6 === > https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/709 KVM guest > > As above for RHEL7 on RHEL6, except that 'virsh console <guestname> serial1' > does show a login prompt because we configure upstart to run a getty on > ttyS1. > > === RHEL5 on RHEL6 === > https://beaker.dcallagh.beakerdevs.lab.eng.bne.redhat.com/jobs/707 KVM guest > > As above for RHEL7 on RHEL6. No login prompt on ttyS1 because we never > configure it (and I don't even know how, with the old initscripts...) The jobs above look OK to me.(In reply to Dan Callaghan from comment #5) > Can you see anything I have missed? Your test matrix doesn't include any paravirt guests on RHEL5, latest RHEL5 KT1 job (J:752959) is from 2014-09-20, so we probably don't have "bad" job to compare it with - I can clone one. Your test matrix doesn't include KVM guest on RHEL5.
(In reply to Jan Stancek from comment #7) > Your test matrix doesn't include KVM guest on RHEL5. I didn't even know that KVM was backported to RHEL5... How can I run that? Does /distribution/virt/install know how to set it up? Does it need a non-standard kernel? Anyway at this point I don't want to hold up this bug for it. There should be nothing that is host-distro-specific in these changes. The --append="" stuff (and hence, the behaviour of virt/install which was looking for ttyS1 in the kickstart) was always present in RHEL6 guests regardless of their host.
(In reply to Dan Callaghan from comment #9) > (In reply to Jan Stancek from comment #7) > > Your test matrix doesn't include KVM guest on RHEL5. > > I didn't even know that KVM was backported to RHEL5... How can I run that? You can add --kvm to guestargs. I think KVM support is there since RHEL5.4 or 5.6. For example: guestargs="--ram=2048 --vcpus=2 --file-size=20 --hvm --kvm --nonsparse" guestname="x86_64_kvm" This job (on production beaker) lists all different combinations we run: J:719191 > Does /distribution/virt/install know how to set it up? Yes, it should be looking for --kvm parameter: if ! kvm_num=$(./get_guest_info.py --kvm-num) ; then On RHEL6+ it always assumes KVM: if [[ ${ver} == 6 || ${ver} > 6 ]]; then kvm_num=1 else > Does it need non-standard kernel? No, KVM uses standard kernel. RHEL5 defaults to xen, because there is "kernel-xen" in Makefile Requires: @echo "Requires: kernel-xen" >> $(METADATA) in case of KVM, it should try to reboot to standard kernel: yum -y erase kernel-xen
Beaker 0.18.2 has been released.
All, Testing following update to Beaker 0.18.2 is successful: [] https://beaker.engineering.redhat.com/jobs/767529 [] https://beaker.engineering.redhat.com/jobs/767536 We have successful Guest installs and console logs. Thank you, Dan. Best, -pbunyan
Was this change reverted? I see the issue again on 0.18.3: https://beaker.engineering.redhat.com/jobs/779981
(In reply to Jan Stancek from comment #15) > Was this change reverted? Yes, it was accidentally reverted in 0.18.3 due to a rebase mistake. Cloned bug 1155331. We are working on fixing this ASAP. Workarounds from comment 6 should still be applicable.