Description of problem: Upgrading RHEV-H from RHEV-H 7.2 for 3.5.6 to RHEV-H 7.2 for 3.6 beta 2, then boot and login RHEV-H 7.2 for 3.6 beta2. Checking /var/log/dmesg contents, here display the RHEV-H 7.2 for 3.5.6 kernel info and command line info. Version-Release number of selected component (if applicable): rhev-hypervisor7-7.2-20151210.1.el7ev ovirt-node-3.6.0-0.24.20151209gitc0fa931.el7ev.noarch How reproducible: 100% Test Steps: 1. TUI installed rhevh-7.2-20151129.1 # This is rhevh 7.2 for 3.5.6 GA version. 2. Registered to RHEVM 3.5.6 3. Upgrade to rhevh-7.2-20151210.1 via RHEV-M. # This is rhevh 7.2 for 3.6 beta2 version 4. Boot and Login rhevh-7.2-20151210.1 after upgrading. 5. # cat /var/log/dmesg | head -10 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-327.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015 [ 0.000000] Command line: initrd=/images/rhevh-vdsm7-7.2-20151129.1_35/initrd0.img ksdevice=bootif rootflags=loop rootflags=ro rd.dm=0 rd_NO_MULTIPATH rd.md=0 crashkernel=256M rootfstype=auto lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 install ro root=live:/rhev-hypervisor7-7.2-20151129.1.iso rd.live.image BOOTIF=01-bc-30-5b-e2-f7-d8 reinstall BOOT_IMAGE=/images/rhevh-vdsm7-7.2-20151129.1_35/vmlinuz0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bf699000-0x00000000bf6aefff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bf6af000-0x00000000bf6cdfff] ACPI data 6. # dmesg | head -10 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-327.3.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Nov 20 05:40:26 EST 2015 [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bf699000-0x00000000bf6aefff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bf6af000-0x00000000bf6cdfff] ACPI data Actual results: Compared step 5 and step 6, the previous RHEV-H 20151129.1 info shows in /var/log/dmesg contents after RHEV-H upgrade to new. Expected results: /var/log/dmesg should show all messages from the current kernel rather than previous kernel. Additional info: # uname -a Linux dell-test-per210-01.qe.lab.eng.nay.redhat.com 3.10.0-327.3.1.el7.x86_64 #1 SMP Fri Nov 20 05:40:26 # cat /run/initramfs/live/grub2/grub.cfg #default saved set timeout=5 #hiddenmenu menuentry "RHEV-H 7.2-20151210.1.el7ev" { set root=(hd0,2) search --no-floppy --label Root --set root linux /vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image initrd /initrd0.img } menuentry "BACKUP RHEV-H 7.2-20151129.1.el7ev" { set root=(hd0,0) search --no-floppy --label RootBackup --set root linux /vmlinuz0 root=live:LABEL=RootBackup ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image initrd /initrd0.img
Created attachment 1105954 [details] varlog.tar.bz2
Created attachment 1105956 [details] sosreport
So it seems that /var/log/dmesg is opened to early in the boot process. That is why /vra/log7dmesg is never updated. But - It's available in /tmp/early-logs/dmesg: We see that /var/log/dmesg differs from dmesg: [root@dell-test-per210-01 admin]# diff -u <(dmesg | head -10) <(head -10 /var/log/dmesg) dmesg: write failed: Broken pipe dmesg: write error --- /dev/fd/63 2015-12-15 09:12:36.857595045 +0000 +++ /dev/fd/62 2015-12-15 09:12:36.857595045 +0000 @@ -1,8 +1,8 @@ [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct -[ 0.000000] Linux version 3.10.0-327.3.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Nov 20 05:40:26 EST 2015 -[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image +[ 0.000000] Linux version 3.10.0-327.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015 +[ 0.000000] Command line: initrd=/images/rhevh-vdsm7-7.2-20151129.1_35/initrd0.img ksdevice=bootif rootflags=loop rootflags=ro rd.dm=0 rd_NO_MULTIPATH rd.md=0 crashkernel=256M rootfstype=auto lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 install ro root=live:/rhev-hypervisor7-7.2-20151129.1.iso rd.live.image BOOTIF=01-bc-30-5b-e2-f7-d8 reinstall BOOT_IMAGE=/images/rhevh-vdsm7-7.2-20151129.1_35/vmlinuz0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable We see /var/log/dmesg is old: [root@dell-test-per210-01 admin]# ls -shal /var/log/dmesg 64K -rw-r--r--. 1 root root 61K Dec 11 13:29 /var/log/dmesg [root@dell-test-per210-01 admin]# date Tue Dec 15 09:13:10 UTC 2015 We see that /tmp/early-logs/dmesg is actually up to date (see kver): [root@dell-test-per210-01 admin]# head -10 /tmp/early-logs/dmesg [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-327.3.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Nov 20 05:40:26 EST 2015 [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bf699000-0x00000000bf6aefff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bf6af000-0x00000000bf6cdfff] ACPI data So, it's not nice, but not dramatic, it doesn't look like a wide spread problem, just limited to this early boot logfile.
Maybe we can prevent that /var/log/dmesg get's written.
Fabian, see comment 8, Thanks.
VERIFIED this bug on the following version: # rpm -qa ovirt-node ovirt-node-3.6.1-8.0.el7ev.noarch # cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160301.1.el7ev) Test steps: 1. TUI installed rhevh-7.2-20151129.1 2. Registered to RHEVM 3.5.8 3. Upgrade to rhev-hypervisor-7.2-20160301.1.el7ev via RHEV-M. 4. Boot and Login rhev-hypervisor-7.2-20160301.1 after upgrading. 5. Check /var/log/dmesg Result: /var/log/dmesg shows all messages from the current kernel.
So nce to see this fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0378.html