Bug 1291643

Summary: The previous kernel info shows in /var/log/dmesg after RHEV-H upgrade to new
Product: Red Hat Enterprise Virtualization Manager Reporter: Ying Cui <ycui>
Component: ovirt-nodeAssignee: Fabian Deutsch <fdeutsch>
Status: CLOSED ERRATA QA Contact: Ying Cui <ycui>
Severity: high Docs Contact:
Priority: high    
Version: 3.6.0CC: bgraveno, fdeutsch, gklein, lsurette, mgoldboi, rbarry, ycui, ykaul
Target Milestone: ovirt-3.6.3   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-09 14:44:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
varlog.tar.bz2
none
sosreport none

Description Ying Cui 2015-12-15 11:41:34 UTC
Description of problem:
Upgrading RHEV-H from RHEV-H 7.2 for 3.5.6 to RHEV-H 7.2 for 3.6 beta 2, then boot and login RHEV-H 7.2 for 3.6 beta2.
Checking /var/log/dmesg contents, here display the RHEV-H 7.2 for 3.5.6 kernel info and command line info.

Version-Release number of selected component (if applicable):
rhev-hypervisor7-7.2-20151210.1.el7ev
ovirt-node-3.6.0-0.24.20151209gitc0fa931.el7ev.noarch

How reproducible:
100%

Test Steps:
1. TUI installed rhevh-7.2-20151129.1   # This is rhevh 7.2 for 3.5.6 GA version.
2. Registered to RHEVM 3.5.6
3. Upgrade to rhevh-7.2-20151210.1 via RHEV-M. # This is rhevh 7.2 for 3.6 beta2 version 
4. Boot and Login rhevh-7.2-20151210.1 after upgrading.
5. 
# cat /var/log/dmesg | head -10
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-327.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015
[    0.000000] Command line: initrd=/images/rhevh-vdsm7-7.2-20151129.1_35/initrd0.img ksdevice=bootif rootflags=loop rootflags=ro rd.dm=0 rd_NO_MULTIPATH rd.md=0 crashkernel=256M rootfstype=auto lang=  max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 install ro root=live:/rhev-hypervisor7-7.2-20151129.1.iso rd.live.image  BOOTIF=01-bc-30-5b-e2-f7-d8 reinstall BOOT_IMAGE=/images/rhevh-vdsm7-7.2-20151129.1_35/vmlinuz0 
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bf699000-0x00000000bf6aefff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bf6af000-0x00000000bf6cdfff] ACPI data


6. 
# dmesg | head -10
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-327.3.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Nov 20 05:40:26 EST 2015
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bf699000-0x00000000bf6aefff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bf6af000-0x00000000bf6cdfff] ACPI data


Actual results:
Compared step 5 and step 6, the previous RHEV-H 20151129.1 info shows in /var/log/dmesg contents after RHEV-H upgrade to new.

Expected results:
/var/log/dmesg should show all messages from the current kernel rather than previous kernel.

Additional info:
# uname -a
Linux dell-test-per210-01.qe.lab.eng.nay.redhat.com 3.10.0-327.3.1.el7.x86_64 #1 SMP Fri Nov 20 05:40:26

# cat /run/initramfs/live/grub2/grub.cfg
#default saved
set timeout=5
#hiddenmenu
menuentry "RHEV-H 7.2-20151210.1.el7ev" {
set root=(hd0,2)
search --no-floppy --label Root --set root
linux /vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
initrd /initrd0.img
}
menuentry "BACKUP RHEV-H 7.2-20151129.1.el7ev" {
set root=(hd0,0)
search --no-floppy --label RootBackup --set root
linux /vmlinuz0 root=live:LABEL=RootBackup ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
initrd /initrd0.img

Comment 1 Ying Cui 2015-12-15 11:52:47 UTC
Created attachment 1105954 [details]
varlog.tar.bz2

Comment 2 Ying Cui 2015-12-15 11:54:20 UTC
Created attachment 1105956 [details]
sosreport

Comment 4 Fabian Deutsch 2015-12-15 13:19:52 UTC
So it seems that /var/log/dmesg is opened to early in the boot process. That is why /vra/log7dmesg is never updated.
But - It's available in /tmp/early-logs/dmesg:

We see that /var/log/dmesg differs from dmesg:

[root@dell-test-per210-01 admin]# diff -u <(dmesg | head -10) <(head -10 /var/log/dmesg)
dmesg: write failed: Broken pipe
dmesg: write error
--- /dev/fd/63	2015-12-15 09:12:36.857595045 +0000
+++ /dev/fd/62	2015-12-15 09:12:36.857595045 +0000
@@ -1,8 +1,8 @@
 [    0.000000] Initializing cgroup subsys cpuset
 [    0.000000] Initializing cgroup subsys cpu
 [    0.000000] Initializing cgroup subsys cpuacct
-[    0.000000] Linux version 3.10.0-327.3.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Nov 20 05:40:26 EST 2015
-[    0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
+[    0.000000] Linux version 3.10.0-327.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015
+[    0.000000] Command line: initrd=/images/rhevh-vdsm7-7.2-20151129.1_35/initrd0.img ksdevice=bootif rootflags=loop rootflags=ro rd.dm=0 rd_NO_MULTIPATH rd.md=0 crashkernel=256M rootfstype=auto lang=  max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 install ro root=live:/rhev-hypervisor7-7.2-20151129.1.iso rd.live.image  BOOTIF=01-bc-30-5b-e2-f7-d8 reinstall BOOT_IMAGE=/images/rhevh-vdsm7-7.2-20151129.1_35/vmlinuz0 
 [    0.000000] e820: BIOS-provided physical RAM map:
 [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
 [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable

We see /var/log/dmesg is old:

[root@dell-test-per210-01 admin]# ls -shal /var/log/dmesg
64K -rw-r--r--. 1 root root 61K Dec 11 13:29 /var/log/dmesg
[root@dell-test-per210-01 admin]# date
Tue Dec 15 09:13:10 UTC 2015

We see that /tmp/early-logs/dmesg is actually up to date (see kver):

[root@dell-test-per210-01 admin]# head -10 /tmp/early-logs/dmesg 
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-327.3.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Nov 20 05:40:26 EST 2015
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bf699000-0x00000000bf6aefff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bf6af000-0x00000000bf6cdfff] ACPI data

So, it's not nice, but not dramatic, it doesn't look like a wide spread problem, just limited to this early boot logfile.

Comment 7 Fabian Deutsch 2016-02-02 09:14:18 UTC
Maybe we can prevent that /var/log/dmesg get's written.

Comment 9 Ying Cui 2016-03-02 03:51:48 UTC
Fabian, see comment 8, Thanks.

Comment 10 Ying Cui 2016-03-02 04:03:56 UTC
VERIFIED this bug on the following version:

# rpm -qa ovirt-node
ovirt-node-3.6.1-8.0.el7ev.noarch
# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160301.1.el7ev)

Test steps:
1. TUI installed rhevh-7.2-20151129.1 
2. Registered to RHEVM 3.5.8
3. Upgrade to rhev-hypervisor-7.2-20160301.1.el7ev via RHEV-M. 
4. Boot and Login rhev-hypervisor-7.2-20160301.1 after upgrading.
5. Check /var/log/dmesg

Result:
/var/log/dmesg shows all messages from the current kernel.

Comment 11 Fabian Deutsch 2016-03-02 12:01:24 UTC
So nce to see this fixed.

Comment 13 errata-xmlrpc 2016-03-09 14:44:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0378.html