This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1291643 - The previous kernel info shows in /var/log/dmesg after RHEV-H upgrade to new
The previous kernel info shows in /var/log/dmesg after RHEV-H upgrade to new
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node (Show other bugs)
3.6.0
Unspecified Unspecified
high Severity high
: ovirt-3.6.3
: ---
Assigned To: Fabian Deutsch
Ying Cui
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-15 06:41 EST by Ying Cui
Modified: 2016-03-09 09:44 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-09 09:44:13 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
varlog.tar.bz2 (6.43 MB, application/x-bzip)
2015-12-15 06:52 EST, Ying Cui
no flags Details
sosreport (9.08 MB, application/x-xz)
2015-12-15 06:54 EST, Ying Cui
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 52959 master MERGED init: Restart rhel-dmesg to ensure that /var/log/dmesg 2016-02-04 11:30 EST
oVirt gerrit 53465 ovirt-3.6 MERGED init: Restart rhel-dmesg to ensure that /var/log/dmesg 2016-02-12 08:55 EST

  None (edit)
Description Ying Cui 2015-12-15 06:41:34 EST
Description of problem:
Upgrading RHEV-H from RHEV-H 7.2 for 3.5.6 to RHEV-H 7.2 for 3.6 beta 2, then boot and login RHEV-H 7.2 for 3.6 beta2.
Checking /var/log/dmesg contents, here display the RHEV-H 7.2 for 3.5.6 kernel info and command line info.

Version-Release number of selected component (if applicable):
rhev-hypervisor7-7.2-20151210.1.el7ev
ovirt-node-3.6.0-0.24.20151209gitc0fa931.el7ev.noarch

How reproducible:
100%

Test Steps:
1. TUI installed rhevh-7.2-20151129.1   # This is rhevh 7.2 for 3.5.6 GA version.
2. Registered to RHEVM 3.5.6
3. Upgrade to rhevh-7.2-20151210.1 via RHEV-M. # This is rhevh 7.2 for 3.6 beta2 version 
4. Boot and Login rhevh-7.2-20151210.1 after upgrading.
5. 
# cat /var/log/dmesg | head -10
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-327.el7.x86_64 (mockbuild@x86-034.build.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015
[    0.000000] Command line: initrd=/images/rhevh-vdsm7-7.2-20151129.1_35/initrd0.img ksdevice=bootif rootflags=loop rootflags=ro rd.dm=0 rd_NO_MULTIPATH rd.md=0 crashkernel=256M rootfstype=auto lang=  max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 install ro root=live:/rhev-hypervisor7-7.2-20151129.1.iso rd.live.image  BOOTIF=01-bc-30-5b-e2-f7-d8 reinstall BOOT_IMAGE=/images/rhevh-vdsm7-7.2-20151129.1_35/vmlinuz0 
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bf699000-0x00000000bf6aefff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bf6af000-0x00000000bf6cdfff] ACPI data


6. 
# dmesg | head -10
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-327.3.1.el7.x86_64 (mockbuild@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Nov 20 05:40:26 EST 2015
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bf699000-0x00000000bf6aefff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bf6af000-0x00000000bf6cdfff] ACPI data


Actual results:
Compared step 5 and step 6, the previous RHEV-H 20151129.1 info shows in /var/log/dmesg contents after RHEV-H upgrade to new.

Expected results:
/var/log/dmesg should show all messages from the current kernel rather than previous kernel.

Additional info:
# uname -a
Linux dell-test-per210-01.qe.lab.eng.nay.redhat.com 3.10.0-327.3.1.el7.x86_64 #1 SMP Fri Nov 20 05:40:26

# cat /run/initramfs/live/grub2/grub.cfg
#default saved
set timeout=5
#hiddenmenu
menuentry "RHEV-H 7.2-20151210.1.el7ev" {
set root=(hd0,2)
search --no-floppy --label Root --set root
linux /vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
initrd /initrd0.img
}
menuentry "BACKUP RHEV-H 7.2-20151129.1.el7ev" {
set root=(hd0,0)
search --no-floppy --label RootBackup --set root
linux /vmlinuz0 root=live:LABEL=RootBackup ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
initrd /initrd0.img
Comment 1 Ying Cui 2015-12-15 06:52 EST
Created attachment 1105954 [details]
varlog.tar.bz2
Comment 2 Ying Cui 2015-12-15 06:54 EST
Created attachment 1105956 [details]
sosreport
Comment 4 Fabian Deutsch 2015-12-15 08:19:52 EST
So it seems that /var/log/dmesg is opened to early in the boot process. That is why /vra/log7dmesg is never updated.
But - It's available in /tmp/early-logs/dmesg:

We see that /var/log/dmesg differs from dmesg:

[root@dell-test-per210-01 admin]# diff -u <(dmesg | head -10) <(head -10 /var/log/dmesg)
dmesg: write failed: Broken pipe
dmesg: write error
--- /dev/fd/63	2015-12-15 09:12:36.857595045 +0000
+++ /dev/fd/62	2015-12-15 09:12:36.857595045 +0000
@@ -1,8 +1,8 @@
 [    0.000000] Initializing cgroup subsys cpuset
 [    0.000000] Initializing cgroup subsys cpu
 [    0.000000] Initializing cgroup subsys cpuacct
-[    0.000000] Linux version 3.10.0-327.3.1.el7.x86_64 (mockbuild@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Nov 20 05:40:26 EST 2015
-[    0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
+[    0.000000] Linux version 3.10.0-327.el7.x86_64 (mockbuild@x86-034.build.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015
+[    0.000000] Command line: initrd=/images/rhevh-vdsm7-7.2-20151129.1_35/initrd0.img ksdevice=bootif rootflags=loop rootflags=ro rd.dm=0 rd_NO_MULTIPATH rd.md=0 crashkernel=256M rootfstype=auto lang=  max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 install ro root=live:/rhev-hypervisor7-7.2-20151129.1.iso rd.live.image  BOOTIF=01-bc-30-5b-e2-f7-d8 reinstall BOOT_IMAGE=/images/rhevh-vdsm7-7.2-20151129.1_35/vmlinuz0 
 [    0.000000] e820: BIOS-provided physical RAM map:
 [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
 [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable

We see /var/log/dmesg is old:

[root@dell-test-per210-01 admin]# ls -shal /var/log/dmesg
64K -rw-r--r--. 1 root root 61K Dec 11 13:29 /var/log/dmesg
[root@dell-test-per210-01 admin]# date
Tue Dec 15 09:13:10 UTC 2015

We see that /tmp/early-logs/dmesg is actually up to date (see kver):

[root@dell-test-per210-01 admin]# head -10 /tmp/early-logs/dmesg 
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-327.3.1.el7.x86_64 (mockbuild@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Nov 20 05:40:26 EST 2015
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro ksdevice=bootif rd.dm=0 rd.md=0 crashkernel=256M lang= max_loop=256 rhgb quiet elevator=deadline rd.live.check rd.luks=0 rd.live.image
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf698fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bf699000-0x00000000bf6aefff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bf6af000-0x00000000bf6cdfff] ACPI data

So, it's not nice, but not dramatic, it doesn't look like a wide spread problem, just limited to this early boot logfile.
Comment 7 Fabian Deutsch 2016-02-02 04:14:18 EST
Maybe we can prevent that /var/log/dmesg get's written.
Comment 9 Ying Cui 2016-03-01 22:51:48 EST
Fabian, see comment 8, Thanks.
Comment 10 Ying Cui 2016-03-01 23:03:56 EST
VERIFIED this bug on the following version:

# rpm -qa ovirt-node
ovirt-node-3.6.1-8.0.el7ev.noarch
# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160301.1.el7ev)

Test steps:
1. TUI installed rhevh-7.2-20151129.1 
2. Registered to RHEVM 3.5.8
3. Upgrade to rhev-hypervisor-7.2-20160301.1.el7ev via RHEV-M. 
4. Boot and Login rhev-hypervisor-7.2-20160301.1 after upgrading.
5. Check /var/log/dmesg

Result:
/var/log/dmesg shows all messages from the current kernel.
Comment 11 Fabian Deutsch 2016-03-02 07:01:24 EST
So nce to see this fixed.
Comment 13 errata-xmlrpc 2016-03-09 09:44:13 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0378.html

Note You need to log in before you can comment on or make changes to this bug.