Bug 1907746

Summary: RHVH cannot enter the new layer after upgrade testing with STIG profile selected.
Product: Red Hat Enterprise Virtualization Manager Reporter: peyu
Component: imgbasedAssignee: Sandro Bonazzola <sbonazzo>
Status: CLOSED ERRATA QA Contact: peyu
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.4.3CC: arachman, cshao, dfediuck, lsvaty, mavital, peyu, qiyuan, sbonazzo, shlei, weiwang, yaniwang
Target Milestone: ovirt-4.4.5Keywords: ZStream
Target Release: 4.4.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: imgbased-1.2.17-0.1.el8ev Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-14 11:44:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log none

Description peyu 2020-12-15 06:30:23 UTC
Created attachment 1739241 [details]
/var/log

Description of problem:
Select the STIG profile during the installation of RHVH. Then upgrade the host to the latest build. Upgrade looks successful. But when the system reboots and enters the new layer, the system will halt.


Version-Release number of selected component (if applicable):
RHVH: redhat-virtualization-host-4.4.3-20201116.0.el8_3
      redhat-virtualization-host-4.4.3-20201210.0.el8_3

How reproducible:
100%

Steps to Reproduce:
1. Install RHVH-4.4-20201117.0-RHVH-x86_64-dvd1.iso and choose the STIG profile for "security policy" in Anaconda
2. Login host, check the files in /var/imgbased/openscap
   # cat /var/imgbased/openscap/config
   ~~~~~~
   [openscap]
   configured = 1
   datastream = /usr/share/xml/scap/ssg/content/ssg-rhv4-ds.xml
   profile = xccdf_org.ssgproject.content_profile_rhvh-stig
   ~~~~~~

   # ls -al /var/imgbased/openscap/reports/
   ~~~~~~
   total 0
   dr-xr-x---. 2 root root  6 Dec 15 04:26 .
   dr-xr-x---. 3 root root 35 Dec 15 04:26 ..
   ~~~~~~
3. Upgrade RHVH to latest build "redhat-virtualization-host-4.4.3-20201210.0.el8_3"
4. Reboot and login the new layer


Actual results:
The system cannot enter the new layer, the message is as follows:
...
[  11.977158] qla2xxx [0000:41:00.01-fffe:2: Adapter shutdown successfully.
[  11.981441] reboot: System halted


Expected results:
RHVH upgrade is successful and the system enters the new layer.


Additional info:
~~~~~~
# yum update
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Red Hat update to latest                                                                                            256 kB/s | 1.1 kB     00:00    
Dependencies resolved.
====================================================================================================================================================
 Package                                                 Architecture           Version                                Repository              Size
====================================================================================================================================================
Installing:
 redhat-virtualization-host-image-update                 noarch                 4.4.3-20201210.0.el8_3                 update                 821 M
     replacing  redhat-virtualization-host-image-update-placeholder.noarch 4.4.3-1.el8ev

Transaction Summary
====================================================================================================================================================
Install  1 Package

Total download size: 821 M
Is this ok [y/N]: y
Downloading Packages:
redhat-virtualization-host-image-update-latest.rpm                                                                   96 MB/s | 821 MB     00:08    
----------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                96 MB/s | 821 MB     00:08     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                            1/1 
  Running scriptlet: redhat-virtualization-host-image-update-4.4.3-20201210.0.el8_3.noarch                                                      1/2 
  Installing       : redhat-virtualization-host-image-update-4.4.3-20201210.0.el8_3.noarch                                                      1/2 
  Running scriptlet: redhat-virtualization-host-image-update-4.4.3-20201210.0.el8_3.noarch                                                      1/2 
  Obsoleting       : redhat-virtualization-host-image-update-placeholder-4.4.3-1.el8ev.noarch                                                   2/2 
  Verifying        : redhat-virtualization-host-image-update-4.4.3-20201210.0.el8_3.noarch                                                      1/2 
  Verifying        : redhat-virtualization-host-image-update-placeholder-4.4.3-1.el8ev.noarch                                                   2/2 
Unpersisting: redhat-virtualization-host-image-update-placeholder-4.4.3-1.el8ev.noarch.rpm
Installed products updated.

Installed:
  redhat-virtualization-host-image-update-4.4.3-20201210.0.el8_3.noarch                                                                             

Complete!
~~~~~~

Comment 2 Nir Levy 2021-02-02 07:59:41 UTC
Analysis: (logs are from different runs, so numbering are not consistent)

during installation from anaconda,

Jan 26 21:17:17 localhost anaconda[1970]: program: Running in chroot '/mnt/sysroot'... kernel-install add 4.18.0-240.12.1.el8_3.x86_64 /lib/modules/4.18.0-240.12.1.el8_3.x86_64/vmlinuz

where
kernel-install - Add and remove kernel and initramfs images to and from /boot

when we upgrade from imgbase on upgrade we copy from newlayer vmlinuz into boot entry
but not to the /boot directory itself

(MainThread) safe_copy_file: /tmp/mnt.XXXXX//boot/vmlinuz-4.18.0-240.10.1.el8_3.x86_64 to /boot/rhvh-4.4.4.1-0.20210201.0+1 

the copied vmlinuz is set in the /boot/loader/entries

when on fips mode :
the cmdline option BOOT=uuid={some uuid}

causes installation to:
Mounting /dev/disk-by-uuids/{some uuid} as /boot

and right after that it reports that it cannot open file /boot/

we do not encounter that on non fips mode.

system booted successfully when either 
in dracut shell
mount -o remount,rw /boot
cp /boot/rhvh-4.4.4.1-0.20210131.0+1/vmlinuz-4.18.0-240.10.1.el8_3.x86_64 to /boot

or when the cmdline uuid removed.
(as  a long term solution that is not recommended, it will be probably an issue when booting from multipath)

Solution suggested:
imgbased to copy also to /boot

Comment 3 Sandro Bonazzola 2021-02-02 08:26:51 UTC
Sounds reasonable to me, but have we got enough space on /boot with this additional copy? What about after 4 or 5 updates?

Comment 4 Nir Levy 2021-02-07 14:35:12 UTC
kernel is removed also from /boot/ once base is removed

Comment 6 peyu 2021-02-18 07:45:47 UTC
This issue has been resolved on "redhat-virtualization-host-4.4.5-20210215.0.el8_3"


Test Steps:
1. Install RHVH-4.4-20210202.0-RHVH-x86_64-dvd1.iso and choose the STIG profile for "security policy" in Anaconda
2. Login host, check the files in /var/imgbased/openscap
   # cat /var/imgbased/openscap/config
   # ls -al /var/imgbased/openscap/reports/
3. Upgrade RHVH to latest build "redhat-virtualization-host-4.4.5-20210215.0.el8_3"
4. Reboot and login the new layer
5. Check the files in /var/imgbased/openscap as Step 2
   # cat /var/imgbased/openscap/config
   # ls -al /var/imgbased/openscap/reports/

Test result:
1. RHVH upgrade was successful.
2. A scan report was generated.
~~~~~~
# imgbase w
You are on rhvh-4.4.5.3-0.20210215.0+1

# imgbase layout
rhvh-4.4.4.1-0.20210201.0
 +- rhvh-4.4.4.1-0.20210201.0+1
rhvh-4.4.5.3-0.20210215.0
 +- rhvh-4.4.5.3-0.20210215.0+1

# cat /var/imgbased/openscap/config
[openscap]
configured = 1
datastream = /usr/share/xml/scap/ssg/content/ssg-rhv4-ds.xml
profile = xccdf_org.ssgproject.content_profile_rhvh-stig

# ls -al /var/imgbased/openscap/reports/
total 5212
dr-xr-x---. 2 root root      45 Feb 18 07:18 .
dr-xr-x---. 3 root root      35 Feb 18 07:05 ..
-rw-r--r--. 1 root root 5333923 Feb 18 07:18 scap-report-20210218071446.html
~~~~~~

Will move bug Status to "VERIFIED".

Comment 15 errata-xmlrpc 2021-04-14 11:44:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Virtualization security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1189

Comment 16 errata-xmlrpc 2021-04-14 11:48:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Virtualization security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1189