Bug 1794485

The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

(In reply to Yaniv Liberman from comment #0)
> Description of problem:
> The vdsm-hook-nestedvt package is required for nested virtualization and is
> not included in the base RHVH image, meaning that it has to be installed
> manually and gets wiped after every upgrade.

Please be sure to install it using yum.
package should be persisted on upgrade.

I asked Denis to test it next time he upgrades the RHV-TLV RHVH hosts.

We'll keep you posted.

Thanks!

Sandro,
Hosts are been updated with rhvh-4.3.8.1-0.20200126.0 and vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm has been manually installed via yum.
Will let you know if it will persist after next version update.

Chen, can you test this on upgrade from 4.3.8 to 4.3.9?

(In reply to Sandro Bonazzola from comment #5)
> Chen, can you test this on upgrade from 4.3.8 to 4.3.9?

Test steps:
1. Download vdsm-hook-nestedvt from brewweb(or from RHV agent channel).
2. yum install vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm in rhvh-4.3.8.1-0.20200126.0+1.
3. upgrade to 4.3.9
4. Check vdsm-hook-nestedvt

before upgrade:
rhvh-4.3.8.1-0.20200126.0+1.
# rpm -qa| grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

after upgrade to 4.3.9
rhvh-4.3.9.2-0.20200219.0+1

# imgbase  layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200219.0
 +- rhvh-4.3.9.2-0.20200219.0+1

# rpm -qa| grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch


Test result:
vdsm-hook-nestedvt can be persisted after upgrade.

Awsesome!
Thanks!

Thanks, so closing not a bug.

Clearing the NEEDINFO flag. Sorry if it re-opens it again, but I see no other way to clear the NEEDINFO flag and it spams my email. Just close it if it does.

I guess we'll approach you again if there are any problems in future upgrades.

Thanks!

Seems like bug persist in newer version of image, cause after upgrade from 4.3.8 to latest 4.3.9 nested hook not persist any more.
rhvh image: rhvh-4.3.9.2-0.20200324.0

Denis, was the package installed using yum install or rpm -i ?
Because persistence is ensured only with yum install.

Nir, let's ensure the package is in RHV-H Optional channel and documentation is properly updated.

see also bug 1759205

Sandro,
Packages has been installed as: 
yum install vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm

To note, that there is only one repo enabled on the hosts:
[root@rhvh-01 ~]# yum repolist
Loaded plugins: imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade, versionlock
Excluding 4 updates due to versionlock (use "yum versionlock status" to show them)
repo id                                                                          repo name                                                                             status
rhel-7-server-rhvh-4-rpms/x86_64                                                 Red Hat Virtualization Host 7 (RPMs)                                                  463
repolist: 463

I did not reproduce this issue in newer version of image(rhvh-4.3.9.2-0.20200324.0+1)

Test Steps:
1. Install rhvh-4.3.8.1-0.20200126.0+1
2. Download vdsm-hook-nestedvt from brewweb
3. yum install vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch.rpm in rhvh-4.3.8.1-0.20200126.0+1
4. Upgrade RHVH to rhvh-4.3.9.2-0.20200324.0+1
5. Check vdsm-hook-nestedvt


Test Results:

1) before upgrade:
# imgbase w
You are on rhvh-4.3.8.1-0.20200126.0+1
# rpm -qa| grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch


2) after upgrade to 4.3.9
# imgbase w
You are on rhvh-4.3.9.2-0.20200324.0+1

# imgbase layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200324.0
 +- rhvh-4.3.9.2-0.20200324.0+1

# rpm -qa| grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch


So vdsm-hook-nestedvt has been persisted after upgrade.

According to Bug 1421746 and Bug 1759205, I did a few tests to verify whether the configuration of vdsm-hook-nestedvt was persisted after upgrade. I still haven't reproduced this bug.

Test1:
1. Install rhvh-4.3.8.1-0.20200126.0+1
2. Download vdsm-hook-nestedvt from brewweb
3. yum install vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch.rpm in rhvh-4.3.8.1-0.20200126.0+1
~~~~~~
# rpm -qa | grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch

# cat /sys/module/kvm*/parameters/nested
N
~~~~~~
4. reboot host
5. Check if nested virtualizationit is enabled
~~~~~~
# cat /sys/module/kvm*/parameters/nested
Y
~~~~~~
4. Upgrade RHVH to rhvh-4.3.9.2-0.20200324.0+1
5. Check vdsm-hook-nestedvt
~~~~~~
# imgbase layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200324.0
 +- rhvh-4.3.9.2-0.20200324.0+1

# rpm -qa | grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch

# cat /sys/module/kvm*/parameters/nested
Y
~~~~~~


Test2:
1. Install rhvh-4.3.8.1-0.20200126.0+1
2. Download vdsm-hook-nestedvt from brewweb
3. yum install vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm in rhvh-4.3.8.1-0.20200126.0+1
~~~~~~
# rpm -qa | grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

# cat /sys/module/kvm*/parameters/nested
N
~~~~~~
4. Add host to RHVM and enable nested in Edit Host -> Kernel -> Nested Virtualization
5. Reboot and check if nested virtualizationit is enabled
~~~~~~
# cat /sys/module/kvm*/parameters/nested
Y
~~~~~~
4. Upgrade RHVH to rhvh-4.3.9.2-0.20200324.0+1
5. Check vdsm-hook-nestedvt
~~~~~~
# imgbase w
You are on rhvh-4.3.9.2-0.20200324.0+1

# imgbase layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200324.0
 +- rhvh-4.3.9.2-0.20200324.0+1

# rpm -qa | grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

# cat /sys/module/kvm*/parameters/nested
Y
~~~~~~

Pengshan,
We have this happened on 14 RHV-H hosts just after the update.

Hi Denis,
Is there anything I need to add to my test steps？ If so, could you give me some specific steps or version to reproduce this bug?

Pengshan,
Perhaps, it is better to look into this specific case.
Please, reach me out (irc, gchat, email) and I'll provide you with access to one of the affected rhv-h host for further investigation.

Checked two Envs which encountered the persisting problem, in summary, there are 3 main issues:

1. The persisted rpm can't be reinstalled on new layer if its dependencies provided by the image are updated to newer versions on new layer. See logs from one Env:

2020-05-05 13:57:51,980 [DEBUG] (MainThread) Running ['systemd-nspawn', '--uuid', '7e47a030ce90406b996c442369063d80', '--machine', 'rhvh-01.engineering.redhat.com', '-D', u'/tmp/mnt.F9l6q//', 'yum', 'install', '-y', '--noplugins', '/var/imgbased/persisted-rpms/nss-pam-ldapd-0.8.13-16.el7.x86_64.rpm', '/var/imgbased/persisted-rpms/nscd-2.17-260.el7_6.3.x86_64.rpm', '/var/imgbased/persisted-rpms/pam_krb5-2.4.8-6.el7.x86_64.rpm', '/var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm', '/var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm']
2020-05-05 13:57:57,824 [INFO] (MainThread) Failed to reinstall persisted RPMs!
2020-05-05 13:57:57,824 [INFO] (MainThread) Result: Spawning container rhvh-01.engineering.redhat.com on /tmp/mnt.F9l6q.
Press ^] three times within 1s to kill container.
Examining /var/imgbased/persisted-rpms/nss-pam-ldapd-0.8.13-16.el7.x86_64.rpm: nss-pam-ldapd-0.8.13-16.el7.x86_64
Marking /var/imgbased/persisted-rpms/nss-pam-ldapd-0.8.13-16.el7.x86_64.rpm to be installed
Examining /var/imgbased/persisted-rpms/nscd-2.17-260.el7_6.3.x86_64.rpm: nscd-2.17-260.el7_6.3.x86_64
Marking /var/imgbased/persisted-rpms/nscd-2.17-260.el7_6.3.x86_64.rpm to be installed
Examining /var/imgbased/persisted-rpms/pam_krb5-2.4.8-6.el7.x86_64.rpm: pam_krb5-2.4.8-6.el7.x86_64
Marking /var/imgbased/persisted-rpms/pam_krb5-2.4.8-6.el7.x86_64.rpm to be installed
Examining /var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm: vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch
Marking /var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm to be installed
Examining /var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm: vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch
Marking /var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package nscd.x86_64 0:2.17-260.el7_6.3 will be installed
--> Processing Dependency: glibc = 2.17-260.el7_6.3 for package: nscd-2.17-260.el7_6.3.x86_64
---> Package nss-pam-ldapd.x86_64 0:0.8.13-16.el7 will be installed
---> Package pam_krb5.x86_64 0:2.4.8-6.el7 will be installed
---> Package vdsm-hook-nestedvt.noarch 0:4.30.11-1.el7ev will be installed
---> Package vdsm-hook-nestedvt.noarch 0:4.30.40-1.el7ev will be installed
--> Finished Dependency Resolution
Error: Package: nscd-2.17-260.el7_6.3.x86_64 (/nscd-2.17-260.el7_6.3.x86_64)
           Requires: glibc = 2.17-260.el7_6.3
           Installed: glibc-2.17-307.el7.1.x86_64 (installed)
               glibc = 2.17-307.el7.1
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
Container rhvh-01.engineering.redhat.com failed with error code 1. 

As you can see, when reinstall the persisted nscd-2.17-260.el7_6.3.x86_64.rpm on new layer, it requires an old glibc version 2.17-260.el7_6.3, but the new layer provides 2.17-307.el7.1. 


2. All persisted rpms are installed in one yum install cmd, one package error will cause the whole yum install cmd to fail.

As you can see in the above log info, the dependency error of nscd-2.17-260.el7_6.3.x86_64 will prevent yum from installing other packages, like vdsm-hook-nestedvt.


3. If update a persisted rpm by running 'yum install/update newer-version-rpm', the newer-version-rpm can't be recorded in /var/imgbased/persisted-rpms/, when upgrade rhvh again, it's still the old rpm but not the newer-version-rpm that will be reinstalled on new rhvh layer. See infos on another Env:

$ sudo imgbase w
You are on rhvh-4.3.9.2-0.20200324.0+1

$ sudo imgbase layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200324.0
 +- rhvh-4.3.9.2-0.20200324.0+1

$ sudo rpm -qa |grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

$ sudo ls -al /var/imgbased/persisted-rpms
total 16
drwxr-xr-x. 2 root root 4096 Mar 17 13:20 .
dr-xr-x---. 4 root root 4096 Mar 17 13:20 ..
-rw-r--r--. 1 root root 4736 Mar 17 13:19 vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm

$ sudo cat /var/log/yum.log
Mar 17 13:20:13 Persisting: vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm
Mar 17 13:20:20 Installed: vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch
May 05 14:09:19 Installed: vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch
May 05 14:11:46 Installed: redhat-virtualization-host-image-update-4.3.9-20200324.0.el7_8.noarch
May 05 14:11:46 Erased: redhat-virtualization-host-image-update-placeholder-4.3.8-1.el7ev.noarch
May 06 15:12:38 Updated: vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

$ sudo vi /var/log/imgbased.log
2020-05-05 14:09:08,660 [DEBUG] (MainThread) Running ['systemd-nspawn', '--uuid', '4e0aa2cda9f44a279de2a9b58ac05c81', '--machine', 'rhev-rdu06.rhevdev.lab.eng.rdu2.redhat.com', '-D', u'/tmp/mnt.GB5Tz//', 'yum', 'install', '-y', '--noplugins', '/var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm']

As you can see, vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm is the persisted rpm, it was reinstalled on new layer at 2020-05-05 14:09. vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch is the current installed version on new layer, it was installed at May 06 15:12:38 manually.
But there is no vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm in /var/imgbased/persisted-rpms, this will cause the newer vdsm-hook-nestedvt version can't be reinstalled on the next new layer when upgrade rhvh again.


How to solve the above 3 issues?
1. For the dependencies issue, suggest users to manually install newer rpms according to the new denpendencies provided by new rhvh layer? 
2. For the all in one yum cmd issue, should we use --skip-broken?
3. For the not persisting updated rpm issue, maybe we could improve the yum plugin, which is imgbased-persist.py:
   
def pretrans_hook(conduit):
    ts = conduit.getTsInfo()
    if ts.installed:       <---> for persisting the updated rpm, need to check ts.updated. Also need to consider removing the old rpm in /var/imgbased/persisted-rpms.

We need to include it in RHV-H, it's now required by host-deploy flow when nested vt is enabled on the engine.
Let's add it to node and rhv-h kickstart

(In reply to Sandro Bonazzola from comment #21)
> We need to include it in RHV-H, it's now required by host-deploy flow when
> nested vt is enabled on the engine.
> Let's add it to node and rhv-h kickstart

it sorted out it was a user hook modification, back to original plan.

errata updated,
vdsm-4.40.16-1.el8ev now in RHEL-8-RHEV-S-4.4 ( mapped to rhel-8-server-rhvh-4-rpms repo)
product listing adjusted to include vdsm-hook-nestedvt only in whitelist.

correcting comment 23
RHEL-8-RHV-4.4-TOOLS

Change QA Contact to Wei according #c21 & 23.

Tested on redhat-virtualization-host-4.4.0-20200518.0.el8_2 using RHSM stage account,  the test failed.

Test Steps:
1. Install redhat-virtualization-host-4.4.0-20200518.0.el8_2
2. subscribe to RHSM
3. enable rhvh-4-for-rhel-8-x86_64-rpms repo
# subscription-manager repos --enable=rhvh-4-beta-for-rhel-8-x86_64-rpms

4. check the packages vdsm-hook-nestedvt
~~~~~~
# yum search vdsm-hook-nestedvt
Updating Subscription Management repositories.
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                              3.1  B/s |  10  B     00:03    
Errors during downloading metadata for repository 'rhel-8-for-x86_64-baseos-rpms':
  - Status code: 404 for https://cdn.redhat.com/content/dist/rhel8/8RedHatVirtualizationHost/x86_64/baseos/os/repodata/repomd.xml (IP: 23.1.44.251)
Error: Failed to download metadata for repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
~~~~~~


According to Bug 1779459, rhvh-4.4 cahnnel is not ready yet. So move bug status to "ASSIGNED".

This https://cdn.redhat.com/content/dist/rhel8 is a path to a GA release not a Beta release: /content/beta/layered/rhel8/x86_64/rhvh/4/os/repodata/repomd.xml

The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHV RHEL Host (ovirt-host) 4.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:3246

It won't be fixed for 4.3.z, because 4.3 comes withing EUS