Bug 1794485 - [RFE] Please Add the vdsm-hook-nestedvt Package to RHVH optional channel
Summary: [RFE] Please Add the vdsm-hook-nestedvt Package to RHVH optional channel
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: redhat-virtualization-host
Version: 4.3.7
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-4.4.1
: 4.4.1
Assignee: Nir Levy
QA Contact: shiyi lei
URL:
Whiteboard:
Depends On:
Blocks: 1759205 1832721
TreeView+ depends on / blocked
 
Reported: 2020-01-23 17:09 UTC by Yaniv Liberman
Modified: 2020-11-06 10:40 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
This enhancement adds the vdsm-hook-nestedvt rpm package to the Red Hat Virtualization Host (RHVH) optional channel. This package enables you to run virtual machines within virtual machines. It is only intended for evaluation purposes and not supported for production purposes.
Clone Of:
Environment:
Last Closed: 2020-08-04 13:27:53 UTC
oVirt Team: Node
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 5553261 0 None None None 2020-11-06 10:31:53 UTC
Red Hat Product Errata RHEA-2020:3246 0 None None None 2020-08-04 13:28:24 UTC
oVirt gerrit 108984 0 ovirt-4.3 ABANDONED test build including vdsm-hook-nestedvt package 2021-01-21 15:49:39 UTC

Description Yaniv Liberman 2020-01-23 17:09:03 UTC
Description of problem:
The vdsm-hook-nestedvt package is required for nested virtualization and is not included in the base RHVH image, meaning that it has to be installed manually and gets wiped after every upgrade.

Version-Release number of selected component (if applicable):
4.3.7 (in this particular case, but applicable to every version)

How reproducible:
100%

Additional info:
I see no reason not adding this package to the base RHVH image as it's required for nested virtualization and shouldn't impact anything else. Upgrades can be relatively frequent and having to repeat the process of reinstalling the package, which requires rebooting the host afterwards, seems nonsensical.

Comment 1 RHEL Program Management 2020-01-29 06:54:29 UTC
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 2 Sandro Bonazzola 2020-02-25 09:38:33 UTC
(In reply to Yaniv Liberman from comment #0)
> Description of problem:
> The vdsm-hook-nestedvt package is required for nested virtualization and is
> not included in the base RHVH image, meaning that it has to be installed
> manually and gets wiped after every upgrade.

Please be sure to install it using yum.
package should be persisted on upgrade.

Comment 3 Yaniv Liberman 2020-02-25 12:39:02 UTC
I asked Denis to test it next time he upgrades the RHV-TLV RHVH hosts.

We'll keep you posted.

Thanks!

Comment 4 Denis Prezhevalsky 2020-02-25 16:09:17 UTC
Sandro,
Hosts are been updated with rhvh-4.3.8.1-0.20200126.0 and vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm has been manually installed via yum.
Will let you know if it will persist after next version update.

Comment 5 Sandro Bonazzola 2020-02-26 08:33:37 UTC
Chen, can you test this on upgrade from 4.3.8 to 4.3.9?

Comment 6 cshao 2020-02-26 12:32:46 UTC
(In reply to Sandro Bonazzola from comment #5)
> Chen, can you test this on upgrade from 4.3.8 to 4.3.9?

Test steps:
1. Download vdsm-hook-nestedvt from brewweb(or from RHV agent channel).
2. yum install vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm in rhvh-4.3.8.1-0.20200126.0+1.
3. upgrade to 4.3.9
4. Check vdsm-hook-nestedvt

before upgrade:
rhvh-4.3.8.1-0.20200126.0+1.
# rpm -qa| grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

after upgrade to 4.3.9
rhvh-4.3.9.2-0.20200219.0+1

# imgbase  layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200219.0
 +- rhvh-4.3.9.2-0.20200219.0+1

# rpm -qa| grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch


Test result:
vdsm-hook-nestedvt can be persisted after upgrade.

Comment 7 Denis Prezhevalsky 2020-02-26 14:34:28 UTC
Awsesome!
Thanks!

Comment 8 Sandro Bonazzola 2020-03-03 16:30:54 UTC
Thanks, so closing not a bug.

Comment 9 Yaniv Liberman 2020-03-04 12:18:09 UTC
Clearing the NEEDINFO flag. Sorry if it re-opens it again, but I see no other way to clear the NEEDINFO flag and it spams my email. Just close it if it does.

I guess we'll approach you again if there are any problems in future upgrades.

Thanks!

Comment 10 Denis Prezhevalsky 2020-05-06 11:29:16 UTC
Seems like bug persist in newer version of image, cause after upgrade from 4.3.8 to latest 4.3.9 nested hook not persist any more.
rhvh image: rhvh-4.3.9.2-0.20200324.0

Comment 11 Sandro Bonazzola 2020-05-07 07:43:14 UTC
Denis, was the package installed using yum install or rpm -i ?
Because persistence is ensured only with yum install.

Comment 12 Sandro Bonazzola 2020-05-07 07:46:08 UTC
Nir, let's ensure the package is in RHV-H Optional channel and documentation is properly updated.

Comment 13 Michal Skrivanek 2020-05-07 08:07:15 UTC
see also bug 1759205

Comment 14 Denis Prezhevalsky 2020-05-07 12:18:54 UTC
Sandro,
Packages has been installed as: 
yum install vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm

To note, that there is only one repo enabled on the hosts:
[root@rhvh-01 ~]# yum repolist
Loaded plugins: imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade, versionlock
Excluding 4 updates due to versionlock (use "yum versionlock status" to show them)
repo id                                                                          repo name                                                                             status
rhel-7-server-rhvh-4-rpms/x86_64                                                 Red Hat Virtualization Host 7 (RPMs)                                                  463
repolist: 463

Comment 15 peyu 2020-05-08 03:17:36 UTC
I did not reproduce this issue in newer version of image(rhvh-4.3.9.2-0.20200324.0+1)

Test Steps:
1. Install rhvh-4.3.8.1-0.20200126.0+1
2. Download vdsm-hook-nestedvt from brewweb
3. yum install vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch.rpm in rhvh-4.3.8.1-0.20200126.0+1
4. Upgrade RHVH to rhvh-4.3.9.2-0.20200324.0+1
5. Check vdsm-hook-nestedvt


Test Results:

1) before upgrade:
# imgbase w
You are on rhvh-4.3.8.1-0.20200126.0+1
# rpm -qa| grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch


2) after upgrade to 4.3.9
# imgbase w
You are on rhvh-4.3.9.2-0.20200324.0+1

# imgbase layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200324.0
 +- rhvh-4.3.9.2-0.20200324.0+1

# rpm -qa| grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch


So vdsm-hook-nestedvt has been persisted after upgrade.

Comment 16 peyu 2020-05-09 06:04:28 UTC
According to Bug 1421746 and Bug 1759205, I did a few tests to verify whether the configuration of vdsm-hook-nestedvt was persisted after upgrade. I still haven't reproduced this bug.

Test1:
1. Install rhvh-4.3.8.1-0.20200126.0+1
2. Download vdsm-hook-nestedvt from brewweb
3. yum install vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch.rpm in rhvh-4.3.8.1-0.20200126.0+1
~~~~~~
# rpm -qa | grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch

# cat /sys/module/kvm*/parameters/nested
N
~~~~~~
4. reboot host
5. Check if nested virtualizationit is enabled
~~~~~~
# cat /sys/module/kvm*/parameters/nested
Y
~~~~~~
4. Upgrade RHVH to rhvh-4.3.9.2-0.20200324.0+1
5. Check vdsm-hook-nestedvt
~~~~~~
# imgbase layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200324.0
 +- rhvh-4.3.9.2-0.20200324.0+1

# rpm -qa | grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.46-1.el7ev.noarch

# cat /sys/module/kvm*/parameters/nested
Y
~~~~~~


Test2:
1. Install rhvh-4.3.8.1-0.20200126.0+1
2. Download vdsm-hook-nestedvt from brewweb
3. yum install vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm in rhvh-4.3.8.1-0.20200126.0+1
~~~~~~
# rpm -qa | grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

# cat /sys/module/kvm*/parameters/nested
N
~~~~~~
4. Add host to RHVM and enable nested in Edit Host -> Kernel -> Nested Virtualization
5. Reboot and check if nested virtualizationit is enabled
~~~~~~
# cat /sys/module/kvm*/parameters/nested
Y
~~~~~~
4. Upgrade RHVH to rhvh-4.3.9.2-0.20200324.0+1
5. Check vdsm-hook-nestedvt
~~~~~~
# imgbase w
You are on rhvh-4.3.9.2-0.20200324.0+1

# imgbase layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200324.0
 +- rhvh-4.3.9.2-0.20200324.0+1

# rpm -qa | grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

# cat /sys/module/kvm*/parameters/nested
Y
~~~~~~

Comment 17 Denis Prezhevalsky 2020-05-12 08:52:29 UTC
Pengshan,
We have this happened on 14 RHV-H hosts just after the update.

Comment 18 peyu 2020-05-12 09:07:55 UTC
Hi Denis,
Is there anything I need to add to my test steps? If so, could you give me some specific steps or version to reproduce this bug?

Comment 19 Denis Prezhevalsky 2020-05-12 09:12:36 UTC
Pengshan,
Perhaps, it is better to look into this specific case.
Please, reach me out (irc, gchat, email) and I'll provide you with access to one of the affected rhv-h host for further investigation.

Comment 20 Qin Yuan 2020-05-13 08:03:52 UTC
Checked two Envs which encountered the persisting problem, in summary, there are 3 main issues:

1. The persisted rpm can't be reinstalled on new layer if its dependencies provided by the image are updated to newer versions on new layer. See logs from one Env:

2020-05-05 13:57:51,980 [DEBUG] (MainThread) Running ['systemd-nspawn', '--uuid', '7e47a030ce90406b996c442369063d80', '--machine', 'rhvh-01.engineering.redhat.com', '-D', u'/tmp/mnt.F9l6q//', 'yum', 'install', '-y', '--noplugins', '/var/imgbased/persisted-rpms/nss-pam-ldapd-0.8.13-16.el7.x86_64.rpm', '/var/imgbased/persisted-rpms/nscd-2.17-260.el7_6.3.x86_64.rpm', '/var/imgbased/persisted-rpms/pam_krb5-2.4.8-6.el7.x86_64.rpm', '/var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm', '/var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm']
2020-05-05 13:57:57,824 [INFO] (MainThread) Failed to reinstall persisted RPMs!
2020-05-05 13:57:57,824 [INFO] (MainThread) Result: Spawning container rhvh-01.engineering.redhat.com on /tmp/mnt.F9l6q.
Press ^] three times within 1s to kill container.
Examining /var/imgbased/persisted-rpms/nss-pam-ldapd-0.8.13-16.el7.x86_64.rpm: nss-pam-ldapd-0.8.13-16.el7.x86_64
Marking /var/imgbased/persisted-rpms/nss-pam-ldapd-0.8.13-16.el7.x86_64.rpm to be installed
Examining /var/imgbased/persisted-rpms/nscd-2.17-260.el7_6.3.x86_64.rpm: nscd-2.17-260.el7_6.3.x86_64
Marking /var/imgbased/persisted-rpms/nscd-2.17-260.el7_6.3.x86_64.rpm to be installed
Examining /var/imgbased/persisted-rpms/pam_krb5-2.4.8-6.el7.x86_64.rpm: pam_krb5-2.4.8-6.el7.x86_64
Marking /var/imgbased/persisted-rpms/pam_krb5-2.4.8-6.el7.x86_64.rpm to be installed
Examining /var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm: vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch
Marking /var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm to be installed
Examining /var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm: vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch
Marking /var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package nscd.x86_64 0:2.17-260.el7_6.3 will be installed
--> Processing Dependency: glibc = 2.17-260.el7_6.3 for package: nscd-2.17-260.el7_6.3.x86_64
---> Package nss-pam-ldapd.x86_64 0:0.8.13-16.el7 will be installed
---> Package pam_krb5.x86_64 0:2.4.8-6.el7 will be installed
---> Package vdsm-hook-nestedvt.noarch 0:4.30.11-1.el7ev will be installed
---> Package vdsm-hook-nestedvt.noarch 0:4.30.40-1.el7ev will be installed
--> Finished Dependency Resolution
Error: Package: nscd-2.17-260.el7_6.3.x86_64 (/nscd-2.17-260.el7_6.3.x86_64)
           Requires: glibc = 2.17-260.el7_6.3
           Installed: glibc-2.17-307.el7.1.x86_64 (installed)
               glibc = 2.17-307.el7.1
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
Container rhvh-01.engineering.redhat.com failed with error code 1. 

As you can see, when reinstall the persisted nscd-2.17-260.el7_6.3.x86_64.rpm on new layer, it requires an old glibc version 2.17-260.el7_6.3, but the new layer provides 2.17-307.el7.1. 


2. All persisted rpms are installed in one yum install cmd, one package error will cause the whole yum install cmd to fail.

As you can see in the above log info, the dependency error of nscd-2.17-260.el7_6.3.x86_64 will prevent yum from installing other packages, like vdsm-hook-nestedvt.


3. If update a persisted rpm by running 'yum install/update newer-version-rpm', the newer-version-rpm can't be recorded in /var/imgbased/persisted-rpms/, when upgrade rhvh again, it's still the old rpm but not the newer-version-rpm that will be reinstalled on new rhvh layer. See infos on another Env:

$ sudo imgbase w
You are on rhvh-4.3.9.2-0.20200324.0+1

$ sudo imgbase layout
rhvh-4.3.8.1-0.20200126.0
 +- rhvh-4.3.8.1-0.20200126.0+1
rhvh-4.3.9.2-0.20200324.0
 +- rhvh-4.3.9.2-0.20200324.0+1

$ sudo rpm -qa |grep vdsm-hook-nestedvt
vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

$ sudo ls -al /var/imgbased/persisted-rpms
total 16
drwxr-xr-x. 2 root root 4096 Mar 17 13:20 .
dr-xr-x---. 4 root root 4096 Mar 17 13:20 ..
-rw-r--r--. 1 root root 4736 Mar 17 13:19 vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm

$ sudo cat /var/log/yum.log
Mar 17 13:20:13 Persisting: vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm
Mar 17 13:20:20 Installed: vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch
May 05 14:09:19 Installed: vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch
May 05 14:11:46 Installed: redhat-virtualization-host-image-update-4.3.9-20200324.0.el7_8.noarch
May 05 14:11:46 Erased: redhat-virtualization-host-image-update-placeholder-4.3.8-1.el7ev.noarch
May 06 15:12:38 Updated: vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch

$ sudo vi /var/log/imgbased.log
2020-05-05 14:09:08,660 [DEBUG] (MainThread) Running ['systemd-nspawn', '--uuid', '4e0aa2cda9f44a279de2a9b58ac05c81', '--machine', 'rhev-rdu06.rhevdev.lab.eng.rdu2.redhat.com', '-D', u'/tmp/mnt.GB5Tz//', 'yum', 'install', '-y', '--noplugins', '/var/imgbased/persisted-rpms/vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm']

As you can see, vdsm-hook-nestedvt-4.30.11-1.el7ev.noarch.rpm is the persisted rpm, it was reinstalled on new layer at 2020-05-05 14:09. vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch is the current installed version on new layer, it was installed at May 06 15:12:38 manually.
But there is no vdsm-hook-nestedvt-4.30.40-1.el7ev.noarch.rpm in /var/imgbased/persisted-rpms, this will cause the newer vdsm-hook-nestedvt version can't be reinstalled on the next new layer when upgrade rhvh again.


How to solve the above 3 issues?
1. For the dependencies issue, suggest users to manually install newer rpms according to the new denpendencies provided by new rhvh layer? 
2. For the all in one yum cmd issue, should we use --skip-broken?
3. For the not persisting updated rpm issue, maybe we could improve the yum plugin, which is imgbased-persist.py:
   
def pretrans_hook(conduit):
    ts = conduit.getTsInfo()
    if ts.installed:       <---> for persisting the updated rpm, need to check ts.updated. Also need to consider removing the old rpm in /var/imgbased/persisted-rpms.

Comment 21 Sandro Bonazzola 2020-05-13 11:45:24 UTC
We need to include it in RHV-H, it's now required by host-deploy flow when nested vt is enabled on the engine.
Let's add it to node and rhv-h kickstart

Comment 22 Sandro Bonazzola 2020-05-13 11:54:13 UTC
(In reply to Sandro Bonazzola from comment #21)
> We need to include it in RHV-H, it's now required by host-deploy flow when
> nested vt is enabled on the engine.
> Let's add it to node and rhv-h kickstart

it sorted out it was a user hook modification, back to original plan.

Comment 23 Nir Levy 2020-05-17 07:17:40 UTC
errata updated,
vdsm-4.40.16-1.el8ev now in RHEL-8-RHEV-S-4.4 ( mapped to rhel-8-server-rhvh-4-rpms repo)
product listing adjusted to include vdsm-hook-nestedvt only in whitelist.

Comment 24 Nir Levy 2020-05-17 07:24:18 UTC
correcting comment 23
RHEL-8-RHV-4.4-TOOLS

Comment 30 cshao 2020-05-25 07:55:06 UTC
Change QA Contact to Wei according #c21 & 23.

Comment 31 peyu 2020-05-25 08:32:21 UTC
Tested on redhat-virtualization-host-4.4.0-20200518.0.el8_2 using RHSM stage account,  the test failed.

Test Steps:
1. Install redhat-virtualization-host-4.4.0-20200518.0.el8_2
2. subscribe to RHSM
3. enable rhvh-4-for-rhel-8-x86_64-rpms repo
# subscription-manager repos --enable=rhvh-4-beta-for-rhel-8-x86_64-rpms

4. check the packages vdsm-hook-nestedvt
~~~~~~
# yum search vdsm-hook-nestedvt
Updating Subscription Management repositories.
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                              3.1  B/s |  10  B     00:03    
Errors during downloading metadata for repository 'rhel-8-for-x86_64-baseos-rpms':
  - Status code: 404 for https://cdn.redhat.com/content/dist/rhel8/8RedHatVirtualizationHost/x86_64/baseos/os/repodata/repomd.xml (IP: 23.1.44.251)
Error: Failed to download metadata for repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
~~~~~~


According to Bug 1779459, rhvh-4.4 cahnnel is not ready yet. So move bug status to "ASSIGNED".

Comment 32 Assaf 2020-06-01 14:38:48 UTC
This https://cdn.redhat.com/content/dist/rhel8 is a path to a GA release not a Beta release: /content/beta/layered/rhel8/x86_64/rhvh/4/os/repodata/repomd.xml

Comment 39 RHEL Program Management 2020-06-22 07:49:22 UTC
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 45 errata-xmlrpc 2020-08-04 13:27:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHV RHEL Host (ovirt-host) 4.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:3246

Comment 46 Olimp Bockowski 2020-11-06 10:39:44 UTC
It won't be fixed for 4.3.z, because 4.3 comes withing EUS


Note You need to log in before you can comment on or make changes to this bug.