Bug 1718873 - Hosted Engine install removes rhvm-appliance, therefore purges system of RHV at end of hosted-engine install
Summary: Hosted Engine install removes rhvm-appliance, therefore purges system of RHV ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhvm-appliance
Version: 4.3.1
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: ovirt-4.4.1
: 4.4.1
Assignee: Sandro Bonazzola
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-10 13:19 UTC by Jason Woods
Modified: 2020-08-04 16:21 UTC (History)
9 users (show)

Fixed In Version: ovirt-engine-appliance-4.4.1_rc6
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-04 16:21:42 UTC
oVirt Team: Integration
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2020:3315 0 None None None 2020-08-04 16:21:51 UTC
oVirt gerrit 109914 0 master MERGED packaging: spec: don't require ovirt-hosted-engine-setup 2020-11-09 20:04:59 UTC

Description Jason Woods 2019-06-10 13:19:35 UTC
Description of problem:
If rhvm-appliance RPM is installed before any other ovirt/RHV packages on a RHEL 7.6 system, rhvm-appliance has dependencies that will install all ovirt packages.
Hosted Engine install will eventually remove rhvm-appliance, therefore purges system of RHV at end of hosted-engine install.

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup 
2.3.7-1

rhvm-appliance
4.3-20190506.0

ovirt-ansible-hosted-engine-setup
1.0.17-1

ovirt-host
4.3.2-1

How reproducible:
100%

Steps to Reproduce:
1. Install RHEL 7.6 OS on a system
2. yum install rhvm-appliance
3. hosted-engine --deploy
4. [hosted-engine deploy process removes rhvm-appliance, which removes all RPMs brought in as dependencies, therefore purging the system of RHV]
5. hosted-engine --deploy fails with error: "ERROR! the playbook: /usr/share/ovirt-hosted-engine-steup/ansible/trigger_role.yml could not be found"
6. RHEL system no longer has any RPMs for RHV, and RHV HE being built fails and is removed.

Actual results:
HE fails to install
RHV RPMs purged from system

Expected results:
HE appliance installed

Workaround:
Install rhvm-appliance RPM after ovirt-hosted-engine-setup RPM and dependencies.
This order of install requirement is not documented that I could find, yet this is critical to installing RHVM HE on a RHEL Host.

Additional info:
Questions I have:
1. Why does the current rhvm-appliance have dependencies? Should it not be a standalone RPM? I believe it was in the past, which is how I used to install the rhvm-appliance package just to get the OVA out of it. If there were no dependencies in rhvm-appliance, I would have then installed ovirt-hosted-engine-setup as a second step, as I have done in the past. The reason I installed rhvm-appliance first is I have seen issues where there was not enough free space to install rhvm-appliance RPM, and I would rather know this before HE deploy, not during it. This lack of space was found at the current customer, which I fixed by expanding /var to 10GB then re-ran yum install rhvm-appliance.
2. Why does hosted-engine remove the rhvm-appliance RPM?
2a. If the rhvm-appliance was already installed before running hosted-engine deploy, why is it removed during hosted-engine?
2b. Why would the rhvm-appliance RPM need to be removed anyway? What is the point of this except to save a little over 1 GB? I understand this a little, but if the combination of rhvm-appliance having ovirt RPM dependencies and hosted-engine removing this package means hosted engine deploy fails, why do this? Let customer/installer decide to purge, or add prompt for this as one of the questions during install.

Comment 1 Simone Tiraboschi 2019-06-10 14:38:10 UTC
(In reply to Jason Woods from comment #0)
> 4. [hosted-engine deploy process removes rhvm-appliance, which removes all
> RPMs brought in as dependencies, therefore purging the system of RHV]

This is not the expected behavior:
we have:
  - name: Remove ovirt-engine-appliance rpm
    yum:
      name: ovirt-engine-appliance
      state: absent
    when: he_remove_appliance_rpm|bool

without specifying any value for autoremove parameter and according to 
https://docs.ansible.com/ansible/latest/modules/yum_module.html
its default value is no so ansible shouldn't require yum to remove also ovirt-engine-appliance dependencies. 

Can you please attach logs?

> 2. Why does hosted-engine remove the rhvm-appliance RPM?

it comes from:
https://bugzilla.redhat.com/1676825

> 2b. Why would the rhvm-appliance RPM need to be removed anyway? What is the
> point of this except to save a little over 1 GB? I understand this a little,
> but if the combination of rhvm-appliance having ovirt RPM dependencies and
> hosted-engine removing this package means hosted engine deploy fails, why do
> this? Let customer/installer decide to purge, or add prompt for this as one
> of the questions during install.

the main reason is that the appliance is shipped as an rpm so, keeping it, on each engine release you are going to upgrade a 1 GB rpm absolutely not needed anymore.

Comment 2 Simone Tiraboschi 2019-06-10 14:43:57 UTC
By the way we already have:
https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/blob/master/tasks/install_packages.yml#L2

that ensures that the packages is there if not (and in your case it was there and it has been installed as a ovirt-engine-appliance dendency)
I'm wondering if there is any way to force ansible/yum to flag ovirt-hosted-engine-setup as a user installed packages instead of a dependency of ovirt-engine-appliance if it was already there.

Comment 3 Jason Woods 2019-06-10 15:02:29 UTC
I am unable to attach logs, as the environment restricts this.

It should be easy to reproduce this problem using the procedure I listed, as it was 100% for us. This environment had used a Satellite repo. The Satellite refreshed every night, and the repos were directly from Satellite library (no CVs, no lifecycles in use).

Yum showed the removal of the RPMs as instructed by a removal of rhvm-appliance RPM. This was instructed from an Ansible playbook. Logs are no longer existing, so unable to show exact lines. This is my re-typing the lines from the error, from a text grab we did at the time:
-----
[ ERROR ] fatal: [localhost]: FAILED! => {"reason": "Could not find or access '/usr/share/ovirt-hosted-engine-setup/ansible/fetch_engine_logs.yml' on the Ansible Controller."}
[ ERROR ]  [WARNING]: Unable to find 'hooks/after_setup' in expected paths (use -vvvvv to
[ ERROR ] see paths)
[ ERROR ] Failed to execute state 'Closing up': Failed executing ansible-playbook
[ INFO  ] State: Clean up
[ INFO  ] Cleaning temporary resources
[ ERROR ] ERROR! the playbook: /usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml could not be found

[ ERROR ] Failed to execute state 'Clean up': Failed executing ansible-playbook
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-$DATATIME-evij02.log
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-$DATETIME-evij02.log
--
ERROR! the playbook: /usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml could not be found
-----

Comment 4 Simone Tiraboschi 2019-06-10 15:03:26 UTC
It seams that a plain 'yum install' is not updating package install reason from dep -> user if the package is already there as a dependency.
'yum reinstall' instead is but it's not directly available in yum ansible module.

Small example with another rpm:


[root@c76he20190521h2 ~]# yumdb info libsoup | grep reason
     reason = dep
[root@c76he20190521h2 ~]# yum install -y libsoup
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade
This system is not registered with an entitlement server. You can use subscription-manager to register.
 * base: mirrors.prometeus.net
 * epel: mirror.infonline.de
 * extras: mirrors.prometeus.net
 * ovirt-4.3: ftp.nluug.nl
 * ovirt-4.3-epel: mirror.infonline.de
 * updates: mirrors.prometeus.net
Package libsoup-2.62.2-2.el7.x86_64 already installed and latest version
Nothing to do
Uploading Enabled Repositories Report
Cannot upload enabled repos report, is this client registered?
[root@c76he20190521h2 ~]# yumdb info libsoup | grep reason
     reason = dep
[root@c76he20190521h2 ~]# yum reinstall -y libsoup
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
 * base: mirrors.prometeus.net
 * epel: mirror.de.leaseweb.net
 * extras: mirrors.prometeus.net
 * ovirt-4.3: ftp.nluug.nl
 * ovirt-4.3-epel: mirror.de.leaseweb.net
 * updates: mirrors.prometeus.net
Resolving Dependencies
--> Running transaction check
---> Package libsoup.x86_64 0:2.62.2-2.el7 will be reinstalled
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================================================================================================================
 Package                                                             Arch                                                               Version                                                                    Repository                                                        Size
==========================================================================================================================================================================================================================================================================================
Reinstalling:
 libsoup                                                             x86_64                                                             2.62.2-2.el7                                                               base                                                             411 k

Transaction Summary
==========================================================================================================================================================================================================================================================================================
Reinstall  1 Package

Total download size: 411 k
Installed size: 1.5 M
Downloading packages:
libsoup-2.62.2-2.el7.x86_64.rpm                                                                                                                                                                                                                                    | 411 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libsoup-2.62.2-2.el7.x86_64                                                                                                                                                                                                                                            1/1 
Uploading Package Profile
Cannot upload package profile. Is this client registered?
  Verifying  : libsoup-2.62.2-2.el7.x86_64                                                                                                                                                                                                                                            1/1 

Installed:
  libsoup.x86_64 0:2.62.2-2.el7                                                                                                                                                                                                                                                           

Complete!
Uploading Enabled Repositories Report
Cannot upload enabled repos report, is this client registered?
[root@c76he20190521h2 ~]# yumdb info libsoup | grep reason
     reason = user

Comment 5 Jason Woods 2019-06-10 15:07:39 UTC
My question comes back to:
Why does rhvm-appliance have any dependencies?

This should be a standalone package. If it had not installed all of the ovirt packages as dependencies, I would have installed them myself, and the removal of the rhvm-appliance would have had no consequence to the RHV HE installation.

Comment 6 Simone Tiraboschi 2019-06-10 15:16:28 UTC
Yes, this is also true.
So I see two options:

1. run
  yumdb set reason user ovirt-hosted-engine-setup
as a command in the ovirt-ansible-hosted-engine-setup role to avoid getting it deleted ans an orphan dependency by yum autoremove in the future.

2. Remove ovirt-hosted-engine-setup from the dependency of ovirt-engine-appliance so that the user has to manually install both getting it correctly flagged as a user installed packages.

Yuval, what do you think?

Comment 7 Jason Woods 2019-06-10 16:27:00 UTC
I just ran another RHVM HE deploy. This time I did not install rhvm-appliance at all, and let the deploy install as needed. The RPM was installed, then near the end of the deploy, something automatically removed rhvm-appliance.

I am ok with the removal of the rhvm-appliance RPM. I am just wanting a fix so removing it doesn't auto-purge RHVM from the host.

Having had this happen at a customer I was engaged with wasn't great, but I imagine a customer trying to install RHVM HE without a consultant involved, this might really set them back. This would also have high probability of creating a support ticket.

Comment 8 Marina Kalinin 2019-06-12 20:02:46 UTC
Changing it to downstream bug, as agreed, so that we can customer tickets to it.

Comment 9 Yuval Turgeman 2019-06-16 07:47:20 UTC
(In reply to Simone Tiraboschi from comment #6)
> Yes, this is also true.
> So I see two options:
> 
> 1. run
>   yumdb set reason user ovirt-hosted-engine-setup
> as a command in the ovirt-ansible-hosted-engine-setup role to avoid getting
> it deleted ans an orphan dependency by yum autoremove in the future.
> 
> 2. Remove ovirt-hosted-engine-setup from the dependency of
> ovirt-engine-appliance so that the user has to manually install both getting
> it correctly flagged as a user installed packages.
> 
> Yuval, what do you think?

I'm fine with dropping the dependency, in theory a user can install the rpm and import the ova

Comment 10 Nikolai Sednev 2019-07-10 08:49:39 UTC
(In reply to Yuval Turgeman from comment #9)
> (In reply to Simone Tiraboschi from comment #6)
> > Yes, this is also true.
> > So I see two options:
> > 
> > 1. run
> >   yumdb set reason user ovirt-hosted-engine-setup
> > as a command in the ovirt-ansible-hosted-engine-setup role to avoid getting
> > it deleted ans an orphan dependency by yum autoremove in the future.
> > 
> > 2. Remove ovirt-hosted-engine-setup from the dependency of
> > ovirt-engine-appliance so that the user has to manually install both getting
> > it correctly flagged as a user installed packages.
> > 
> > Yuval, what do you think?
> 
> I'm fine with dropping the dependency, in theory a user can install the rpm
> and import the ova

There are 3 options for the deployment:
1.Just install ovirt-hosted-engine-setup and then deploy, while during deployment customer asked to allow the deployment to install on its own the appliance.
2.Manually install appliance before or after installing ovirt-hosted-engine-setup. 
3.Manually install get ova image of the appliance before or after installing ovirt-hosted-engine-setup.

In all cases, after deployment, appliance should be wiped off, that was once a requirement from the customers, in order to prevent unnecessary consumption of space on hosts.

Comment 11 Sandro Bonazzola 2019-07-11 07:02:18 UTC
Re-targeting to 4.3.6 not being identified as blocker for 4.3.5.

Comment 13 Yedidyah Bar David 2019-12-23 07:16:50 UTC
IIUC the reason that the appliance requires hosted-engine-setup is that its a conf file 10-appliance.conf under a directory that is owned by hosted-engine-setup, /etc/ovirt-hosted-engine . Is it really that important? If we remove this requirement and users install the appliance, they get this file owned by the appliance package, inside a directory that is not owned by any package. Is that a problem? If not, I think that's a reasonable solution.

Re whether we should remove the appliance at all: IMO if it was not installed by the deploy script, it should not be removed by it. As a user, I'd find it quite annoying if some software silently removed an RPM that I installed by myself, just because it thinks I do not need it anymore.

Comment 14 Sandro Bonazzola 2020-01-07 11:12:20 UTC
(In reply to Yedidyah Bar David from comment #13)
> IIUC the reason that the appliance requires hosted-engine-setup is that its
> a conf file 10-appliance.conf under a directory that is owned by
> hosted-engine-setup, /etc/ovirt-hosted-engine . Is it really that important?
> If we remove this requirement and users install the appliance, they get this
> file owned by the appliance package, inside a directory that is not owned by
> any package. Is that a problem? If not, I think that's a reasonable solution.

I agree, let's drop the dependency.

> Re whether we should remove the appliance at all: IMO if it was not
> installed by the deploy script, it should not be removed by it. As a user,
> I'd find it quite annoying if some software silently removed an RPM that I
> installed by myself, just because it thinks I do not need it anymore.

I tend to agree but maybe can be tracked as a separate issue.
Moving this bug to appliance for removing the dependency.

Comment 17 Nikolai Sednev 2020-07-08 12:24:21 UTC
Works for me on latest components:
rhvm-appliance-4.4-20200707.0.el8ev.ova
ovirt-hosted-engine-setup-2.4.5-1.el8ev.noarch
ovirt-hosted-engine-ha-2.4.4-1.el8ev.noarch
Linux 4.18.0-193.12.1.el8_2.x86_64 #1 SMP Thu Jul 2 15:48:14 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.2 (Ootpa)

Reported issue no longer exists and dependency had been removed.
New appliance has no ovirt-hosted-engine-setup dependency:

puma18 ~]# rpm -qpR /root/rhvm-appliance-4.4-20200707.0.el8ev.x86_64.rpm
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1

Older appliances like rhvm-appliance-4.4-20200604.0.el8ev.x86_64.rpm had required ovirt-hosted-engine-setup as a dependency:

alma07 ~]# rpm -qpR /var/cache/dnf/rhv-4.4.1-8c1afb5569f744b5/packages/rhvm-appliance-4.4-20200604.0.el8ev.x86_64.rpm
warning: /var/cache/dnf/rhv-4.4.1-8c1afb5569f744b5/packages/rhvm-appliance-4.4-20200604.0.el8ev.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
ovirt-hosted-engine-setup
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1


Deployment went just fine with the new rhvm-appliance-4.4-20200707.0.el8ev.x86_64 appliance.

Comment 19 errata-xmlrpc 2020-08-04 16:21:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RHV Appliance (rhvm-appliance) 4.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:3315


Note You need to log in before you can comment on or make changes to this bug.