Bug 1380498

Summary:	[RFE] host upgrade should upgrade the whole host, not specific packages
Product:	[oVirt] ovirt-engine	Reporter:	Marina Kalinin <mkalinin>
Component:	BLL.Infra	Assignee:	Ondra Machacek <omachace>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Pavol Brilla <pbrilla>
Severity:	medium	Docs Contact:
Priority:	high
Version:	---	CC:	bgraveno, bugs, lsurette, lsvaty, mgoldboi, michal.skrivanek, mperina, pstehlik, rbalakri, Rhev-m-bugs, srevivo, yamakasi.014, ykaul, ylavi
Target Milestone:	ovirt-4.2.0	Keywords:	FutureFeature
Target Release:	4.2.0	Flags:	rule-engine: ovirt-4.2+ lsvaty: testing_plan_complete+ mgoldboi: planning_ack+ mperina: devel_ack+ lsvaty: testing_ack+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:	Previously, the host upgrade process upgraded just a few relevant virtualization packages that were specified with the following variables: - PackageNamesForCheckUpdate - UserPackageNamesForCheckUpdate - OvirtNodePackageNamesForCheckUpdate Now the upgrade process updates all packages and these variables have been removed as a result. Additionally, an option has been added to choose whether to reboot the host after the upgrade. By default it will reboot the host.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-02-12 10:12:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1460609
Bug Blocks:	1490386

Description Marina Kalinin 2016-09-29 18:56:20 UTC

Host upgrade should upgrade the whole host, not specific packages.
If it will upgrade specific packages only, we will end up with an unsupported/untested configuration of the packages, which may lead to unexpected behavior of the system. As an example: very simple scenario of failing live migration due to package mis-match, which is a very common issue, when the customers upgrade selected packages only.

In 3.6 we upgrade vdsm only, in 4.0.4 we upgrade selected packages (see bz#1344020 ). This is incorrect and should be fixed ASAP.

This will bring to scenarios where host kernel would be kernel from RHEL 7.2 and qemu-kvm and libvirt and other packages would be from RHEL 7.3 or later. Those scenarios have never been tested by RHEV QE and it is irresponsible from our software putting the customer in an unsupported configuration.

Also, based on my experience, when the customers click the upgrade button, they expect the whole host to be upgraded, same way they do with RHEV-H. They want to get all the latest updates and all the latest CVE available. Not partial update of specific packages.

Comment 1 Marina Kalinin 2016-09-29 19:12:59 UTC

As a mean term, if we cannot fix this bug right away, we should add a tooltip or information message or other type of notifications, that right now only "THIS" list of packages are going to be updated, kernel etc not included. If you are interested to upgrade the whole host, run `yum update` on the host.
I would also mention that this is unsupported configuration, as we were told multiple times.

Comment 2 Martin Perina 2016-09-30 13:15:01 UTC

(In reply to Marina from comment #1)
> As a mean term, if we cannot fix this bug right away, we should add a
> tooltip or information message or other type of notifications, that right
> now only "THIS" list of packages are going to be updated, kernel etc not
> included. If you are interested to upgrade the whole host, run `yum update`
> on the host.
> I would also mention that this is unsupported configuration, as we were told
> multiple times.

Even now for each host, if we found some available upgrades, we send an event (and also add log message to engine.log):

  Host ${VdsName} has available updates: ${Packages}.

This event (log) is created after each check for host upgrade (by default once a day).

Comment 3 Marina Kalinin 2016-10-03 16:09:41 UTC

(In reply to Martin Perina from comment #2)
> (In reply to Marina from comment #1)
> > As a mean term, if we cannot fix this bug right away, we should add a
> > tooltip or information message or other type of notifications, that right
> > now only "THIS" list of packages are going to be updated, kernel etc not
> > included. If you are interested to upgrade the whole host, run `yum update`
> > on the host.
> > I would also mention that this is unsupported configuration, as we were told
> > multiple times.
> 
> Even now for each host, if we found some available upgrades, we send an
> event (and also add log message to engine.log):
> 
>   Host ${VdsName} has available updates: ${Packages}.
> 
> This event (log) is created after each check for host upgrade (by default
> once a day).

Martin, does it check for vdsm required packages only or all?

Comment 4 Martin Perina 2016-10-04 08:42:09 UTC

(In reply to Marina from comment #3)
> (In reply to Martin Perina from comment #2)
> > (In reply to Marina from comment #1)
> > > As a mean term, if we cannot fix this bug right away, we should add a
> > > tooltip or information message or other type of notifications, that right
> > > now only "THIS" list of packages are going to be updated, kernel etc not
> > > included. If you are interested to upgrade the whole host, run `yum update`
> > > on the host.
> > > I would also mention that this is unsupported configuration, as we were told
> > > multiple times.
> > 
> > Even now for each host, if we found some available upgrades, we send an
> > event (and also add log message to engine.log):
> > 
> >   Host ${VdsName} has available updates: ${Packages}.
> > 
> > This event (log) is created after each check for host upgrade (by default
> > once a day).
> 
> Martin, does it check for vdsm required packages only or all?

Is checks for all packages that are defined in following config values:

1. RHEL hosts
    PackageNamesForCheckUpdate
      - cannot be changed by users, contains following packages:
          ioprocess
          mom
          libvirt-client
          libvirt-daemon-config-nwfilter
          libvirt-daemon-kvm
          libvirt-lock-sanlock
          libvirt-python
          lvm2
          ovirt-imageio-common
          ovirt-imageio-daemon
          ovirt-vmconsole
          ovirt-vmconsole-host
          python-ioprocess
          qemu-kvm
          qemu-img
          sanlock
          vdsm
          vdsm-cli

    UserPackageNamesForCheckUpdate
      - editable using engine-config, by default empty

2. RHVH hosts
    OvirtNodePackageNamesForCheckUpdate
      - cannot be changed by users, contains following packages:
          ovirt-node-ng-image-update

We check for upgrade of all packages listed in above configuration values.

Comment 5 Yamakasi 2017-01-23 12:53:39 UTC

I agree that there should be more update options as I posted to the mailinglist:

http://lists.ovirt.org/pipermail/users/2017-January/079062.html

- Only oVirt packages
or
- Yum update all packages


Selectbox:
- Reboot when upgrade finished


This with/in the maintenance mode will save a lot of time.

Comment 6 Yaniv Kaul 2017-06-07 19:23:20 UTC

(In reply to Yamakasi from comment #5)
> I agree that there should be more update options as I posted to the
> mailinglist:
> 
> http://lists.ovirt.org/pipermail/users/2017-January/079062.html
> 
> - Only oVirt packages
> or
> - Yum update all packages
> 
> 
> Selectbox:
> - Reboot when upgrade finished
> 
> 
> This with/in the maintenance mode will save a lot of time.

Makes sense to me.
Martin - can you scope it?

Comment 7 Yaniv Lavi 2017-06-11 12:42:11 UTC

Why are we no maintaining the packages that need to be updated in the host?
It will make it much easier to diff between type and archs. Can we move that list to the host and only trigger according to that list?

Comment 8 Martin Perina 2017-06-12 08:44:16 UTC

There are no customer tickets attached, so moving upstream

Comment 9 Martin Perina 2017-06-12 08:48:38 UTC

(In reply to Yaniv Kaul from comment #6)
> (In reply to Yamakasi from comment #5)
> > I agree that there should be more update options as I posted to the
> > mailinglist:
> > 
> > http://lists.ovirt.org/pipermail/users/2017-January/079062.html
> > 
> > - Only oVirt packages
> > or
> > - Yum update all packages
> > 
> > 
> > Selectbox:
> > - Reboot when upgrade finished
> > 
> > 
> > This with/in the maintenance mode will save a lot of time.
> 
> Makes sense to me.
> Martin - can you scope it?

It works for me, we will have 2 upgrade options (with user requested reboot of the host at the end):

1. Upgrade only ovirt-host package, which should upgrade all oVirt dependencies on the host (see BZ1460609)
2. Perform all available package upgrades

Comment 10 Moran Goldboim 2017-06-18 11:14:10 UTC

(In reply to Martin Perina from comment #9)
> (In reply to Yaniv Kaul from comment #6)
> > (In reply to Yamakasi from comment #5)
> > > I agree that there should be more update options as I posted to the
> > > mailinglist:
> > > 
> > > http://lists.ovirt.org/pipermail/users/2017-January/079062.html
> > > 
> > > - Only oVirt packages
> > > or
> > > - Yum update all packages
> > > 
> > > 
> > > Selectbox:
> > > - Reboot when upgrade finished
> > > 
> > > 
> > > This with/in the maintenance mode will save a lot of time.
> > 
> > Makes sense to me.
> > Martin - can you scope it?
> 
> It works for me, we will have 2 upgrade options (with user requested reboot
> of the host at the end):
> 
> 1. Upgrade only ovirt-host package, which should upgrade all oVirt
> dependencies on the host (see BZ1460609)
> 2. Perform all available package upgrades

We should strive for having an homogeneous status in a cluster with regards to upgrade policy. hence it should be a cluster level configuration rather than a host one. does it makes sense?

Comment 11 Oved Ourfali 2017-06-18 11:28:11 UTC

> > 
> > 1. Upgrade only ovirt-host package, which should upgrade all oVirt
> > dependencies on the host (see BZ1460609)
> > 2. Perform all available package upgrades
> 
> We should strive for having an homogeneous status in a cluster with regards
> to upgrade policy. hence it should be a cluster level configuration rather
> than a host one. does it makes sense?

Today it is system-wide.
I didn't see any requirement for a more fine-grained settings.
I vote to either leave this as-is for now, or to upgrade everything (and not only ovirt dependencies).

Comment 12 Martin Perina 2017-06-19 13:15:45 UTC

(In reply to Oved Ourfali from comment #11)
> > > 
> > > 1. Upgrade only ovirt-host package, which should upgrade all oVirt
> > > dependencies on the host (see BZ1460609)
> > > 2. Perform all available package upgrades
> > 
> > We should strive for having an homogeneous status in a cluster with regards
> > to upgrade policy. hence it should be a cluster level configuration rather
> > than a host one. does it makes sense?
> 
> Today it is system-wide.
> I didn't see any requirement for a more fine-grained settings.
> I vote to either leave this as-is for now, or to upgrade everything (and not
> only ovirt dependencies).

It makes sense to perform all available updates, but we also need to check if ovirt-host package is installed and if not then install it (as it's introduced for 4.2 and it will not be present on 4.0/4.1 hosts, so it won't be in the list of available updates for those hosts).

So yes, it makes sense to perform only "all available" updates (with the ability to perform host restart at the end) + an expection to ovirt-host package

Comment 13 Michal Skrivanek 2017-06-23 06:19:55 UTC

(In reply to Martin Perina from comment #12)

> ... 
> (with the ability to perform host restart at the end)
> ...

great. as long as that is the default option

Comment 14 Oved Ourfali 2017-06-23 06:49:33 UTC

(In reply to Michal Skrivanek from comment #13)
> (In reply to Martin Perina from comment #12)
> 
> > ... 
> > (with the ability to perform host restart at the end)
> > ...
> 
> great. as long as that is the default option

That will be a bigger change in behavior. 
We should let the user chose if to restart. Or give some warning around it.

Comment 15 Michal Skrivanek 2017-06-23 06:53:19 UTC

that's why I say "default". Sure they can decide not to. But it should not be the default choice. Let's finally fix the misconception introduced couple releases back that leads to unpredictable behavior after an update of a system package. For people who choose to apply specific updates and know what they are doing it's fine, but very few do that.

Comment 16 Martin Perina 2017-06-23 08:29:04 UTC

The plan is to have 'Restart host after upgrade' checkbox in the dialog, which by default will be checked, but user can uncheck it if needed.

Comment 17 Pavol Brilla 2018-01-25 13:32:52 UTC

Manually tested on 7.4 rhel beta 1 to beta 2 candidate + system updates
Tested on rhvh also

Comment 18 Sandro Bonazzola 2018-02-12 10:12:16 UTC

This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.