Bug 1369415 - [z-stream clone - 3.6.9] [InClusterUpgrade] Possible race condition with large amount of VMs in cluster
Summary: [z-stream clone - 3.6.9] [InClusterUpgrade] Possible race condition with larg...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.6.8
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: ovirt-3.6.9
: ---
Assignee: Arik
QA Contact: sefi litmanovich
URL:
Whiteboard:
Depends On: 1366786
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-23 11:09 UTC by rhev-integ
Modified: 2022-07-09 08:22 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the update of the compatibility version of a cluster with many running virtual machines that are installed with the guest-agent caused a deadlock that caused the update to fail. In some cases, these clusters could not be upgraded to a newer compatibility version. Now, the deadlock in the database has been prevented so that a cluster with many running virtual machines that are installed with the guest-agent can be upgraded to newer compatibility version.
Clone Of: 1366786
Environment:
Last Closed: 2016-09-21 18:06:00 UTC
oVirt Team: Virt
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-47350 0 None None None 2022-07-09 08:22:04 UTC
Red Hat Knowledge Base (Solution) 2525531 0 None None None 2016-08-23 11:09:35 UTC
Red Hat Product Errata RHSA-2016:1929 0 normal SHIPPED_LIVE Moderate: Red Hat Virtualization Manager (RHV) bug fix 3.6.9 2016-09-21 21:57:10 UTC
oVirt gerrit 62372 0 ovirt-engine-4.0 MERGED core: fix monitoring of guest agent nics 2016-08-23 11:09:35 UTC
oVirt gerrit 62392 0 ovirt-engine-3.6 MERGED core: Cluster update updates VMs only if necessary 2016-08-23 11:09:35 UTC
oVirt gerrit 62514 0 ovirt-engine-4.0 MERGED core: make VmBase comparable 2016-08-23 11:09:35 UTC
oVirt gerrit 62515 0 ovirt-engine-4.0 MERGED core: update cluster to query only static vm data 2016-08-23 11:09:35 UTC
oVirt gerrit 62516 0 ovirt-engine-4.0 MERGED core: determine the order of vm statistics updates 2016-08-23 11:09:35 UTC
oVirt gerrit 62517 0 ovirt-engine-4.0 MERGED core: determine the order of guest agent nic updates 2016-08-23 11:09:35 UTC
oVirt gerrit 62518 0 ovirt-engine-4.0 MERGED core: fix possible deadlock on update cluster version 2016-08-23 11:09:35 UTC
oVirt gerrit 62521 0 master MERGED core: make VmBase comparable 2016-08-23 11:09:35 UTC
oVirt gerrit 62522 0 master MERGED core: update cluster to query only static vm data 2016-08-23 11:09:35 UTC
oVirt gerrit 62523 0 master MERGED core: determine the order of vm statistics updates 2016-08-23 11:09:35 UTC
oVirt gerrit 62524 0 master MERGED core: determine the order of guest agent nic updates 2016-08-23 11:09:35 UTC
oVirt gerrit 62525 0 master MERGED core: fix possible deadlock on update cluster version 2016-08-23 11:09:35 UTC
oVirt gerrit 62637 0 ovirt-engine-3.6 MERGED core: fix monitoring of guest agent nics 2016-08-23 11:09:35 UTC

Comment 2 sefi litmanovich 2016-08-29 17:02:24 UTC 
Verified with rhevm-3.6.9-0.1.el6.noarch.

Had a cluster with 126 Vms running.
Changed cluster compatibility version from 3.5 to 3.6 (hosts were 3.6 all the time) and monitored the updateVm calls with tail on engine log.
Repeated this several times (each time setting the cluster compatibility back to 3.5 via DB.
Ran the upgrade for 5 times, no race has occurred.
Please advise if this test isn't sufficient.
Comment 3 sefi litmanovich 2016-08-30 09:04:47 UTC 
I see this test was approved in the 4.0.4 version of it. 10x.
Comment 5 errata-xmlrpc 2016-09-21 18:06:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-1929.html
Comment 6 Frank DeLorey 2016-10-26 20:12:50 UTC 
I have a customer running 3.6.9 that just hit this in a cluster that only has 30 VMs. I am gathering all the data and will open a new BZ.

Regards,

Frank
Comment 8 Frank DeLorey 2016-10-27 10:06:40 UTC 
This is happening on every attempt to upgrade the cluster. I will grab the engine log and post the related errors into this BZ.

Frank
Comment 9 Michal Skrivanek 2016-10-29 06:14:34 UTC 
(In reply to Frank DeLorey from comment #8)
> This is happening on every attempt to upgrade the cluster. I will grab the
> engine log and post the related errors into this BZ.
> 
> Frank

It is a different issue, tracked in bug 1389764 now
Note You need to log in before you can comment on or make changes to this bug.