Bug 1369418 - [z-stream clone - 4.0.3] [InClusterUpgrade] Possible race condition with large amount of VMs in cluster
Summary: [z-stream clone - 4.0.3] [InClusterUpgrade] Possible race condition with larg...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.6.8
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: ovirt-4.0.3
: 4.0.3
Assignee: Arik
QA Contact: sefi litmanovich
URL:
Whiteboard:
Depends On: 1366786
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-23 11:17 UTC by rhev-integ
Modified: 2019-12-16 06:27 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Update of compatibility version of a cluster with many running VMs that are installed with guest-agent could lead to a deadlock that fails the update. Consequence: In some cases, such clusters could not be upgraded to a newer compatibility version. Fix: Prevent the deadlock in the database from happening. Result: Cluster with many running VMs installed with guest-agent can be upgraded to newer compatibility version.
Clone Of: 1366786
Environment:
Last Closed: 2016-09-15 08:06:42 UTC
oVirt Team: Virt
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2525531 0 None None None 2016-08-23 11:18:15 UTC
oVirt gerrit 62372 0 ovirt-engine-4.0 MERGED core: fix monitoring of guest agent nics 2016-08-23 11:18:15 UTC
oVirt gerrit 62392 0 ovirt-engine-3.6 MERGED core: Cluster update updates VMs only if necessary 2016-08-23 11:18:15 UTC
oVirt gerrit 62514 0 ovirt-engine-4.0 MERGED core: make VmBase comparable 2016-08-23 11:18:15 UTC
oVirt gerrit 62515 0 ovirt-engine-4.0 MERGED core: update cluster to query only static vm data 2016-08-23 11:18:15 UTC
oVirt gerrit 62516 0 ovirt-engine-4.0 MERGED core: determine the order of vm statistics updates 2016-08-23 11:18:15 UTC
oVirt gerrit 62517 0 ovirt-engine-4.0 MERGED core: determine the order of guest agent nic updates 2016-08-23 11:18:15 UTC
oVirt gerrit 62518 0 ovirt-engine-4.0 MERGED core: fix possible deadlock on update cluster version 2016-08-23 11:18:15 UTC
oVirt gerrit 62521 0 master MERGED core: make VmBase comparable 2016-08-23 11:18:15 UTC
oVirt gerrit 62522 0 master MERGED core: update cluster to query only static vm data 2016-08-23 11:18:15 UTC
oVirt gerrit 62523 0 master MERGED core: determine the order of vm statistics updates 2016-08-23 11:18:15 UTC
oVirt gerrit 62524 0 master MERGED core: determine the order of guest agent nic updates 2016-08-23 11:18:15 UTC
oVirt gerrit 62525 0 master MERGED core: fix possible deadlock on update cluster version 2016-08-23 11:18:15 UTC
oVirt gerrit 62637 0 ovirt-engine-3.6 MERGED core: fix monitoring of guest agent nics 2016-08-23 11:18:15 UTC

Comment 2 Sandro Bonazzola 2016-08-25 11:55:27 UTC 
Arik can you check target milestone? This bug is referenced in 4.0.3 changelog.
Comment 3 sefi litmanovich 2016-08-29 15:08:25 UTC 
Verified with rhevm-4.0.3-0.1.el7ev.noarch.

Had a cluster with 144 Vms running.
Set the cluster to InClusterUpgrade - this change didn't invoke update of vm's configuration.
Then changed cluster compatibility version from 3.6 to 4.0 (hosts were 4.0 all the time) and monitored the updateVm calls with tail on engine log and after it's done check in DB that indeed all the vms were updated.
Repeated this several times (each time setting the cluster compatibility back to 3.6 via DB and the vm's custom compatibility version in vm_static from '3.6' to null - a bit "cheating there").
Ran the upgrade for 5 times, no race has occurred.
Please advise if this test isn't sufficient.
Comment 4 sefi litmanovich 2016-08-29 15:09:13 UTC 
Didn't mention in comment 3 - but all the vms had rhevm-guest-agent running.
Comment 5 Arik 2016-08-30 08:00:20 UTC 
(In reply to sefi litmanovich from comment #3)
This test is good.
Note You need to log in before you can comment on or make changes to this bug.