Bug 1896181

Summary: [ovirt] install fails: due to terraform error "Cannot run VM. VM is being updated" on vm resource
Product: OpenShift Container Platform Reporter: W. Trevor King <wking>
Component: InstallerAssignee: Janos Bonic <jpasztor>
Installer sub component: OpenShift on RHV QA Contact: Michael Burman <mburman>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: low CC: emesika, hpopal, mburman
Version: 4.7   
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-10 10:35:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2082283    
Bug Blocks:    

Description W. Trevor King 2020-11-09 21:58:56 UTC
Occasionally in 4.7+ CI:

$ w3m -dump -cols 200 'https://search.ci.openshift.org/?search=VM+is+being+updated.*409+Conflict&maxAge=48h&context=1&type=build-log' | grep 'failures match' | sort 
pull-ci-openshift-installer-master-e2e-ovirt - 9 runs, 89% failed, 13% of failures match
release-openshift-ocp-installer-e2e-ovirt-4.7 - 12 runs, 67% failed, 13% of failures match

Example job [1]:

level=error
level=error msg=Error: Fault reason is "Operation Failed". Fault detail is "[Cannot run VM. VM is being updated.]". HTTP response code is "409". HTTP response message is "409 Conflict".
level=error
level=error msg=  on ../tmp/openshift-install-072387181/masters/main.tf line 1, in resource "ovirt_vm" "master":
level=error msg=   1: resource "ovirt_vm" "master" {
level=error
level=error
level=fatal msg=failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

Not sure if that's "file a ticket with the provider" or "installer's Terraform should be more robust about retries in the face of conflicts", but filing a bug so other folks who hit this failure mode can find whatever the plan ends up being.

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-ovirt-4.7/1325865204830965760

Comment 2 Gal Zaidman 2021-03-30 13:40:36 UTC
due to capacity constraints, we will be revisiting this bug in the upcoming sprint

Comment 3 Janos Bonic 2021-07-01 11:11:10 UTC
This will be resolved automatically when we switch to the new client library.

Comment 6 Janos Bonic 2022-06-21 07:08:57 UTC
Fixed in 2082283

Comment 7 Michael Burman 2022-06-21 08:29:03 UTC
(In reply to Janos Bonic from comment #6)
> Fixed in 2082283

Fixed in bz 2082283

Comment 8 Michael Burman 2022-06-23 12:52:32 UTC
The issue is gone.
Such terraform errors no longer visible in the installer.

Verified on - 4.11.0-0.nightly-2022-06-22-235234 and rhvm-4.5.1.2-0.11.el8ev.noarch

Comment 10 errata-xmlrpc 2022-08-10 10:35:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069