Bug 1936859

Summary:	ovirt 4.4 -> 4.5 upgrade jobs are permafailing
Product:	OpenShift Container Platform	Reporter:	Vadim Rutkovsky <vrutkovs>
Component:	Installer	Assignee:	Gal Zaidman <gzaidman>
Installer sub component:	OpenShift on RHV	QA Contact:	Gal Zaidman <gzaidman>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	low
Priority:	medium	CC:	bretm, gzaidman, mstaeble
Version:	4.4
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:	operator conditions monitoring
Last Closed:	2021-07-27 22:51:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Vadim Rutkovsky 2021-03-09 11:21:02 UTC

test:
operator conditions monitoring 

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=operator+conditions+monitoring

4.4.z install on oVirt can't complete as monitoring can't rollout. This blocks 4.4.z -> 4.5 upgrade tests.

example job: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/1369211579001737216

level=info msg="Cluster operator monitoring Available is False with : "
level=info msg="Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack."
level=error msg="Cluster operator monitoring Degraded is True with UpdatingPrometheusK8SFailed: Failed to rollout the stack. Error: running task Updating Prometheus-k8s failed: waiting for Prometheus object changes failed: waiting for Prometheus: expected 2 replicas, updated 0 and available 0"
level=fatal msg="failed to initialize the cluster: Cluster operator monitoring is still updating"

Comment 1 Vadim Rutkovsky 2021-03-09 11:41:58 UTC

Also happens on 4.5 - https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.6-upgrade-from-stable-4.5-e2e-ovirt-upgrade/1369166292912705536

Seems resolved in 4.8/4.7/4.6

Comment 2 Gal Zaidman 2021-03-30 13:34:53 UTC

This is a CI only issue and is in progress
due to capacity constraints we will be revisiting this bug in the upcoming sprint

Comment 3 Douglas Schilling Landgraf 2021-04-20 14:34:36 UTC

Should be the same problem as 4.5 jobs. Please see: https://bugzilla.redhat.com/show_bug.cgi?id=1936857#c5
Moving to Gal.

Comment 5 Vadim Rutkovsky 2021-04-27 08:14:59 UTC

Not yet fixed - see https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade
* https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/1386606443997696000
* https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/1386244053321912320
* https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/1385881662205726720

Comment 6 Gal Zaidman 2021-04-28 07:04:56 UTC

Hi(In reply to Vadim Rutkovsky from comment #5)
> Not yet fixed - see
> https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-
> openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade
> *
> https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-
> openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/
> 1386606443997696000
> *
> https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-
> openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/
> 1386244053321912320
> *
> https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-
> openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/
> 1385881662205726720

Hi those are upgrade jobs and that is a different workflow.
I believe that the upgrade issue was resolved by https://github.com/openshift/release/pull/18091.
But anyway I think you can mark it as verified since 4.4 jobs are passing installation (hitting docker rate limiting on test cases but that is a different problem)

Comment 7 Gal Zaidman 2021-05-20 11:21:52 UTC

Moving to verified now

Comment 8 Vadim Rutkovsky 2021-05-20 12:46:22 UTC

>Hi those are upgrade jobs and that is a different workflow.

This is a 4.4 -> 4.5 upgrade job and it still fails:
* https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/1394940974563594240
* https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/1394578900389990400
* https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.5-upgrade-from-stable-4.4-e2e-ovirt-upgrade/1394216508506247168

If it needs to use a different workflow please update release configuration

Comment 9 Vadim Rutkovsky 2021-05-20 12:50:56 UTC

Although, lets create a new issue for that

Comment 15 errata-xmlrpc 2021-07-27 22:51:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438