Bug 1996555 - OpenStack 4.8 -> 4.9 upgrade is failing periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-openstack-upgrade
Summary: OpenStack 4.8 -> 4.9 upgrade is failing periodic-ci-openshift-release-master-...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.9.0
Assignee: Martin André
QA Contact: Jon Uriarte
URL:
Whiteboard:
: 2077270 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-23 08:21 UTC by Pierre Prinetti
Modified: 2022-09-21 15:17 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1995387
Environment:
job=periodic-ci-openshift-verification-tests-master-stable-4.9-upgrade-from-stable-4.8-openstack-ipi=all
Last Closed: 2022-09-21 15:17:51 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift release pull 21379 0 None None None 2021-08-27 06:03:41 UTC

Description Pierre Prinetti 2021-08-23 08:21:39 UTC 
Once Bug 1995387 has been fixed and the base image for the tests is available, the test started showing legit failures.

Job history (relevant jobs AFTER Aug 21): https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-openstack-upgrade

One example failure: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-openstack-upgrade/1429521411877113856

In particular, ETCD seems to not be healthy.
Comment 6 Pierre Prinetti 2021-08-31 12:23:56 UTC 
Tests seem to be still failing...
Comment 7 Martin André 2021-09-01 15:24:46 UTC 
We still need to investigate why this job is failing. Still high prio.
Comment 8 Vadim Rutkovsky 2021-09-06 15:33:02 UTC 
Analyzing this job - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-openstack-upgrade/1434468962166378496

PromeCIeus shows that:
* at 13:05 etcd commit duration and wal sync times on .163 node spiked
* at approx. the same time different node - .32 - shows increased network round trip time

Seems infra is responsible for this
Comment 9 Martin André 2021-09-30 12:00:02 UTC 
Wonder if https://github.com/openshift/machine-config-operator/pull/2782 wouldn't help there, cf https://bugzilla.redhat.com/show_bug.cgi?id=2002121.
Comment 10 ShiftStack Bugwatcher 2021-11-25 16:12:10 UTC 
Removing the Triaged keyword because:

* the QE automation assessment (flag qe_test_coverage) is missing
Comment 11 Martin André 2022-05-11 13:25:19 UTC 
*** Bug 2077270 has been marked as a duplicate of this bug. ***
Comment 12 Stephen Finucane 2022-09-21 15:17:51 UTC 
We've integrated support for scheduled CI tasks with jitter into upstream Kubernetes CI infra and shouldn't be seeing this anymore. Closing as CURRENTRELEASE. We can open new bugs if this pops up again.
Note You need to log in before you can comment on or make changes to this bug.