Bug 1998643 - e2e-metal-ipi-virtualmedia and e2e-metal-ipi-ovn-ipv6 are failing to install
Summary: e2e-metal-ipi-virtualmedia and e2e-metal-ipi-ovn-ipv6 are failing to install
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.9
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.9.0
Assignee: Arda Guclu
QA Contact: Raviv Bar-Tal
URL:
Whiteboard:
: 1999594 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-27 19:44 UTC by Stephen Benjamin
Modified: 2021-10-18 17:49 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
job=periodic-ci-openshift-release-master-nightly-4.9-e2e-metal-ipi-ovn-dualstack-local-gateway=all job=periodic-ci-openshift-release-master-nightly-4.9-e2e-metal-ipi-virtualmedia=all job=periodic-ci-openshift-release-master-nightly-4.9-e2e-metal-ipi-ovn-ipv6=all job=periodic-ci-openshift-release-master-nightly-4.9-e2e-metal-ipi-ovn-dualstack=all
Last Closed: 2021-10-18 17:49:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 5180 0 None None None 2021-09-01 06:23:15 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:49:58 UTC

Comment 1 Arda Guclu 2021-08-31 09:35:40 UTC
After investigating the problem with @afasano, we found out that the first bootstrap error date is 26th August 18:15 and 16:18;

https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/job-history/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-metal-ipi-ovn-ipv6?buildId=1431332397214863360
https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/job-history/gs/origin-ci-test/pr-logs/directory/pull-ci-openshift-oc-master-e2e-metal-ipi-ovn-ipv6

Checking the merged PR's after these times, most suspected PRs are in here;
https://amd64.ocp.releases.ci.openshift.org/releasestream/4.9.0-0.nightly/release/4.9.0-0.nightly-2021-08-26-151113

Investigation still continue, but it is likely that one of these PRs causing this bootstrap control plane error.

Comment 2 Stephen Benjamin 2021-08-31 10:44:24 UTC
Looks like that release was garbage collected, any chance you saved the list of PR's?

Just looking at GitHub,there was an installer RHCOS bump on that day: https://github.com/openshift/installer/pull/5168. It passed e2e-metal-ipi-ovn-ipv6, though.

Comment 3 Arda Guclu 2021-08-31 10:50:00 UTC
I reproduced it on my local and see;
Aug 31 08:40:25 master-1.ostest.test.metalkube.org nm-dispatcher[150065]: req:1 'dhcp6-change' [br-ex], "/etc/NetworkManager/dispatcher.d/30-resolv-prepender": complete: failed with Script '/etc/NetworkManager/dispatcher.d/30-resolv-prepender' exited with error status 1.

I suspected https://github.com/openshift/machine-config-operator/pull/2706 and opened a revert PR to test this https://github.com/openshift/machine-config-operator/pull/2744

It looks like network issue. 

I copied the list of https://amd64.ocp.releases.ci.openshift.org/releasestream/4.9.0-0.nightly/release/4.9.0-0.nightly-2021-08-26-151113 not to lose history;

BAREMETAL-OPERATOR
Bug 1983190: Add LIVE_ISO_FORCE_PERSISTENT_BOOT_DEVICE variable #173
Full changelog
CLUSTER-KUBE-APISERVER-OPERATOR
Bug 1997420: revert wrong change on the api-usage rules #1204
Full changelog
CLUSTER-MONITORING-OPERATOR
Bug 1997528: remove use of etcd_object_counts metric #1345
Bug 1996941: adding label check for node when creating daemon set #1339
Full changelog
CLUSTER-NODE-TUNING-OPERATOR
Bug 1997486: Ship the latest TuneD and stalld. #265
Full changelog
CLUSTER-VERSION-OPERATOR
Bug 1986707: lib/resourcedelete/helper: Never-installed alternative in deletion log message #642
Full changelog
CONSOLE
Bug 1997102: Update gherkin for observe tab in workload sidebar #9865
Bug 1987344: Set openshift doc version to 4.8 #9889
Bug 1997655: Remove unused data-test-id which logs a react warning #9883
Full changelog
CSI-DRIVER-MANILA, OPENSTACK-CINDER-CSI-DRIVER, OPENSTACK-CLOUD-CONTROLLER-MANAGER
Bug 1988374: UPSTREAM: 1988374: Disable uuid checks on XFS (#1614) #72
Bug 1996031: Merge upstream tag ‘v1.22.0’ #70
Full changelog
MACHINE-CONFIG-OPERATOR
Bug 1971715: configure-ovs: fix RHEL7 specific issues #2706
Full changelog
OPERATOR-LIFECYCLE-MANAGER, OPERATOR-REGISTRY
Bug 1994648: fix(sub): Reset ResolutionFailed cond when error is resolved #176
Bug 1996878: Add deprecation warnings for CLIs that use or depend on sqlite #177
Full changelog
SERVICE-CA-OPERATOR
OWNERS: remove s-urbaniak #175
Full changelog
TESTS
Bug 1986562: Only check status for image trigger tests #26411
Full changelog

Comment 5 Stephen Benjamin 2021-09-01 01:45:08 UTC
TRT marking this a blocker due to impact on IPv6/dualstack.

Comment 39 Arda Guclu 2021-10-18 10:42:52 UTC
*** Bug 1999594 has been marked as a duplicate of this bug. ***

Comment 40 errata-xmlrpc 2021-10-18 17:49:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.