Both of these jobs are failing with installer errors, starting from yesterday:
is failing frequently in CI, see:
Here is a list of job runs that had installer errors
After investigating the problem with @afasano, we found out that the first bootstrap error date is 26th August 18:15 and 16:18;
Checking the merged PR's after these times, most suspected PRs are in here;
Investigation still continue, but it is likely that one of these PRs causing this bootstrap control plane error.
Looks like that release was garbage collected, any chance you saved the list of PR's?
Just looking at GitHub,there was an installer RHCOS bump on that day: https://github.com/openshift/installer/pull/5168. It passed e2e-metal-ipi-ovn-ipv6, though.
I reproduced it on my local and see;
Aug 31 08:40:25 master-1.ostest.test.metalkube.org nm-dispatcher: req:1 'dhcp6-change' [br-ex], "/etc/NetworkManager/dispatcher.d/30-resolv-prepender": complete: failed with Script '/etc/NetworkManager/dispatcher.d/30-resolv-prepender' exited with error status 1.
I suspected https://github.com/openshift/machine-config-operator/pull/2706 and opened a revert PR to test this https://github.com/openshift/machine-config-operator/pull/2744
It looks like network issue.
I copied the list of https://amd64.ocp.releases.ci.openshift.org/releasestream/4.9.0-0.nightly/release/4.9.0-0.nightly-2021-08-26-151113 not to lose history;
Bug 1983190: Add LIVE_ISO_FORCE_PERSISTENT_BOOT_DEVICE variable #173
Bug 1997420: revert wrong change on the api-usage rules #1204
Bug 1997528: remove use of etcd_object_counts metric #1345
Bug 1996941: adding label check for node when creating daemon set #1339
Bug 1997486: Ship the latest TuneD and stalld. #265
Bug 1986707: lib/resourcedelete/helper: Never-installed alternative in deletion log message #642
Bug 1997102: Update gherkin for observe tab in workload sidebar #9865
Bug 1987344: Set openshift doc version to 4.8 #9889
Bug 1997655: Remove unused data-test-id which logs a react warning #9883
CSI-DRIVER-MANILA, OPENSTACK-CINDER-CSI-DRIVER, OPENSTACK-CLOUD-CONTROLLER-MANAGER
Bug 1988374: UPSTREAM: 1988374: Disable uuid checks on XFS (#1614) #72
Bug 1996031: Merge upstream tag ‘v1.22.0’ #70
Bug 1971715: configure-ovs: fix RHEL7 specific issues #2706
Bug 1994648: fix(sub): Reset ResolutionFailed cond when error is resolved #176
Bug 1996878: Add deprecation warnings for CLIs that use or depend on sqlite #177
OWNERS: remove s-urbaniak #175
Bug 1986562: Only check status for image trigger tests #26411
Running a revert PR https://github.com/openshift/installer/pull/5180 of https://github.com/openshift/installer/pull/5168. e2e-metal-ipi-ovn-ipv6 jobs passed.
TRT marking this a blocker due to impact on IPv6/dualstack.
*** Bug 1999594 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.