Bug 2006945 - race condition can cause crashlooping bootstrap kube-apiserver in cluster-bootstrap
Summary: race condition can cause crashlooping bootstrap kube-apiserver in cluster-boo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Abu Kashem
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On:
Blocks: 2007324
TreeView+ depends on / blocked
 
Reported: 2021-09-22 17:14 UTC by Devan Goodwin
Modified: 2023-09-15 01:36 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2007324 (view as bug list)
Environment:
Last Closed: 2022-03-10 16:12:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-bootstrap pull 64 0 None open bug 2006945: extend hardcoded restmapper for cluster-bootstrap to avoid crashlooping bootstrap kube-apiserver 2021-09-22 18:48:40 UTC
Github openshift library-go pull 1219 0 None open bug 2006945: extend hardcoded restmapper for cluster-bootstrap to avoid crashlooping bootstrap kube-apiserver 2021-09-22 18:48:44 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:13:13 UTC

Description Devan Goodwin 2021-09-22 17:14:14 UTC
Version:

failed 4.9 installs in CI, examples: 

OVN: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-aws-ovn-upgrade/1439917064897695744

SDN: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-aws-upgrade/1440456016905900032

Platform: aws IPI

In the artifacts for these failed installs a log-bundle.tar is generated due to bootstrap failure.

In the bootkube.log you can see a large number of messages such as: 

bootstrap/journals/bootkube.log:Sep 21 23:44:37 ip-10-0-15-48 bootkube.sh[2293]: "0000_20_kube-apiserver-operator_00_cr-scc-hostaccess.yaml": unable to get REST mapping for "0000_20_kube-apiserver-operator_00_cr-scc-hostaccess.yaml": no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1"

Which is for a core type that should be present.

deads has prepared a:

proposed fix: https://github.com/openshift/library-go/pull/1219
proof pr: https://github.com/openshift/cluster-bootstrap/pull/64

Slack thread with deads and stts: https://coreos.slack.com/archives/C02FFM5PNSG/p1632319918039000

We are still trying to determine if these are why the installs are failing, but there's still a problem to be fixed here regardless.

Comment 8 Ke Wang 2021-12-09 07:03:45 UTC
$ w3m -dump -cols 200 'https://search.ci.openshift.org/?search=unable+to+get+REST+mapping&maxAge=336h&context=1&type=junit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job' | grep '4.10' | sort
No results found

And checked Jenkins Flexy install jobs with AWS + OVN profiles from 10/28/2021 until today,there are total 113 jobs including failed 25 jobs, made random selection of tests did not find the same problem.

Based on above, the PR fix works fine, move the bug VERIFIED.

Comment 11 errata-xmlrpc 2022-03-10 16:12:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056

Comment 12 Red Hat Bugzilla 2023-09-15 01:36:13 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days


Note You need to log in before you can comment on or make changes to this bug.