Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2092945

Summary: Unsupported kubelet minor versions on 6 nodes are ahead of the target API server version (1.23.3)
Product: OpenShift Container Platform Reporter: Micah Abbott <miabbott>
Component: kube-apiserverAssignee: Abu Kashem <akashem>
Status: CLOSED CURRENTRELEASE QA Contact: Ke Wang <kewang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.11CC: mfojtik, miabbott, walters, wking, xxia
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
[sig-arch][Early] Managed cluster should start all core operators [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
Last Closed: 2022-06-09 14:27:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Micah Abbott 2022-06-02 15:10:59 UTC 
We started to see the following test fail:

[sig-arch][Early] Managed cluster should start all core operators [Skipped:Disconnected] [Suite:openshift/conformance/parallel]

The most obvious error seems to be:

Jun  2 09:04:56.473: FAIL: Some cluster operators are not ready: kube-apiserver (Upgradeable=Unknown KubeletMinorVersion_KubeletMinorVersionAhead: KubeletMinorVersionUpgradeable: Unsupported kubelet minor versions on 6 nodes are ahead of the target API server version (1.23.3).)


This was observed both while testing a custom release payload with an RHCOS using RHEL 8.6 content:

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-aws-modern/1532275588608299008

...but is also showing up in the broader CI system a bit:

https://search.ci.openshift.org/?search=Unsupported+kubelet+minor+versions+on+6+nodes+are+ahead+of+the+target+API+server+version&maxAge=48h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job


Appears to be a recent error, as the earliest incident of this error is only 7hrs ago.
Comment 1 Colin Walters 2022-06-03 14:04:16 UTC 
I think the answer to this is likely simple, the last accepted nightly was 48 hours ago:
https://amd64.ocp.releases.ci.openshift.org/#4.11.0-0.nightly
Which probably has a bump to the apiserver.
Yet, we're going to be shipping the very latest kubelet in RHCOS.

Although I can't say why the failure wouldn't be entirely reproducible if that's the case.

What was the base image for the custom release payload?  If that's before the apiserver bump, that's likely the problem.
Comment 2 Colin Walters 2022-06-03 14:10:54 UTC 
OK yes, digging into it this testing image was based on an old nightly that doesn't have
https://github.com/openshift/cluster-kube-apiserver-operator/commit/35365fcd3dd4eda06962b37dd880ee56254113bf

I think let's:

- Get a PR up to openshift/os which goes back to 8.6 (per discussion in https://github.com/openshift/os/pull/803)
- Merge that PR
- Get 8.6 back into 4.11 nightlies (verify promotion job is green)
- If that happens, close this BZ as the test will have passed
Comment 3 Colin Walters 2022-06-09 14:27:19 UTC 
I believe this is fixed, current nightlies are 8.6 and are passing many tests.