Bug 2001244

Summary: Enforce OpenShift's defined kubelet version skew policies
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: kube-apiserverAssignee: Luis Sanchez <sanchezl>
Status: CLOSED ERRATA QA Contact: Rahul Gangwar <rgangwar>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.9CC: aos-bugs, mfojtik, rgangwar, sanchezl, xxia
Target Milestone: ---Flags: rgangwar: needinfo-
Target Release: 4.8.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-12 06:01:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1998552    
Bug Blocks: 2001243    

Description OpenShift BugZilla Robot 2021-09-04 19:48:56 UTC 
+++ This bug was initially created as a clone of Bug #1998552 +++

The API Server Operator will set Upgradeable=False whenever any of the nodes within the cluster are at the skew limit; that is, when an upgrade of the API Server would exceed the allowable kubelet version skew.
Comment 3 Rahul Gangwar 2021-10-04 10:08:04 UTC 
 oc get clusterversion                                                                                                     
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-10-01-170057   True        False         34m     Cluster version is 4.6.0-0.nightly-2021-10-01-17005

oc get node -A       
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-138-250.us-east-2.compute.internal   Ready    worker   72m   v1.19.0+d5ed12c
ip-10-0-142-215.us-east-2.compute.internal   Ready    master   76m   v1.19.0+d5ed12c
ip-10-0-164-42.us-east-2.compute.internal    Ready    worker   68m   v1.19.0+d5ed12c
ip-10-0-171-39.us-east-2.compute.internal    Ready    master   76m   v1.19.0+d5ed12c
ip-10-0-203-163.us-east-2.compute.internal   Ready    worker   68m   v1.19.0+d5ed12c
ip-10-0-223-43.us-east-2.compute.internal    Ready    master   77m   v1.19.0+d5ed12c


After pausing Machineconfigpool for worker and upgrade to 4.8.

 oc get clusterversion           
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.11    True        False         30m     Cluster version is 4.8.11

oc get node -A                  
NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-0-138-250.us-east-2.compute.internal   Ready    worker   3h52m   v1.19.0+d5ed12c
ip-10-0-142-215.us-east-2.compute.internal   Ready    master   3h57m   v1.21.1+9807387
ip-10-0-164-42.us-east-2.compute.internal    Ready    worker   3h48m   v1.19.0+d5ed12c
ip-10-0-171-39.us-east-2.compute.internal    Ready    master   3h56m   v1.21.1+9807387
ip-10-0-203-163.us-east-2.compute.internal   Ready    worker   3h48m   v1.19.0+d5ed12c
ip-10-0-223-43.us-east-2.compute.internal    Ready    master   3h57m   v1.21.1+9807387

Not see any message regarding skew policies 

oc get co kube-apiserver -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  annotations:
    exclude.release.openshift.io/internal-openshift-hosted: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
  creationTimestamp: "2021-10-04T05:39:22Z"
  generation: 1
  managedFields:
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:exclude.release.openshift.io/internal-openshift-hosted: {}
          f:include.release.openshift.io/self-managed-high-availability: {}
      f:spec: {}
      f:status:
        .: {}
        f:extension: {}
    manager: cluster-version-operator
    operation: Update
    time: "2021-10-04T05:39:22Z"
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions: {}
        f:relatedObjects: {}
        f:versions: {}
    manager: cluster-kube-apiserver-operator
    operation: Update
    time: "2021-10-04T08:14:03Z"
  name: kube-apiserver
  resourceVersion: "119230"
  uid: 18d1b2ae-badc-4470-91f4-b401d5cb0bd6
spec: {}
status:
  conditions:
  - lastTransitionTime: "2021-10-04T05:58:22Z"
    message: 'NodeControllerDegraded: All master nodes are ready'
    reason: AsExpected
    status: "False"
    type: Degraded
  - lastTransitionTime: "2021-10-04T09:03:25Z"
    message: 'NodeInstallerProgressing: 3 nodes are at revision 8'
    reason: AsExpected
    status: "False"
    type: Progressing
  - lastTransitionTime: "2021-10-04T05:50:45Z"
    message: 'StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 8'
    reason: AsExpected
    status: "True"
    type: Available
  - lastTransitionTime: "2021-10-04T05:48:54Z"
    message: All is well
    reason: AsExpected
    status: "True"
    type: Upgradeable
  extension: null
  relatedObjects:
  - group: operator.openshift.io
    name: cluster
    resource: kubeapiservers
  - group: apiextensions.k8s.io
    name: ""
    resource: customresourcedefinitions
  - group: security.openshift.io
    name: ""
    resource: securitycontextconstraints
  - group: ""
    name: openshift-config
    resource: namespaces
  - group: ""
    name: openshift-config-managed
    resource: namespaces
  - group: ""
    name: openshift-kube-apiserver-operator
    resource: namespaces
  - group: ""
    name: openshift-kube-apiserver
    resource: namespaces
  - group: admissionregistration.k8s.io
    name: ""
    resource: mutatingwebhookconfigurations
  - group: admissionregistration.k8s.io
    name: ""
    resource: validatingwebhookconfigurations
  - group: controlplane.operator.openshift.io
    name: ""
    namespace: openshift-kube-apiserver
    resource: podnetworkconnectivitychecks
  - group: apiserver.openshift.io
    name: ""
    resource: apirequestcounts
  versions:
  - name: raw-internal
    version: 4.8.11
  - name: kube-apiserver
    version: 1.21.1
  - name: operator
    version: 4.8.11

@Luis : Need info that above verification is correct?
Comment 4 Rahul Gangwar 2021-10-05 07:12:45 UTC 
oc get clusterversion                                                                                                     
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-10-01-170057   True        False         34m     Cluster version is 4.6.0-0.nightly-2021-10-01-17005

oc get node -A       
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-138-250.us-east-2.compute.internal   Ready    worker   72m   v1.19.0+d5ed12c
ip-10-0-142-215.us-east-2.compute.internal   Ready    master   76m   v1.19.0+d5ed12c
ip-10-0-164-42.us-east-2.compute.internal    Ready    worker   68m   v1.19.0+d5ed12c
ip-10-0-171-39.us-east-2.compute.internal    Ready    master   76m   v1.19.0+d5ed12c
ip-10-0-203-163.us-east-2.compute.internal   Ready    worker   68m   v1.19.0+d5ed12c
ip-10-0-223-43.us-east-2.compute.internal    Ready    master   77m   v1.19.0+d5ed12c


After pausing Machineconfigpool for worker and upgrade to 4.8.

oc get clusterversion                               
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-10-02-013545   True        False         27m     Cluster version is 4.8.0-0.nightly-2021-10-02-013545

oc get co kube-apiserver -o yaml

    message: 'KubeletMinorVersionUpgradeable: Kubelet minor versions on nodes ip-10-0-149-105.us-east-2.compute.internal, ip-10-0-170-190.us-east-2.compute.internal, and ip-10-0-206-233.us-east-2.compute.internal will not be supported in the next OpenShift minor version upgrade.'
    reason: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade
    status: "False"
    type: Upgradeable

apiversion:
  versions:
  - name: raw-internal
    version: 4.8.0-0.nightly-2021-10-02-013545
  - name: operator
    version: 4.8.0-0.nightly-2021-10-02-013545
  - name: kube-apiserver
    version: 1.21.4

Kubelet version"
oc get node -A|grep worker
ip-10-0-149-105.us-east-2.compute.internal   Ready    worker   16h   v1.19.0+d5ed12c
ip-10-0-170-190.us-east-2.compute.internal   Ready    worker   16h   v1.19.0+d5ed12c
ip-10-0-206-233.us-east-2.compute.internal   Ready    worker   16h   v1.19.0+d5ed12c

After unpausing machineconfigpool, kubelet version and apiversion in sync.

oc get node -A|grep worker                                                        
ip-10-0-149-105.us-east-2.compute.internal   Ready    worker   16h   v1.21.1+a620f50
ip-10-0-170-190.us-east-2.compute.internal   Ready    worker   16h   v1.21.1+a620f50
ip-10-0-206-233.us-east-2.compute.internal   Ready    worker   16h   v1.21.1+a620f50

 oc get co kube-apiserver -o yaml

 - lastTransitionTime: "2021-10-05T07:08:07Z"
    message: 'KubeletMinorVersionUpgradeable: Kubelet and API server minor versions are synced.'
    reason: AsExpected
    status: "True"
    type: Upgradeable
  extension: null

  versions:
  - name: raw-internal
    version: 4.8.0-0.nightly-2021-10-02-013545
  - name: operator
    version: 4.8.0-0.nightly-2021-10-02-013545
  - name: kube-apiserver
    version: 1.21.4
Comment 7 errata-xmlrpc 2021-10-12 06:01:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.14 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3682