Bug 2035757

Version:
$ openshift-install version
openshift-install 4.10.0-0.nightly-2021-12-23-153012
built from commit 94a3ed9cbe4db66dc50dab8b85d2abf40fb56426
release image registry.ci.openshift.org/ocp/release@sha256:39cacdae6214efce10005054fb492f02d26b59fe9d23686dc17ec8a42f428534
release architecture amd64

Platform: alibabacloud

Please specify:
* IPI (automated install with `openshift-install`. If you don't know, then it's IPI)

What happened?
One master node turned NotReady, with "Get \"http://127.0.0.1:10258/healthz\": dial tcp 127.0.0.1:10258: connect: connection refused". 

$ oc get nodes
NAME                                          STATUS     ROLES    AGE    VERSION
jiwei-cc-7cgps-master-0                       Ready      master   179m   v1.22.1+6859754
jiwei-cc-7cgps-master-1                       Ready      master   179m   v1.22.1+6859754
jiwei-cc-7cgps-master-2                       NotReady   master   3h1m   v1.22.1+6859754
jiwei-cc-7cgps-worker-ap-northeast-1a-sgfhf   Ready      worker   168m   v1.22.1+6859754
jiwei-cc-7cgps-worker-ap-northeast-1b-z9wt4   Ready      worker   167m   v1.22.1+6859754
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          3h3m    Unable to apply 4.10.0-0.nightly-2021-12-23-153012: an unknown error has occurred: MultipleErrors
$ 

What did you expect to happen?
All nodes should be Ready and installation should succeed.

How to reproduce it (as minimally and precisely as possible)?
Not sure, it's just we got the issue 1-2 times every week.

Anything else we need to know?
>> master-2 kubelet logs around "2021-12-27 07:38:40":
ec 27 07:38:33 jiwei-cc-7cgps-master-2 hyperkube[1351]: I1227 07:38:33.516968    1351 patch_prober.go:29] interesting pod/alibaba-cloud-controller-manager-66cc9ff74b-8s8lr container/cloud-controller-manager namespace/openshift-cloud-controller-manager: Liveness probe status=failure output="Get \"http://127.0.0.1:10258/healthz\": dial tcp 127.0.0.1:10258: connect: connection refused" start-of-body=
Dec 27 07:38:33 jiwei-cc-7cgps-master-2 hyperkube[1351]: I1227 07:38:33.517017    1351 prober.go:116] "Probe failed" probeType="Liveness" pod="openshift-cloud-controller-manager/alibaba-cloud-controller-manager-66cc9ff74b-8s8lr" podUID=2903ee36-c237-41fc-afb6-9cb069b45c41 containerName="cloud-controller-manager" probeResult=failure output="Get \"http://127.0.0.1:10258/healthz\": dial tcp 127.0.0.1:10258: connect: connection refused"
Dec 27 07:38:33 jiwei-cc-7cgps-master-2 hyperkube[1351]: E1227 07:38:33.680772    1351 kubelet_node_status.go:497] "Error updating node status, will retry" err="error getting node \"jiwei-cc-7cgps-master-2\": Get \"https://api-int.jiwei-cc.alicloud-qe.devcluster.openshift.com:6443/api/v1/nodes/jiwei-cc-7cgps-master-2?timeout=10s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Dec 27 07:38:33 jiwei-cc-7cgps-master-2 hyperkube[1351]: E1227 07:38:33.680798    1351 kubelet_node_status.go:484] "Unable to update node status" err="update node status exceeds retry count"
Dec 27 07:38:34 jiwei-cc-7cgps-master-2 hyperkube[1351]: E1227 07:38:34.180310    1351 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://api-int.jiwei-cc.alicloud-qe.devcluster.openshift.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/jiwei-cc-7cgps-master-2?timeout=10s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Dec 27 07:38:43 jiwei-cc-7cgps-master-2 hyperkube[1351]: I1227 07:38:43.516885    1351 patch_prober.go:29] interesting pod/alibaba-cloud-controller-manager-66cc9ff74b-8s8lr container/cloud-controller-manager namespace/openshift-cloud-controller-manager: Liveness probe status=failure output="Get \"http://127.0.0.1:10258/healthz\": dial tcp 127.0.0.1:10258: connect: connection refused" start-of-body=
Dec 27 07:38:43 jiwei-cc-7cgps-master-2 hyperkube[1351]: I1227 07:38:43.516948    1351 prober.go:116] "Probe failed" probeType="Liveness" pod="openshift-cloud-controller-manager/alibaba-cloud-controller-manager-66cc9ff74b-8s8lr" podUID=2903ee36-c237-41fc-afb6-9cb069b45c41 containerName="cloud-controller-manager" probeResult=failure output="Get \"http://127.0.0.1:10258/healthz\": dial tcp 127.0.0.1:10258: connect: connection refused"
Dec 27 07:38:45 jiwei-cc-7cgps-master-2 hyperkube[1351]: I1227 07:38:45.535993    1351 generic.go:296] "Generic (PLEG): container finished" podID=2903ee36-c237-41fc-afb6-9cb069b45c41 containerID="8176a37c73ee84b38872f1774263b4225f4ef8b678d232499a71e097ff00e08c" exitCode=1
Dec 27 07:38:45 jiwei-cc-7cgps-master-2 hyperkube[1351]: I1227 07:38:45.536014    1351 kubelet.go:2115] "SyncLoop (PLEG): event for pod" pod="openshift-cloud-controller-manager/alibaba-cloud-controller-manager-66cc9ff74b-8s8lr" event=&{ID:2903ee36-c237-41fc-afb6-9cb069b45c41 Type:ContainerDied Data:8176a37c73ee84b38872f1774263b4225f4ef8b678d232499a71e097ff00e08c}
Dec 27 07:38:45 jiwei-cc-7cgps-master-2 hyperkube[1351]: I1227 07:38:45.536050    1351 scope.go:110] "RemoveContainer" containerID="9d3363d96f7b7c3bdebdf5a52b75d7814e8ae2bfcda391aa6ee7d16ef54f7751"
Dec 27 07:38:45 jiwei-cc-7cgps-master-2 hyperkube[1351]: I1227 07:38:45.536526    1351 scope.go:110] "RemoveContainer" containerID="8176a37c73ee84b38872f1774263b4225f4ef8b678d232499a71e097ff00e08c"
Dec 27 07:38:45 jiwei-cc-7cgps-master-2 hyperkube[1351]: E1227 07:38:45.536934    1351 pod_workers.go:836] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cloud-controller-manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=cloud-controller-manager pod=alibaba-cloud-controller-manager-66cc9ff74b-8s8lr_openshift-cloud-controller-manager(2903ee36-c237-41fc-afb6-9cb069b45c41)\"" pod="openshift-cloud-controller-manager/alibaba-cloud-controller-manager-66cc9ff74b-8s8lr" podUID=2903ee36-c237-41fc-afb6-9cb069b45c41
Dec 27 07:38:48 jiwei-cc-7cgps-master-2 hyperkube[1351]: E1227 07:38:48.077850    1351 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"revision-pruner-7-jiwei-cc-7cgps-master-2.16c48c6840bd4760", GenerateName:"", Namespace:"openshift-kube-scheduler", SelfLink:"", UID:"", ResourceVersion:"14325", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-scheduler", Name:"revision-pruner-7-jiwei-cc-7cgps-master-2", UID:"3a7c704e-e07d-49c0-953e-27286b15a126", APIVersion:"v1", ResourceVersion:"13950", FieldPath:""}, Reason:"FailedKillPod", Message:"error killing pod: failed to \"KillPodSandbox\" for \"3a7c704e-e07d-49c0-953e-27286b15a126\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_revision-pruner-7-jiwei-cc-7cgps-master-2_openshift-kube-scheduler_3a7c704e-e07d-49c0-953e-27286b15a126_0(555c82ebb103d7812504c55ad2edeff2ee4890f227ac6e293b06fad4541a0bd1): error removing pod openshift-kube-scheduler_revision-pruner-7-jiwei-cc-7cgps-master-2 from CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): Multus: [openshift-kube-scheduler/revision-pruner-7-jiwei-cc-7cgps-master-2/3a7c704e-e07d-49c0-953e-27286b15a126]: error waiting for pod: Get \\\"https://[api-int.jiwei-cc.alicloud-qe.devcluster.openshift.com]:6443/api/v1/namespaces/openshift-kube-scheduler/pods/revision-pruner-7-jiwei-cc-7cgps-master-2?timeout=1m0s\\\": dial tcp 10.0.11.98:6443: connect: connection refused\"", Source:v1.EventSource{Component:"kubelet", Host:"jiwei-cc-7cgps-master-2"}, FirstTimestamp:time.Date(2021, time.December, 27, 7, 35, 43, 0, time.Local), LastTimestamp:time.Date(2021, time.December, 27, 7, 36, 7, 658824812, time.Local), Count:3, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Patch "https://api-int.jiwei-cc.alicloud-qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-kube-scheduler/events/revision-pruner-7-jiwei-cc-7cgps-master-2.16c48c6840bd4760": dial tcp 10.0.11.98:6443: i/o timeout'(may retry after sleeping)

>>"oc get co"/etc.
$ oc get co | grep -Ev 'True        False         False'
NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.0-0.nightly-2021-12-23-153012   False       True          True       146m    WellKnownAvailable: The well-known endpoint is not yet available: kube-apiserver oauth endpoint https://10.0.11.100:6443/.well-known/oauth-authorization-server is not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance)
console                                                                                                                      
dns                                        4.10.0-0.nightly-2021-12-23-153012   True        True          False      144m    DNS "default" reports Progressing=True: "Have 4 available node-resolver pods, want 5."
etcd                                       4.10.0-0.nightly-2021-12-23-153012   True        True          True       145m    NodeControllerDegraded: The master nodes not ready: node "jiwei-cc-7cgps-master-2" not ready since 2021-12-27 07:38:40 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
kube-apiserver                             4.10.0-0.nightly-2021-12-23-153012   False       True          True       147m    StaticPodsAvailable: 0 nodes are active; 3 nodes are at revision 0; 0 nodes have achieved new revision 8
kube-controller-manager                    4.10.0-0.nightly-2021-12-23-153012   True        True          True       145m    NodeControllerDegraded: The master nodes not ready: node "jiwei-cc-7cgps-master-2" not ready since 2021-12-27 07:38:40 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
kube-scheduler                             4.10.0-0.nightly-2021-12-23-153012   True        True          True       144m    InstallerPodContainerWaitingDegraded: Pod "installer-7-jiwei-cc-7cgps-master-2" on node "jiwei-cc-7cgps-master-2" container "installer" is waiting since 2021-12-27 07:35:31 +0000 UTC because ContainerCreating...
machine-config                             4.10.0-0.nightly-2021-12-23-153012   False       False         True       128m    Cluster not available for [{operator 4.10.0-0.nightly-2021-12-23-153012}]
network                                    4.10.0-0.nightly-2021-12-23-153012   True        True          True       146m    DaemonSet "openshift-multus/multus" rollout is not making progress - last change 2021-12-27T07:40:23Z...
openshift-multus/multus" rollout is not making progress - last change 2021-12-27T07:40:23Z...
openshift-apiserver                        4.10.0-0.nightly-2021-12-23-153012   True        True          False      141m    APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 1/3 pods have been updated to the latest generation
openshift-controller-manager               4.10.0-0.nightly-2021-12-23-153012   True        True          False      135m    Progressing: daemonset/controller-manager: updated number scheduled is 2, desired number scheduled is 3
openshift-samples
storage                                    4.10.0-0.nightly-2021-12-23-153012   True        True          False      142m    AlibabaDiskCSIDriverOperatorCRProgressing: AlibabaCloudDriverNodeServiceControllerProgressing: Waiting for DaemonSet to deploy node pods
$ 
$ oc describe co kube-controller-manager | grep Message
    Message:               NodeControllerDegraded: The master nodes not ready: node "jiwei-cc-7cgps-master-2" not ready since 2021-12-27 07:38:40 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
    Message:               NodeInstallerProgressing: 2 nodes are at revision 6; 1 nodes are at revision 7; 0 nodes have achieved new revision 9
    Message:               StaticPodsAvailable: 3 nodes are active; 2 nodes are at revision 6; 1 nodes are at revision 7; 0 nodes have achieved new revision 9
    Message:               All is well
$ 
$ oc describe co machine-config | grep Message
    Message:               Cluster version is 4.10.0-0.nightly-2021-12-23-153012
    Message:               Failed to resync 4.10.0-0.nightly-2021-12-23-153012 because: failed to apply machine config daemon manifests: timed out waiting for the condition during waitForDaemonsetRollout: Daemonset machine-config-daemon is not ready. status: (desired: 5, updated: 5, ready: 4, unavailable: 1)
    Message:               Cluster not available for [{operator 4.10.0-0.nightly-2021-12-23-153012}]
    Message:               One or more machine config pools are updating, please see `oc get mcp` for further details
$ 
$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-44cfabe16ea78c26ecf76cbe9009ba2b   False     True       False      3              2                   3                     0                      174m
worker   rendered-worker-8e4b60bb75a931a47212b087caf28333   True      False      False      2              2                   2                     0                      174m
$ oc -n openshift-kube-controller-manager get pods | grep -Ev 'Completed|Running'
NAME                                              READY   STATUS              RESTARTS         AGE
installer-7-jiwei-cc-7cgps-master-2               0/1     Terminating         0                167m
installer-8-jiwei-cc-7cgps-master-2               0/1     Terminating         0                165m
installer-9-jiwei-cc-7cgps-master-2               0/1     Pending             0                164m
revision-pruner-7-jiwei-cc-7cgps-master-2         0/1     ContainerCreating   0                169m
revision-pruner-8-jiwei-cc-7cgps-master-2         0/1     Pending             0                165m
revision-pruner-9-jiwei-cc-7cgps-master-2         0/1     Pending             0                164m
$