Bug 2032998

Description of problem:
Attempting to validate 250 max-pods a node per OpenShift worker node is failing with many pods stuck in ContainerCreating. This seems to occur after several times roughly reaching 250 max-pods a node.

The current cluster under test is a 13 node cluster hosted in ibmcloud. There are 3 control-plane nodes and 10 worker nodes.  The 10 worker nodes are sub-divided into nodedensity and worker role nodes:
# oc get no 
NAME          STATUS   ROLES                AGE   VERSION
jetlag-bm10   Ready    master               20h   v1.22.1+6859754
jetlag-bm11   Ready    master               21h   v1.22.1+6859754
jetlag-bm12   Ready    master               21h   v1.22.1+6859754
jetlag-bm13   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm14   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm15   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm16   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm17   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm18   Ready    worker               20h   v1.22.1+6859754
jetlag-bm19   Ready    worker               20h   v1.22.1+6859754
jetlag-bm20   Ready    worker               20h   v1.22.1+6859754
jetlag-bm21   Ready    worker               20h   v1.22.1+6859754
jetlag-bm22   Ready    worker               20h   v1.22.1+6859754

The test places workload pods on the nodedensity role type nodes.  The nodedensity nodes were cordoned, drained, then uncordoned prior to testing to "push" any steady state workloads off those 5 nodes such that they should run on the remaining 5 worker nodes and not interfere with the capacity of the nodedensity role-type workers.

After reviewing capacity for the nodes, I set the pod/node count to 234 such that we would create 5 * 234 namespaces each with one deployment with one pod replica with one container per pod. This results in 1170 namespaces, 1170 deployments and 1170 pods and 1170 total workload containers running across the 5 "nodedensity" role nodes. 234pods/node leaves room for 10 pods across the 5 nodes under test and actually does not test to the exact limit of 250pods/node.

The test repeat to run several iterations to see if the performance of loading up the nodes varies at all.  The test iterations until failure are:

1. 6 x 1170 namespaces with 1 deployment, 1 service, 1 route, 1 pod replica per deployment, 1 workload container per pod replica, no probes, no secrets/configmaps
2. 3 x 1170 namespaces same as above but setting requests and limits to establish Guaranteed QOS pods.  <--- Test failed/stuck on 3rd iteration


Layout of stuck pods:
# oc get po -A -o wide | grep ContainerCreating | awk '{print $8}' | sort | uniq -c
      5 jetlag-bm10
      4 jetlag-bm11
      6 jetlag-bm12
    235 jetlag-bm13
    234 jetlag-bm14
    234 jetlag-bm15
    233 jetlag-bm16
    234 jetlag-bm17
     42 jetlag-bm18
     38 jetlag-bm20
      1 jetlag-bm21

# oc get po -A -o wide | grep boatload | grep ContainerCreating | awk '{print $8}' | sort | uniq -c
    235 jetlag-bm13
    234 jetlag-bm14
    234 jetlag-bm15
    233 jetlag-bm16
    234 jetlag-bm17

# oc get po -A -o wide | grep boatload | grep ContainerCreating -c
1170

Every workload pod (pods with name prefix boatload) is stuck
Pods are stuck on nodes not hosting the workload (Including master nodes)

Snippet of stuck pods:
# oc get po -A | grep boatload | head
boatload-1                                         boatload-1-1-boatload-7fbbfdd9d4-dbdln                       0/1     ContainerCreating   0             18h
boatload-10                                        boatload-10-1-boatload-9b7c4f96-ntnr5                        0/1     ContainerCreating   0             18h
boatload-100                                       boatload-100-1-boatload-699b746448-nzg5w                     0/1     ContainerCreating   0             18h
boatload-1000                                      boatload-1000-1-boatload-6649fcdfbd-llndn                    0/1     ContainerCreating   0             18h
boatload-1001                                      boatload-1001-1-boatload-76cfbcdcb5-p4w6n                    0/1     ContainerCreating   0             18h
boatload-1002                                      boatload-1002-1-boatload-6d4fff77f8-cpvxk                    0/1     ContainerCreating   0             18h
boatload-1003                                      boatload-1003-1-boatload-5ff58b46c9-n66qk                    0/1     ContainerCreating   0             18h
boatload-1004                                      boatload-1004-1-boatload-6b7797978d-crjzx                    0/1     ContainerCreating   0             18h
boatload-1005                                      boatload-1005-1-boatload-844768f7b9-kxcxt                    0/1     ContainerCreating   0             18h
boatload-1006                                      boatload-1006-1-boatload-78dd5c8f54-2pt2h                    0/1     ContainerCreating   0             18h





Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2021-12-12-232810


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Description from failed pod:

# oc describe po -n boatload-1
Name:           boatload-1-1-boatload-7fbbfdd9d4-dbdln
Namespace:      boatload-1
Priority:       0
Node:           jetlag-bm15/10.5.190.3
Start Time:     Tue, 14 Dec 2021 15:52:02 -0600
Labels:         app=boatload-1-1
                pod-template-hash=7fbbfdd9d4
Annotations:    openshift.io/scc: restricted
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/boatload-1-1-boatload-7fbbfdd9d4
Containers:
  boatload-1:
    Container ID:   
    Image:          quay.io/redhat-performance/test-gohttp-probe:v0.0.2
    Image ID:       
    Port:           8000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     50m
      memory:  100Mi
    Requests:
      cpu:     50m
      memory:  100Mi
    Environment:
      PORT:                         8000
      LISTEN_DELAY_SECONDS:         0
      LIVENESS_DELAY_SECONDS:       0
      READINESS_DELAY_SECONDS:      0
      RESPONSE_DELAY_MILLISECONDS:  0
      LIVENESS_SUCCESS_MAX:         0
      READINESS_SUCCESS_MAX:        0
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9q6zv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-9q6zv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Guaranteed
Node-Selectors:              jetlag=true
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From          Message
  ----     ------                  ----                 ----          -------
  Warning  ErrorAddingLogicalPort  73m (x17 over 18h)   controlplane  failed to ensure namespace locked: failed to create address set for namespace: boatload-1, error: failed to create new address set boatload-1_v4 (error in transact with ops [{Op:insert Table:Address_Set Row:map[external_ids:{GoMap:map[name:boatload-1_v4]} name:a9965174113379253624] Rows:[] Columns:[] Mutations:[] Timeout:0 Where:[] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}]: context deadline exceeded: while awaiting reconnection)
  Warning  FailedCreatePodSandBox  96s (x903 over 18h)  kubelet       (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_boatload-1-1-boatload-7fbbfdd9d4-dbdln_boatload-1_6001f68b-5ae2-4615-8fd7-263152c548ed_0(a5bf992d2e41e0c3a3c6e2b478538819b3c14f1ab44dc55e6dc91a26d7bbe973): error adding pod boatload-1_boatload-1-1-boatload-7fbbfdd9d4-dbdln to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [boatload-1/boatload-1-1-boatload-7fbbfdd9d4-dbdln/6001f68b-5ae2-4615-8fd7-263152c548ed:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[boatload-1/boatload-1-1-boatload-7fbbfdd9d4-dbdln a5bf992d2e41e0c3a3c6e2b478538819b3c14f1ab44dc55e6dc91a26d7bbe973] [boatload-1/boatload-1-1-boatload-7fbbfdd9d4-dbdln a5bf992d2e41e0c3a3c6e2b478538819b3c14f1ab44dc55e6dc91a26d7bbe973] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded
'

All nodes are in Ready State:
# oc get no
NAME          STATUS   ROLES                AGE   VERSION
jetlag-bm10   Ready    master               20h   v1.22.1+6859754
jetlag-bm11   Ready    master               21h   v1.22.1+6859754
jetlag-bm12   Ready    master               21h   v1.22.1+6859754
jetlag-bm13   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm14   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm15   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm16   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm17   Ready    nodedensity,worker   20h   v1.22.1+6859754
jetlag-bm18   Ready    worker               20h   v1.22.1+6859754
jetlag-bm19   Ready    worker               20h   v1.22.1+6859754
jetlag-bm20   Ready    worker               20h   v1.22.1+6859754
jetlag-bm21   Ready    worker               20h   v1.22.1+6859754
jetlag-bm22   Ready    worker               20h   v1.22.1+6859754

I assume a must-gather will fail while the cluster is in the current state so I will capture logs, then attempt to clear the workload, then attempt a must-gather.