Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1967317

Summary: SNO with realtimekernel enabled through PAO becomes unstable after running control plane churn workload and rebooting
Product: OpenShift Container Platform Reporter: Murali Krishnasamy <murali>
Component: Performance Addon OperatorAssignee: Martin Sivák <msivak>
Status: CLOSED CURRENTRELEASE QA Contact: yliu1
Severity: high Docs Contact:
Priority: urgent    
Version: 4.8CC: aos-bugs, bhu, browsell, dblack, djuran, fromani, grajaiya, juri.lelli, keyoung, minmli, mniranja, rkhan, rolove, smalleni, vlaad, yliu1
Target Milestone: ---Keywords: TestBlocker
Target Release: 4.9.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1975348 (view as bug list) Environment:
Last Closed: 2022-08-26 14:20:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1975348    

Description Murali Krishnasamy 2021-06-02 20:54:28 UTC 
Description of problem:

After creating a PAO profile with realTimeKernel on Single Node Openshift, the pod creation becomes very problematic and node goes unresponsive within a few attempts of running a churn test to create and delete pods(not more than 40 pods = 2 pods * 20 namespace).

Version-Release number of selected component (if applicable):
OCP - 4.8.0-fc.3

How reproducible:
Always

Steps to Reproduce:
1. Install 4.8.0-fc.3 SNO cluster on baremetal nodes
2. Create PAO profile with RT - 
3. Update kernel rpms to this https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=36613769 (steps are in https://bugzilla.redhat.com/show_bug.cgi?id=1946798#c48 to fix kernel panic)
4. After reboots the cluster looks normal and node is Ready.
5. Run a churn test to re-create  pods & namespaces continuously 2 to 3 times and watch node starts failing to respond. (Used Kube-burner)

Actual results:
Pod creation takes longer, eventually API requests exceed timeout(oc commands), node becomes unresponsive and cannot ssh even after a force reboot.

Expected results:
Cluster should be stable despite api latency

Additional info:
Hardware - Dell FC640

CRI-O version 
-------------
# rpm -qa | grep cri-o
cri-o-1.21.0-90.rhaos4.8.git07becf8.el8.x86_64

PAO profile 
-----------
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: perf-example
spec:
  cpu:
    isolated: "2-31,34-63"
    reserved: "0,1,32,33"
  hugepages:
    defaultHugepagesSize: 1G
    pages:
    - count: 16
      size: 1G
      node: 0
  nodeSelector:
    node-role.kubernetes.io/master: ""
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/master: ""
  numa:
    topologyPolicy: "restricted"
  realTimeKernel:
    enabled: true
Comment 1 Sai Sindhur Malleni 2021-06-02 21:02:19 UTC 
Murali,

Can you please paste a reproducer script so that it makes it easier.
Comment 3 Martin Sivák 2021-06-03 14:15:05 UTC 
This could be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1939133
Comment 5 Murali Krishnasamy 2021-06-07 16:57:09 UTC 
I could churn up to 160 pods(that hits maximum limit of 250) with the updated kernel, node stayed stable for 48 hours without any noticeable issue, but after reboot many pods stuck in 'CreateContainerError' state forever including system pods. 

# oc get pods -A | grep -v Run | grep -v Compl | wc -l
189

Still after deleting test apps system pods get this error, takes longer to heal. 

# oc describe pod -n openshift-ingress router-default-7b6cfdc88c-nqwfv
...

  Warning  Failed       50m                  kubelet  Error: Kubelet may be retrying requests that are timing out in CRI-O due to system load: context deadline exceeded: error reserving ctr name k8s_router_router-default-7b6cfdc88c-nqwfv_openshift-ingress_458f1561-7035-4b5a-b1ee-bfe80296f7ba_32 for id bd1dbed2dc030b89bf1fe23eee083f80837bbdccb8a20932cdfed5d2cd8c65aa: name is reserved
  Warning  ProbeError   47m (x230 over 67m)  kubelet  Startup probe error: HTTP probe failed with statuscode: 500
body: [-]backend-http failed: reason withheld
[-]has-synced failed: reason withheld
[+]process-running ok
healthz check failed
  Warning  BackOff  42m (x6 over 43m)     kubelet  Back-off restarting failed container
  Warning  Failed   36m                   kubelet  Error: Kubelet may be retrying requests that are timing out in CRI-O due to system load: the requested container k8s_router_router-default-7b6cfdc88c-nqwfv_openshift-ingress_458f1561-7035-4b5a-b1ee-bfe80296f7ba_33 is now ready and will be provided to the kubelet on next retry: error reserving ctr name k8s_router_router-default-7b6cfdc88c-nqwfv_openshift-ingress_458f1561-7035-4b5a-b1ee-bfe80296f7ba_33 for id f85fe525ed8b7777dd27ab0f0e42fb0ec67fb07bf75c7ad8a9cf8e259156438f: name is reserved
  Warning  Failed   7m17s (x17 over 60m)  kubelet  Error: context deadline exceeded
  Warning  Failed   39s                   kubelet  Error: Kubelet may be retrying requests that are timing out in CRI-O due to system load: context deadline exceeded: error reserving ctr name k8s_router_router-default-7b6cfdc88c-nqwfv_openshift-ingress_458f1561-7035-4b5a-b1ee-bfe80296f7ba_34 for id 578bd4beab0a79abd17873b161d83310389a3b78ef71584f14d86587ed199316: name is reserved

This is my PAO spec, 

  cpu:
    isolated: "2-31,34-63"
    reserved: "0,1,32,33"
Comment 6 Murali Krishnasamy 2021-06-08 16:22:04 UTC 
Further testing showed that, node can seamlessly reboot with 20 test pods (overall pod count to 100). 

Adding 10 more would cause below errors

  Warning  Failed   12s (x8 over 22m)   kubelet  Error: context deadline exceeded


Note: Test apps are just dummy pause pods using gcr.io/google_containers/pause-amd64:3.0.
Comment 7 Sai Sindhur Malleni 2021-06-08 16:34:13 UTC 
Changed title to reflect that this happens only after reboot with the updated kernel.
Comment 18 yliu1 2021-09-22 14:34:09 UTC 
Verification on 4.9 is blocked by https://bugzilla.redhat.com/show_bug.cgi?id=2006853.
Comment 19 yliu1 2021-09-22 14:42:52 UTC 
2006853 was a dup and just got fixed. Will verify both issues in next stable build.
Comment 20 yliu1 2021-09-29 15:27:50 UTC 
Verified on 4.9 rc.3. 
RT kernel is enabled via PAO without sideloading. 
Reboot with 43 test pods worked well, and whole cluster recovers within reasonable time.
Comment 21 yliu1 2021-09-29 22:24:16 UTC 
Verified on 4.8.13 with RT kernel enabled via PAO. Rebooted 5 times with 43 test pods, cluster recovered within reasonable time.
Comment 23 yliu1 2022-09-13 14:23:44 UTC 
This is automated and running as part of RAN qe pipeline: https://gitlab.cee.redhat.com/cnf/cnf-gotests/-/tree/master/test/ran/reboot
Comment 24 Martin Sivák 2022-10-17 08:29:19 UTC 
*** Bug 1975050 has been marked as a duplicate of this bug. ***