Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2020694

Summary: SNO with realtimekernel enabled through PAO becomes unstable after running control plane churn workload and rebooting
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: Performance Addon OperatorAssignee: Artyom <alukiano>
Status: CLOSED ERRATA QA Contact: Gowrishankar Rajaiyan <grajaiya>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.8CC: aos-bugs, bhu, browsell, dblack, dgonyier, fromani, grajaiya, juri.lelli, keyoung, mniranja, murali, rkhan, rolove, shajmakh, smalleni
Target Milestone: ---Keywords: TestBlocker
Target Release: 4.7.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: performance-addon-operator-container-v4.7.7-2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-29 07:07:22 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1975348    
Bug Blocks:    
Attachments:
Description Flags
kube-burner YAML files none

Comment 4 Dwaine Gonyier 2021-11-22 21:22:51 UTC 
Just an informational comment: bash script mentioned above is located at /usr/local/bin/low-latency-hooks.sh on the node.
Comment 5 Dwaine Gonyier 2021-11-23 20:49:18 UTC 
sh# oc version
Client Version: 4.9.7
Server Version: 4.7.37
Kubernetes Version: v1.20.10+bbbc079
sh# oc get nodes -o wide
NAME                               STATUS   ROLES    AGE    VERSION            INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                        CONTAINER-RUNTIME
ocp47-master-0.demo.lab.shajmakh   Ready    master   4d7h   v1.20.10+bbbc079   192.168.122.236   <none>        Red Hat Enterprise Linux CoreOS 47.84.202111031903-0 (Ootpa)   4.18.0-305.25.1.rt7.97.el8_4.x86_64   cri-o://1.20.5-7.rhaos4.7.gite80c8db.el8
ocp47-master-1.demo.lab.shajmakh   Ready    master   4d7h   v1.20.10+bbbc079   192.168.122.223   <none>        Red Hat Enterprise Linux CoreOS 47.84.202111031903-0 (Ootpa)   4.18.0-305.25.1.rt7.97.el8_4.x86_64   cri-o://1.20.5-7.rhaos4.7.gite80c8db.el8
ocp47-master-2.demo.lab.shajmakh   Ready    master   4d7h   v1.20.10+bbbc079   192.168.122.149   <none>        Red Hat Enterprise Linux CoreOS 47.84.202111031903-0 (Ootpa)   4.18.0-305.25.1.rt7.97.el8_4.x86_64   cri-o://1.20.5-7.rhaos4.7.gite80c8db.el8
ocp47-worker-0.demo.lab.shajmakh   Ready    worker   4d7h   v1.20.10+bbbc079   192.168.122.92    <none>        Red Hat Enterprise Linux CoreOS 47.84.202111031903-0 (Ootpa)   4.18.0-305.25.1.rt7.97.el8_4.x86_64   cri-o://1.20.5-7.rhaos4.7.gite80c8db.el8
ocp47-worker-1.demo.lab.shajmakh   Ready    worker   4d7h   v1.20.10+bbbc079   192.168.122.63    <none>        Red Hat Enterprise Linux CoreOS 47.84.202111031903-0 (Ootpa)   4.18.0-305.25.1.rt7.97.el8_4.x86_64   cri-o://1.20.5-7.rhaos4.7.gite80c8db.el8

sh# oc describe pod performance-operator-6b6d55847d-dw82d | grep -Ei 'Image ID'
    Image ID:      registry.redhat.io/openshift4/performance-addon-rhel8-operator@sha256:6538c825c23adeeb44d0e58fe6a701b49d5475b0c696cd731d702cf45fd6fbd8


# Apply performance profile
# and wait for nodes to restart

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: perf-example-master
spec:
  cpu:
    isolated: "2-29"
    reserved: "0,1,30,31"
  hugepages:
    defaultHugepagesSize: 1G
    pages:
    - count: 16
      size: 1G
      node: 0
  nodeSelector:
    node-role.kubernetes.io/master: ""
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/master: ""
  numa:
    topologyPolicy: "restricted"
  realTimeKernel:
    enabled: true



# Ran kube-burner for a long time (hours) with attached yaml files in
# api-intensive.tar.gz  

$ oc login --username=kubeadmin --password=***
$ cd api-intensive
$ token=$(oc whoami -t)
$ uuid=$(uuidgen)
$ kube-burner init -c api-intensive.yml -t ${token} --uuid ${uuid}

# Run all of the following commands from debug shell

$ oc debug node/<master_node_name>
sh-4.4# chroot /host
sh-4.4# [...]

# verify reserved CPU mask from running performance profile

sh-4.4# taskset -c -p 1
pid 1's current affinity list: 0,1,30,31
sh-4.4# taskset -p 1
pid 1's current affinity mask: c0000003

# verify interface rps masks have the same value

sh-4.4# grep '^'  /sys/devices/virtual/net/*/queues/rx-0/rps_cpus
/sys/devices/virtual/net/br0/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/lo/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/ovs-system/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/tun0/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth0f004795/queues/rx-0/rps_cpus:00000000
/sys/devices/virtual/net/veth1220ba0a/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth15e53c38/queues/rx-0/rps_cpus:00000000
/sys/devices/virtual/net/veth4be24c98/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth7a66effb/queues/rx-0/rps_cpus:00000000
/sys/devices/virtual/net/veth7f1f479a/queues/rx-0/rps_cpus:00000000
/sys/devices/virtual/net/vetha0e57016/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vetha4a532ff/queues/rx-0/rps_cpus:00000000
/sys/devices/virtual/net/vetha7a4349b/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vetha99cf039/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethad8423f5/queues/rx-0/rps_cpus:00000000
/sys/devices/virtual/net/vethb075958b/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethb2eeca9c/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethc28b7c1f/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethcb5a498e/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethcde19363/queues/rx-0/rps_cpus:00000000
/sys/devices/virtual/net/vethcece476a/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethe3bbcd02/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethe86a6c98/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethece93073/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vxlan_sys_4789/queues/rx-0/rps_cpus:c0000003

# Inspected each veth interface with bad mask (00000000) and determined that they either
# were not attached to a pod, or a pod with no running containers. Correlated
# interfaces to pods/containers by netns value

sh-4-4# cd /tmp

#netns in pods:

sh-4.4# for podid in $(crictl pods | grep -v POD | cut -f1 -d' ') ; do  for netns in  $(crictl inspectp ${podid} | grep -o  '/var/run/netns/[-0-9a-f]\+' | cut -d'/' -f5) ; do echo "${podid} ${netns}" ; done    ; done | sor -u  | tee podid.txt
o

#netns for veths with broken rps masks:

sh--4.4#  for intfx in $(grep '^'  /sys/devices/virtual/net/*/queues/rx-0/rps_cpus  | grep :000 | cut -d'/' -f6 ) ; do echo -n "$intfx " ; ip link show  "$intfx" 2>>/dev/null  grep -A1 "$intfx"  | grep -E -o '[-0-9a-f]{1,36}$ ; done |  tee broken.txt

# Reboot node

# Verify low-latency-hooks.sh includes the correct block of code

sh-4.4# cat /usr/local/bin/low-latency-hooks.sh
[...]
# Updates the RPS mask for the interface inside of the container network namespace
mode=$(ip netns exec "${ns}" [ -w /sys ] && echo "rw" || echo "ro" 2>&1)
[ $? -eq 0 ] || { logger "${0} Failed to determine if the /sys is writable: ${mode}"; exit 0; }

if [ "${mode}" = "ro" ]; then
    res=$(ip netns exec "${ns}" mount -o remount,rw /sys 2>&1)
    [ $? -eq 0 ] || { logger "${0}: Failed to remount /sys as rw: ${res}"; exit 0; }
fi
[...]

# Verify contents of 99-master-rps.rules
sh-4.4# cat /etc/udev/rules.d/99-netdev-rps.rules 
SUBSYSTEM=="net", ACTION=="add", ENV{DEVPATH}!="/devices/virtual/net/veth*", TAG+="systemd", ENV{SYSTEMD_WANTS}="update-rps@%k.service"

# Verified after reboot there were no veth interfaces with broken rps masks
sh-4.4# grep '^'  /sys/devices/virtual/net/*/queues/rx-0/rps_cpus              
/sys/devices/virtual/net/br0/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/lo/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/ovs-system/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/tun0/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth051631d3/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth1d322ba9/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth266a49ad/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth43980cfe/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth50232f15/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth599beb1e/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth5d824e9f/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth814a76fd/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/veth83f46781/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethaf44f2ea/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethf3adc975/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vethfe9eff64/queues/rx-0/rps_cpus:c0000003
/sys/devices/virtual/net/vxlan_sys_4789/queues/rx-0/rps_cpus:c0000003
Comment 6 Dwaine Gonyier 2021-11-24 14:09:32 UTC 
Created attachment 1843410 [details]
kube-burner YAML files

Attachement for verification
Comment 8 errata-xmlrpc 2021-11-29 07:07:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.37 low-latency extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4756