Bug 2100544
| Summary: | [4.10] [Webscale] High OVS cpu usage causing performance issues | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Yanir Quinn <yquinn> |
| Component: | Performance Addon Operator | Assignee: | Yanir Quinn <yquinn> |
| Status: | CLOSED ERRATA | QA Contact: | Niranjan Mallapadi Raghavender <mniranja> |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 4.9 | CC: | ajuarez, amorenoz, anbhat, atheurer, bbennett, bhershbe, browsell, bzhai, cchen, cfields, ctrautma, dacarpen, ealcaniz, eglottma, fbaudin, fdeutsch, ffernand, fleitner, gdiotte, grajaiya, i.maximets, jerward, jhsiao, midzik, mkennell, openshift-bugs-escalate, ralongi, rcernin, rkhan, shajmakh, surya, vjaypurk, vnema, yquinn |
| Target Milestone: | --- | ||
| Target Release: | 4.10.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 2081852 | Environment: | |
| Last Closed: | 2022-08-09 02:52:06 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2081852 | ||
| Bug Blocks: | 2108556, 2108557 | ||
|
Comment 1
Yanir Quinn
2022-06-23 16:18:35 UTC
*** Bug 2108217 has been marked as a duplicate of this bug. *** Verification:
Versions:
OCP: 4.10.25
PAO: 4.10.6-3
Steps:
apply pao profile:
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
name: manual
spec:
cpu:
isolated: "0-2"
reserved: "3"
realTimeKernel:
enabled: true
nodeSelector:
node-role.kubernetes.io/workercnf: ""
and after the nodes are ready apply the gu pod:
apiVersion: v1
kind: Pod
metadata:
name: test2
annotations:
irq-load-balancing.crio.io: "disable"
cpu-quota.crio.io: "disable"
spec:
containers:
- name: test
image: registry.redhat.io/openshift4/performance-addon-rhel8-operator@sha256:9be84526676476c26d5b20728580a67f202ea078ccb7044431f2b5fd6a3b22c8
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args: [ "while true; do sleep 100000; done;" ]
resources:
requests:
cpu: 2
memory: "200M"
limits:
cpu: 2
memory: "200M"
nodeSelector:
node-role.kubernetes.io/workercnf: ""
runtimeClassName: performance-manual
- connect to the pod and check the rps mask is set for veth devices and aligned with the PP:
[root@registry ~]# oc rsh test2
sh-4.4# find /sys/devices/virtual/ -name rps_cpus -printf '%p\n' -exec cat {} \;
/sys/devices/virtual/net/lo/queues/rx-0/rps_cpus
0000,00000000,00000008
/sys/devices/virtual/net/eth0/queues/rx-0/rps_cpus
0000,00000000,00000008
sh-4.4# find /sys/devices/ -name rps_cpus -printf '%p\n' -exec cat {} \;
/sys/devices/virtual/net/lo/queues/rx-0/rps_cpus
0000,00000000,00000008
/sys/devices/virtual/net/eth0/queues/rx-0/rps_cpus
0000,00000000,00000008
sh-4.4#
As can be seen, in both commands only the veth devices shown.
- Verify the new annotation works properly:
when the profile doesn't enable the annotation, only the virtual devices are seen in systemctl:
sh-4.4# systemctl list-units -all | grep update-rps@
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
sh-4.4#
and when the new annotation is added to the profile like below:
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
annotations:
performance.openshift.io/enable-physical-dev-rps: "true"
name: manual
...
all devices are shown now:
sh-4.4# systemctl list-units -all | grep update-rps@
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps <--- loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
sh-4.4#
bug verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.10.26 low-latency extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:5929 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |