Bug 2081852
| Summary: | [Webscale] High OVS cpu usage causing performance issues | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Gabriel Diotte <gdiotte> | |
| Component: | Performance Addon Operator | Assignee: | Yanir Quinn <yquinn> | |
| Status: | CLOSED ERRATA | QA Contact: | Niranjan Mallapadi Raghavender <mniranja> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 4.9 | CC: | ajuarez, amorenoz, anbhat, arajapa, atheurer, bbennett, bhershbe, browsell, cfields, ctrautma, dacarpen, ealcaniz, eglottma, fbaudin, fdeutsch, ffernand, fleitner, fromani, grajaiya, i.maximets, jerward, jhsiao, mkennell, msivak, openshift-bugs-escalate, ralongi, rcernin, rkhan, shajmakh, surya, vjaypurk, vnema, yquinn | |
| Target Milestone: | --- | |||
| Target Release: | 4.11.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause:
1. The RPS cpu mask was set to the reserved CPUs taken from the performance profile for all interfaces by default.
2. A background script that runs as an oci pre-start hook (low-latency-hooks) for guaranteed pods sets the rps mask of all network devices visible from the root path of devices (/sys/devices).
Consequence:
1. This is not practical for sriov-vfs and similar interfaces that carry RAN payload traffic as this is not scalable.
2. The intent of the pre-start hook it to set the rps mask of virtual interfaces (i.e. the veth) but instead it sets the rps mask of all network devices visible in the container.
Fix:
1. Set RPS cpu mask to reserved CPUs taken from the performance profile excluding physical and veth interfaces.
*** NOTE: this is an emergency hook - not to be officially documented***
Add a new annotation A new annotation option to the performance profile: "performance.openshift.io/enable-physical-dev-rps".
When added to a performance profile,it will enable
RPS mask setting with systemd for all network devices by including physical interfaces from netdev-rps rule.
Usage example for enabling RPS mask for all network interfaces:
Oc edit <performanceprofile> for existing profiles OR when creating a new performance profile add the following annotation:
- apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
annotations:
performance.openshift.io/enable-physical-dev-rps: "true"
spec:
cpu:
reserved: <reserved cpus set>
When set to "true" - set RPS mask to the reserved cpus set for all network interfaces.
When omitted or set to false - set RPS mask to net interfaces excluding physical and veth interfaces. (default behavior)
2. Change the devices path in the script to set rps mask only for virtual interfaces.
Result:
1. RPS mask setting defaults for net interfaces excluding physical and vetch devices.
- An annotation added to included all net interfaces as described in the fix section.
2. Only virtual interfaces will be set with rps mask in the crio prestart hook.
More info can be found in the following slide deck:
https://docs.google.com/presentation/d/1ZKlMWwpzI50cdvEOaI1weB1hNhvNyD4HBeOY4ZhTQ3Y/edit?usp=sharing
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 2100544 (view as bug list) | Environment: | ||
| Last Closed: | 2022-08-10 12:16:31 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2096703, 2100544 | |||
|
Comment 13
Darren Carpenter
2022-05-17 15:28:02 UTC
Currently, we're looking for 'concrete' that points to load being transferred/running on the reserved cores. We seem to be able to reasonably speculate based on conditions. I'd expect to see either redis or log-transformer processees spilling over.... if it is simply the usage/execution of those causing ovs to do some weird bleed. *** Bug 2094935 has been marked as a duplicate of this bug. *** *** Bug 2093267 has been marked as a duplicate of this bug. *** Verification:
OCP version: 4.11.0-rc.1
Verified on BM machine with SRIOV support.
Steps:
- Check that rps mask is set only to virtual devices by default:
apply the following performance profile and wait for the nodes to be updated:
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
name: manual
spec:
cpu:
isolated: "0-2"
reserved: "3"
nodeSelector:
node-role.kubernetes.io/worker: ""
then apply a gu pod:
apiVersion: v1
kind: Pod
metadata:
name: test2
annotations:
irq-load-balancing.crio.io: "disable"
cpu-quota.crio.io: "disable"
spec:
containers:
- name: test
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2949f380f89e12ce91655e3b6631b4fa5001f1331f0f85e1713232bcb8e66f1
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args: [ "while true; do sleep 100000; done;" ]
resources:
requests:
cpu: 2
memory: "200M"
limits:
cpu: 2
memory: "200M"
nodeSelector:
node-role.kubernetes.io/worker: ""
runtimeClassName: performance-manual
login to the pod and check the rps is set for veth:
# oc rsh test2
sh-4.4# find /sys/devices/virtual/ -name rps_cpus -printf '%p\n' -exec cat {} \;
/sys/devices/virtual/net/lo/queues/rx-0/rps_cpus
0000,00000000,00000008
/sys/devices/virtual/net/eth0/queues/rx-0/rps_cpus
0000,00000000,00000008
sh-4.4# find /sys/devices/ -name rps_cpus -printf '%p\n' -exec cat {} \;
/sys/devices/virtual/net/lo/queues/rx-0/rps_cpus
0000,00000000,00000008
/sys/devices/virtual/net/eth0/queues/rx-0/rps_cpus
0000,00000000,00000008
As can be seen, in both commands only the veth devices shown.
- Verify the new annotation works properly:
when the profile doesn't enable the annotation, only the virtual devices are seen in systemctl:
sh-4.4# systemctl list-units -all | grep update-rps@
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
and when the profile enable the annotation, all devices are shown: (note that virtual devices had names changed upon the update of the profile)
sh-4.4# systemctl list-units -all | grep update-rps@
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps <-- loaded inactive dead Sets network devices RPS mask
update-rps <-- loaded inactive dead Sets network devices RPS mask
update-rps <-- loaded inactive dead Sets network devices RPS mask
update-rps <-- loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
update-rps loaded inactive dead Sets network devices RPS mask
bug verified.
enabling the annotation in the profile would be as the following:
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
annotations:
performance.openshift.io/enable-physical-dev-rps: "true" <---
name: manual
spec:
...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.11 low-latency extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:5869 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |