Bug 1686003
| Summary: | [OvS] High CPU in ovs-vswitchd on qrouter migration or service restart | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Mauro Oddi <moddi> | |
| Component: | openvswitch | Assignee: | Flavio Leitner <fleitner> | |
| Status: | CLOSED ERRATA | QA Contact: | Roee Agiman <ragiman> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 14.0 (Rocky) | CC: | aconole, apevec, bcafarel, ccamposr, chrisw, cswanson, fhallal, fleitner, njohnston, pmorey, rheslop, rhos-maint, rkhan, shdunne, sputhenp, tredaelli | |
| Target Milestone: | z3 | Keywords: | Triaged, ZStream | |
| Target Release: | 14.0 (Rocky) | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openvswitch2.10-2.10.0-28.el7fdp.3 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1692812 (view as bug list) | Environment: | ||
| Last Closed: | 2019-07-02 20:09:55 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1692813 | |||
| Bug Blocks: | ||||
Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=149 sclass=31
4953 static int selinux_nlmsg_perm(struct sock *sk, struct sk_buff *skb)
[...]
4966 err = selinux_nlmsg_lookup(sksec->sclass, nlh->nlmsg_type, &perm);
152 int selinux_nlmsg_lookup(u16 sclass, u16 nlmsg_type, u32 *perm)
[...]
156 switch (sclass) {
157 case SECCLASS_NETLINK_ROUTE_SOCKET:
#define SECCLASS_NETLINK_ROUTE_SOCKET 31 <==== sclass=31
160 err = nlmsg_perm(nlmsg_type, perm, nlmsg_route_perms,
161 sizeof(nlmsg_route_perms));
162 break;
138 static int nlmsg_perm(u16 nlmsg_type, u32 *perm, struct nlmsg_perm *tab, size_t tabsize)
139 {
140 int i, err = -EINVAL;
141
142 for (i = 0; i < tabsize/sizeof(struct nlmsg_perm); i++)
143 if (nlmsg_type == tab[i].nlmsg_type) {
144 *perm = tab[i].perm;
145 err = 0;
146 break;
147 }
148
149 return err;
150 }
nlmsg_type=149
That type doesn't exist in the table nlmsg_route_perms and causes to return NULL. Same for all other values, for example:
Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=14 sclass=31
and the table:
21 /* Types of messages */
22
23 enum {
24 RTM_BASE = 16,
25 #define RTM_BASE RTM_BASE
26
27 RTM_NEWLINK = 16,
28 #define RTM_NEWLINK RTM_NEWLINK
29 RTM_DELLINK,
30 #define RTM_DELLINK RTM_DELLINK
31 RTM_GETLINK,
32 #define RTM_GETLINK RTM_GETLINK
33 RTM_SETLINK,
34 #define RTM_SETLINK RTM_SETLINK
35
36 RTM_NEWADDR = 20,
There is no 14. The same in RHEL kernel is synchronized with upstream kernel, so this doesn't seem to be Old x New issue.
fbl
Hi, First of all, if the system is not exposed, can the customer disable SELinux while we investigate the issue? This will help to reduce the severity/customer temperature. It is a work around, of course, just while we investigate the root cause and work on a fix. Secondly, it seems that openvswitch daemon in userspace is sending a corrupted message. Most of the code uses hard coded values and I can't see where the problem is. My suggestion is to capture the messages in a pcap so I can try to see what it the message is supposed to be. # modprobe nlmon # ip link add type nlmon # ip link set nlmon0 up You can use wireshark or tcpdump to save the messages. I need them full length. Attach the sniffer and then reproduce the problem. Thanks, fbl perf report on the test env points to OVS though the issue can be triggered with creating/deleting a linux bridge:
# Samples: 75 of event 'probe:selinux_nlmsg_perm'
# Event count (approx.): 75
#
# Children Self Command Shared Object Symbol
# ........ ........ ............ .................. .........................
#
100.00% 100.00% ovs-vswitchd [kernel.vmlinux] [k] selinux_netlink_send
|
---__libc_sendmsg
system_call
sys_sendmsg
__sys_sendmsg
___sys_sendmsg
sock_sendmsg
netlink_sendmsg
security_netlink_send
selinux_netlink_send
Hi Mauro, This is a brew build with a patch to crash OVS when a nlmsg of type 0 or 1 and protocol NETLINK_ROUTE. The idea is to capture the coredump and see the stack trace to see what is going on. https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20664818 Don't forget to enable coredump that is disable by default in RHEL-7. fbl Hi Flavio, Thank you for the build, I have installed it in controller-0 and enabled coredumps in the node but for some reason neuton_ovs_agent container falls in restart loop, I am trying to solve that. Will update you as soon as possible. Thanks and Best Regards, Mauro S. Oddi Created clones to fix OVS: Bug 1692812 - [OvS 2.11] High CPU in ovs-vswitchd on qrouter migration or service restart Bug 1692813 - [OvS 2.10] High CPU in ovs-vswitchd on qrouter migration or service restart Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1671 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: After upgradeing RHOSP13 to RHOSP14 (RPM based deployment), ovs-vswitchd gets to 100% CPU utilization when doing a qrouter migration or at the start up of the service: - Perf data security_netlink_send - 96.77% 0.00% ovs-vswitchd [kernel.kallsyms] [k] system_call_fastpath - system_call_fastpath - 96.49% sys_sendmsg __sys_sendmsg ___sys_sendmsg sock_sendmsg - netlink_sendmsg - 95.02% security_netlink_send selinux_netlink_send printk vprintk_default + vprintk_emit + 1.42% netlink_unicast - /var/log/messages Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=149 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=149 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=14 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=14 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=201 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=201 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=59 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=59 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=235 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=235 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=23 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=23 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=120 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=120 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=70 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=70 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=193 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: protocol=0 nlmsg_type=193 sclass=31 Mar 6 13:26:40 srv-51d03-22 kernel: SELinux: unrecognized netlink message: $ cat messages | grep "unrecognized netlink message" | wc -l 28062 $ cat ../../installed-rpms | egrep 'selinux|openvswitch' container-selinux-2.77-1.el7_6.noarch Wed Mar 6 11:34:42 2019 libselinux-2.5-14.1.el7.x86_64 Wed Mar 6 11:02:25 2019 libselinux-python-2.5-14.1.el7.x86_64 Wed Mar 6 11:34:42 2019 libselinux-utils-2.5-14.1.el7.x86_64 Wed Mar 6 11:02:27 2019 openstack-neutron-openvswitch-12.0.5-3.el7ost.noarch Wed Mar 6 11:36:12 2019 openstack-selinux-0.8.15-1.el7ost.noarch Wed Mar 6 11:34:56 2019 openvswitch2.10-2.10.0-28.el7fdp.1.x86_64 Wed Mar 6 12:39:38 2019 openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch Wed Mar 6 11:35:37 2019 python-openvswitch2.10-2.10.0-28.el7fdp.1.x86_64 Wed Mar 6 12:39:38 2019 python-rhosp-openvswitch-2.10-0.1.el7ost.noarch Wed Mar 6 12:39:38 2019 rhosp-openvswitch-2.10-0.1.el7ost.noarch Wed Mar 6 12:39:38 2019 selinux-policy-3.13.1-229.el7_6.9.noarch Wed Mar 6 11:23:48 2019 selinux-policy-targeted-3.13.1-229.el7_6.9.noarch Wed Mar 6 11:24:11 2019 - SElinux is in permissive $cat ../../sos_commands/selinux/sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: permissive Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 31 $ $ cat ../../sos_commands/yum/yum_-C_repolist Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. repo id repo name status Filebeat-5.0 Filebeat repository 12 ceph-7/x86_64 Ceph packages for x86_64 280 epel/x86_64 EPEL repository - x86_64 996 rhel-7-emergency Red Hat Enterprise Linux 7 - Emer 6 rhel-7-server-extra-rpms/x86_64 Red Hat Enterprise Linux 7 Server 141 rhel-7-server-openstack-13-rpms Red Hat Openstack 13 for RHEL 7 ( 832 rhel-7-server-openstack-optools-13-rpms Red Hat Openstack Optools 13 for 119 rhel-7-server-optional-rpms/x86_64 Red Hat Enterprise Linux 7 Server 5149 rhel-7-server-rh-common-rpms/x86_64 Red Hat Enterprise Linux 7 Server 84 rhel-7-server-rpms/x86_64 Red Hat Enterprise Linux 7 Server 5368+9 rhel-ha-for-rhel-7-server-rpms/x86_64 Red Hat Enterprise Linux High Ava 59 zabbix/x86_64 Zabbix Official Repository - x86_ 183 repolist: 13229 Version-Release number of selected component (if applicable): RHOSP 14 3.10.0-862.3.2.el7.x86_64 container-selinux-2.77-1.el7_6.noarch Wed Mar 6 11:34:42 2019 libselinux-2.5-14.1.el7.x86_64 Wed Mar 6 11:02:25 2019 libselinux-python-2.5-14.1.el7.x86_64 Wed Mar 6 11:34:42 2019 libselinux-utils-2.5-14.1.el7.x86_64 Wed Mar 6 11:02:27 2019 openstack-neutron-openvswitch-12.0.5-3.el7ost.noarch Wed Mar 6 11:36:12 2019 openstack-selinux-0.8.15-1.el7ost.noarch Wed Mar 6 11:34:56 2019 openvswitch2.10-2.10.0-28.el7fdp.1.x86_64 Wed Mar 6 12:39:38 2019 openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch Wed Mar 6 11:35:37 2019 python-openvswitch2.10-2.10.0-28.el7fdp.1.x86_64 Wed Mar 6 12:39:38 2019 python-rhosp-openvswitch-2.10-0.1.el7ost.noarch Wed Mar 6 12:39:38 2019 rhosp-openvswitch-2.10-0.1.el7ost.noarch Wed Mar 6 12:39:38 2019 selinux-policy-3.13.1-229.el7_6.9.noarch Wed Mar 6 11:23:48 2019 selinux-policy-targeted-3.13.1-229.el7_6.9.noarch Wed Mar 6 11:24:11 2019 How reproducible: always Steps to Reproduce: 1. migrate a qrouter -or- 1. systemctl restart neutron-openvswitch-agent Actual results: Expected results: Additional info: