Bug 1736517
Summary: | FDP 19.E: openvswitch service fails to start with DPDK config and driverctl override list is empty | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Rick Alongi <ralongi> | ||||||||
Component: | openvswitch2.11 | Assignee: | David Marchand <dmarchan> | ||||||||
Status: | CLOSED DUPLICATE | QA Contact: | Jean-Tsung Hsiao <jhsiao> | ||||||||
Severity: | unspecified | Docs Contact: | |||||||||
Priority: | unspecified | ||||||||||
Version: | FDP 19.E | CC: | aconole, ctrautma, fhallal, jhsiao, ralongi | ||||||||
Target Milestone: | --- | ||||||||||
Target Release: | --- | ||||||||||
Hardware: | Unspecified | ||||||||||
OS: | Unspecified | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | Environment: | ||||||||||
Last Closed: | 2019-08-07 12:31:39 UTC | Type: | Bug | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Attachments: |
|
Description
Rick Alongi
2019-08-01 18:41:34 UTC
Created attachment 1598818 [details]
ovsdb-server.log
I can't seem to access the ovs-vswitchd.log from the sosreport - can you attach it also? And the journalctl -u openvswitch, journalctl -u ovsdb-server, journalctl -u ovs-vswitchd output, if possible. Actually, I see this from the error logs: 2019-08-01T18:07:09.590Z|00020|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory Can you get the output of /proc/meminfo ? You can also systemctl start ovsdb-server ovs-vsctl --no-wait get Open_vSwitch . other_config ovs-vswitchd.log and ovsdb-server.log are attached to this BZ. I forgot to include this output: [root@netqe6 ~]# ovs-vsctl list Open_vSwitch _uuid : fa94b582-af81-4d68-834f-a937930eb330 bridges : [636054cd-0db1-4990-815c-d008234d6529] cur_cfg : 6 datapath_types : [netdev, system] db_version : "7.16.1" dpdk_initialized : true dpdk_version : "DPDK 18.11.2" external_ids : {hostname="netqe6.knqe.lab.eng.bos.redhat.com", rundir="/var/run/openvswitch", system-id="48a99a25-51d6-474d-8b77-766541b6b8a3"} iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, internal, "ip6erspan", "ip6gre", lisp, patch, stt, system, tap, vxlan] manager_options : [] next_cfg : 6 other_config : {dpdk-init="true", dpdk-lcore-mask="0x000002", dpdk-socket-mem="4096,1", pmd-cpu-mask="0xaa00"} ovs_version : "2.11.0" ssl : [] statistics : {} system_type : rhel system_version : "7.7" Info requested: [root@netqe6 ~]# cat /proc/meminfo MemTotal: 65772872 kB MemFree: 30824936 kB MemAvailable: 30830116 kB Buffers: 2540 kB Cached: 368148 kB SwapCached: 0 kB Active: 304156 kB Inactive: 137724 kB Active(anon): 78368 kB Inactive(anon): 11036 kB Active(file): 225788 kB Inactive(file): 126688 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 32964604 kB SwapFree: 32964604 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 71196 kB Mapped: 34372 kB Shmem: 18212 kB Slab: 159312 kB SReclaimable: 74336 kB SUnreclaim: 84976 kB KernelStack: 5104 kB PageTables: 4312 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 49073824 kB Committed_AS: 307876 kB VmallocTotal: 34359738367 kB VmallocUsed: 436008 kB VmallocChunk: 34325399548 kB HardwareCorrupted: 0 kB AnonHugePages: 6144 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 32 HugePages_Free: 32 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB DirectMap4k: 249552 kB DirectMap2M: 5943296 kB DirectMap1G: 62914560 kB [root@netqe6 ~]# systemctl start ovsdb-server [root@netqe6 ~]# ovs-vsctl --no-wait get Open_vSwitch . other_config {dpdk-init="true", dpdk-lcore-mask="0x000002", dpdk-socket-mem="4096,1", pmd-cpu-mask="0xaa00"} Attaching journalctl output requested. Created attachment 1599041 [details]
journalctl.info
Based on my investigation this is a 2.11 specific issue. And, it could be a dup of bug 1672849. Using the same reproducer with OVS 2.9.0-114 got only one ERR. And, DPDK got initialized successfully, and OVS started successfully. 2019-08-01T20:38:14.053Z|00013|dpdk|INFO|EAL: Probing VFIO support... 2019-08-01T20:38:14.053Z|00014|dpdk|INFO|EAL: VFIO support initialized 2019-08-01T20:38:14.054Z|00015|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function. 2019-08-01T20:38:21.683Z|00016|dpdk|INFO|EAL: PCI device 0000:19:00.0 on NUMA socket 0 2019-08-01T20:38:21.683Z|00017|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2019-08-01T20:38:21.683Z|00018|dpdk|INFO|EAL: PCI device 0000:19:00.1 on NUMA socket 0 2019-08-01T20:38:21.683Z|00019|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2019-08-01T20:38:21.683Z|00020|dpdk|INFO|EAL: PCI device 0000:19:00.2 on NUMA socket 0 2019-08-01T20:38:21.683Z|00021|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2019-08-01T20:38:21.683Z|00022|dpdk|INFO|EAL: PCI device 0000:19:00.3 on NUMA socket 0 2019-08-01T20:38:21.683Z|00023|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2019-08-01T20:38:21.683Z|00024|dpdk|INFO|EAL: PCI device 0000:3b:00.0 on NUMA socket 0 2019-08-01T20:38:21.683Z|00025|dpdk|INFO|EAL: probe driver: 15b3:1013 net_mlx5 2019-08-01T20:38:21.684Z|00026|dpdk|INFO|PMD: net_mlx5: PCI information matches, using device "mlx5_0" (SR-IOV: false) 2019-08-01T20:38:21.687Z|00027|dpdk|INFO|PMD: net_mlx5: 1 port(s) detected 2019-08-01T20:38:21.690Z|00028|dpdk|INFO|PMD: net_mlx5: MPS is disabled 2019-08-01T20:38:21.691Z|00029|dpdk|INFO|PMD: net_mlx5: port 1 MAC address is 24:8a:07:87:22:ce 2019-08-01T20:38:21.693Z|00030|dpdk|INFO|EAL: PCI device 0000:3b:00.1 on NUMA socket 0 2019-08-01T20:38:21.693Z|00031|dpdk|INFO|EAL: probe driver: 15b3:1013 net_mlx5 2019-08-01T20:38:21.693Z|00032|dpdk|INFO|PMD: net_mlx5: PCI information matches, using device "mlx5_1" (SR-IOV: false) 2019-08-01T20:38:21.695Z|00033|dpdk|INFO|PMD: net_mlx5: 1 port(s) detected 2019-08-01T20:38:21.698Z|00034|dpdk|INFO|PMD: net_mlx5: MPS is disabled 2019-08-01T20:38:21.698Z|00035|dpdk|INFO|PMD: net_mlx5: port 1 MAC address is 24:8a:07:87:22:cf 2019-08-01T20:38:21.700Z|00036|dpdk|INFO|EAL: PCI device 0000:5e:00.0 on NUMA socket 0 2019-08-01T20:38:21.700Z|00037|dpdk|INFO|EAL: probe driver: 8086:158b net_i40e 2019-08-01T20:38:21.700Z|00038|dpdk|INFO|EAL: PCI device 0000:5e:00.1 on NUMA socket 0 2019-08-01T20:38:21.700Z|00039|dpdk|INFO|EAL: probe driver: 8086:158b net_i40e 2019-08-01T20:38:21.700Z|00040|dpdk|INFO|EAL: PCI device 0000:5f:00.0 on NUMA socket 0 2019-08-01T20:38:21.700Z|00041|dpdk|INFO|EAL: probe driver: 19ee:4000 net_nfp_pf 2019-08-01T20:38:21.701Z|00042|dpdk|INFO|DPDK Enabled - initialized Additional info: The driver in use for this test is ixgbe. The only other type NIC card/driver in the system under test is bnx2x. [root@netqe6 ~]# ethtool -i p3p1 driver: ixgbe version: 5.1.0-k-rh7.7 firmware-version: 0x80000389 expansion-rom-version: bus-info: 0000:83:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes To reiterate the initial notes for this bugs: This issue does NOT occur when running RHEL 7.6 (kernel 3.10.0-957.27.2.el7.x86_64) with either openvswitch-2.9.0-114.el7fdp.x86_64 or openvswitch2.11-2.11.0-18.el7fdp. It DOES occur with RHEL 7.7 (kernel 3.10.0-1062.el7.x86_64) and openvswitch2.11-2.11.0-18.el7fdp. In the testing I performed, this issue was specific to RHEL 7.7 (kernel 3.10.0-1062.el7.x86_64) and openvswitch2.11-2.11.0-18.el7fdp. NOTE: Got the same issue even with root as the user: #OVS_USER_ID="openvswitch:hugetlbfs" This is addressed with bz1711739. *** This bug has been marked as a duplicate of bug 1711739 *** |