+++ This bug was initially created as a clone of Bug #1769027 +++ Description of problem: [OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function [root@netqe24 dev]# tail -f /var/log/openvswitch/ovs-vswitchd.log -n 100 2019-11-05T18:21:12.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s) 2019-11-05T18:21:12.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes 2019-11-05T18:21:12.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket 2019-11-05T18:21:12.905Z|00017|dpdk|INFO|EAL: Probing VFIO support... 2019-11-05T18:21:13.118Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied 2019-11-05T18:21:13.118Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function. 2019-11-05T18:21:13.430Z|00020|dpdk|ERR|EAL: Cannot init memory 2019-11-05T18:21:13.430Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory 2019-11-05T18:21:15.526Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted)) 2019-11-05T18:21:15.526Z|00003|daemon_unix|EMER|could not detach from foreground session 2019-11-05T18:21:15.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2019-11-05T18:21:15.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0 2019-11-05T18:21:15.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1 2019-11-05T18:21:15.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores 2019-11-05T18:21:15.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2019-11-05T18:21:15.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2019-11-05T18:21:15.877Z|00007|dpdk|INFO|Using DPDK 18.11.2 2019-11-05T18:21:15.877Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2019-11-05T18:21:15.877Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2019-11-05T18:21:15.877Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2019-11-05T18:21:15.877Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2019-11-05T18:21:15.877Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled. 2019-11-05T18:21:15.877Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096. 2019-11-05T18:21:15.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s) 2019-11-05T18:21:15.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes 2019-11-05T18:21:15.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket 2019-11-05T18:21:15.905Z|00017|dpdk|INFO|EAL: Probing VFIO support... 2019-11-05T18:21:16.112Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied 2019-11-05T18:21:16.112Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function. 2019-11-05T18:21:16.425Z|00020|dpdk|ERR|EAL: Cannot init memory 2019-11-05T18:21:16.425Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory 2019-11-05T18:21:18.525Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted)) 2019-11-05T18:21:18.525Z|00003|daemon_unix|EMER|could not detach from foreground session 2019-11-05T18:21:18.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2019-11-05T18:21:18.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0 2019-11-05T18:21:18.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1 2019-11-05T18:21:18.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores 2019-11-05T18:21:18.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2019-11-05T18:21:18.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2019-11-05T18:21:18.878Z|00007|dpdk|INFO|Using DPDK 18.11.2 2019-11-05T18:21:18.878Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2019-11-05T18:21:18.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2019-11-05T18:21:18.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2019-11-05T18:21:18.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2019-11-05T18:21:18.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled. 2019-11-05T18:21:18.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096. 2019-11-05T18:21:18.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s) 2019-11-05T18:21:18.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes 2019-11-05T18:21:18.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket 2019-11-05T18:21:18.905Z|00017|dpdk|INFO|EAL: Probing VFIO support... 2019-11-05T18:21:19.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied 2019-11-05T18:21:19.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function. 2019-11-05T18:21:19.424Z|00020|dpdk|ERR|EAL: Cannot init memory 2019-11-05T18:21:19.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory 2019-11-05T18:21:21.532Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted)) 2019-11-05T18:21:21.532Z|00003|daemon_unix|EMER|could not detach from foreground session 2019-11-05T18:21:21.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2019-11-05T18:21:21.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0 2019-11-05T18:21:21.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1 2019-11-05T18:21:21.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores 2019-11-05T18:21:21.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2019-11-05T18:21:21.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2019-11-05T18:21:21.878Z|00007|dpdk|INFO|Using DPDK 18.11.2 2019-11-05T18:21:21.878Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2019-11-05T18:21:21.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2019-11-05T18:21:21.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2019-11-05T18:21:21.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2019-11-05T18:21:21.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled. 2019-11-05T18:21:21.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096. 2019-11-05T18:21:21.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s) 2019-11-05T18:21:21.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes 2019-11-05T18:21:21.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket 2019-11-05T18:21:21.906Z|00017|dpdk|INFO|EAL: Probing VFIO support... 2019-11-05T18:21:22.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied 2019-11-05T18:21:22.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function. 2019-11-05T18:21:22.424Z|00020|dpdk|ERR|EAL: Cannot init memory 2019-11-05T18:21:22.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory 2019-11-05T18:21:24.523Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted)) 2019-11-05T18:21:24.523Z|00003|daemon_unix|EMER|could not detach from foreground session 2019-11-05T18:21:24.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2019-11-05T18:21:24.877Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0 2019-11-05T18:21:24.877Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1 2019-11-05T18:21:24.877Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores 2019-11-05T18:21:24.877Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2019-11-05T18:21:24.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2019-11-05T18:21:24.878Z|00007|dpdk|INFO|Using DPDK 18.11.2 2019-11-05T18:21:24.878Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2019-11-05T18:21:24.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2019-11-05T18:21:24.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2019-11-05T18:21:24.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2019-11-05T18:21:24.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled. 2019-11-05T18:21:24.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096. 2019-11-05T18:21:24.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s) 2019-11-05T18:21:24.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes 2019-11-05T18:21:24.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket 2019-11-05T18:21:24.905Z|00017|dpdk|INFO|EAL: Probing VFIO support... 2019-11-05T18:21:25.116Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied 2019-11-05T18:21:25.116Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function. 2019-11-05T18:21:25.428Z|00020|dpdk|ERR|EAL: Cannot init memory 2019-11-05T18:21:25.428Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory Version-Release number of selected component (if applicable): [root@netqe24 dev]# rpm -q openvswitch2.12 openvswitch2.12-2.12.0-3.el8fdp.x86_64 [root@netqe24 dev]# uname -r 4.18.0-80.el8.x86_64 [root@netqe24 dev]# How reproducible: Reproducible Steps to Reproduce: 1. Run kernel/networking/ovs-dpdk-tunneling with mlx 2. 3. Actual results: Expected results: Additional info: --- Additional comment from David Marchand on 2019-11-05 21:10:23 CET --- This is the same as #1672849 which was closed as a duplicate of #1711739. Without the changes from this bz, dpdk will require access to physical addresses for the drivers which were not using a workaround in dpdk pci/mem subsystems (highjacking of a poorly documented IOVA_AS_VA flag). The mlx drivers were part of the drivers affected until the backports from #1711739 fixed the situation for our downstream ovs-2.11. --- Additional comment from Jean-Tsung Hsiao on 2019-11-05 21:12:27 CET --- Same issue for mlx(CX-5 & CX-3) under Rhel-7.7. --- Additional comment from Jean-Tsung Hsiao on 2019-11-05 21:24:11 CET --- (In reply to David Marchand from comment #1) > This is the same as #1672849 which was closed as a duplicate of #1711739. > > Without the changes from this bz, dpdk will require access to physical > addresses for the drivers which were not using a workaround in dpdk pci/mem > subsystems (highjacking of a poorly documented IOVA_AS_VA flag). > The mlx drivers were part of the drivers affected until the backports from > #1711739 fixed the situation for our downstream ovs-2.11. Definitely, a regression here --- from 2.11 to 2.12. --- Additional comment from David Marchand on 2019-11-05 21:35:51 CET --- Jean, I created this scratch build with the missing backports. Can you have a try? http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/openvswitch2.12/2.12.0/4.el7fdn/x86_64/ --- Additional comment from Jean-Tsung Hsiao on 2019-11-05 21:40:08 CET --- (In reply to David Marchand from comment #4) > Jean, > > I created this scratch build with the missing backports. > Can you have a try? > > http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/ > openvswitch2.12/2.12.0/4.el7fdn/x86_64/ Sure --- Additional comment from Jean-Tsung Hsiao on 2019-11-05 23:13:48 CET --- Hi David, The scratch build passed: https://beaker.engineering.redhat.com/jobs/3877907 Thanks! Jean --- Additional comment from David Marchand on 2019-11-06 06:35:38 CET --- Ok, thanks, I will push this to our fdn branch for rhel7 (using our ci).
The fix has been verified with https://beaker.engineering.redhat.com/jobs/3882949: https://beaker.engineering.redhat.com/jobs/3882949
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:4207