Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1769027

Summary: [RHEL7][OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Jean-Tsung Hsiao <jhsiao>
Component: openvswitch2.12Assignee: David Marchand <dmarchan>
Status: CLOSED ERRATA QA Contact: Jean-Tsung Hsiao <jhsiao>
Severity: medium Docs Contact:
Priority: medium    
Version: FDP 19.GCC: aconole, ctrautma, dmarchan, fhallal, jhsiao, kfida, ralongi, tredaelli
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch2.12-2.12.0-4.el7fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1769392 (view as bug list) Environment:
Last Closed: 2019-12-11 12:14:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jean-Tsung Hsiao 2019-11-05 18:44:38 UTC
Description of problem: [OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function

[root@netqe24 dev]# tail -f  /var/log/openvswitch/ovs-vswitchd.log -n 100
2019-11-05T18:21:12.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:12.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:12.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:12.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:13.118Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:13.118Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:13.430Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:13.430Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:15.526Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:15.526Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:15.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:15.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:15.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:15.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:15.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:15.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:15.877Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:15.877Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:15.877Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:15.877Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:15.877Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:15.877Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:15.877Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:15.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:15.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:15.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:15.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:16.112Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:16.112Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:16.425Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:16.425Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:18.525Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:18.525Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:18.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:18.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:18.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:18.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:18.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:18.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:18.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:18.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:18.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:18.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:18.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:18.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:18.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:18.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:18.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:18.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:18.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:19.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:19.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:19.424Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:19.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:21.532Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:21.532Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:21.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:21.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:21.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:21.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:21.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:21.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:21.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:21.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:21.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:21.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:21.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:21.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:21.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:21.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:21.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:21.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:21.906Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:22.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:22.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:22.424Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:22.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:24.523Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:24.523Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:24.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:24.877Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:24.877Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:24.877Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:24.877Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:24.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:24.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:24.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:24.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:24.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:24.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:24.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:24.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:24.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:24.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:24.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:24.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:25.116Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:25.116Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:25.428Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:25.428Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory

Version-Release number of selected component (if applicable):

[root@netqe24 dev]# rpm -q openvswitch2.12
openvswitch2.12-2.12.0-3.el8fdp.x86_64
[root@netqe24 dev]# uname -r
4.18.0-80.el8.x86_64
[root@netqe24 dev]#

How reproducible: Reproducible


Steps to Reproduce:
1. Run kernel/networking/ovs-dpdk-tunneling with mlx
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 David Marchand 2019-11-05 20:10:23 UTC
This is the same as #1672849 which was closed as a duplicate of #1711739.

Without the changes from this bz, dpdk will require access to physical addresses for the drivers which were not using a workaround in dpdk pci/mem subsystems (highjacking of a poorly documented IOVA_AS_VA flag).
The mlx drivers were part of the drivers affected until the backports from #1711739 fixed the situation for our downstream ovs-2.11.

Comment 2 Jean-Tsung Hsiao 2019-11-05 20:12:27 UTC
Same issue for mlx(CX-5 & CX-3) under Rhel-7.7.

Comment 3 Jean-Tsung Hsiao 2019-11-05 20:24:11 UTC
(In reply to David Marchand from comment #1)
> This is the same as #1672849 which was closed as a duplicate of #1711739.
> 
> Without the changes from this bz, dpdk will require access to physical
> addresses for the drivers which were not using a workaround in dpdk pci/mem
> subsystems (highjacking of a poorly documented IOVA_AS_VA flag).
> The mlx drivers were part of the drivers affected until the backports from
> #1711739 fixed the situation for our downstream ovs-2.11.

Definitely, a regression here --- from 2.11 to 2.12.

Comment 4 David Marchand 2019-11-05 20:35:51 UTC
Jean,

I created this scratch build with the missing backports.
Can you have a try?

http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/openvswitch2.12/2.12.0/4.el7fdn/x86_64/

Comment 5 Jean-Tsung Hsiao 2019-11-05 20:40:08 UTC
(In reply to David Marchand from comment #4)
> Jean,
> 
> I created this scratch build with the missing backports.
> Can you have a try?
> 
> http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/
> openvswitch2.12/2.12.0/4.el7fdn/x86_64/

Sure

Comment 6 Jean-Tsung Hsiao 2019-11-05 22:13:48 UTC
Hi David,
The scratch build passed: https://beaker.engineering.redhat.com/jobs/3877907
Thanks!
Jean

Comment 7 David Marchand 2019-11-06 05:35:38 UTC
Ok, thanks, I will push this to our fdn branch for rhel7 (using our ci).

Comment 9 Jean-Tsung Hsiao 2019-11-15 20:19:54 UTC
The fix has been verified with openvswitch2.12-2.12.0-4.el7fdp:

https://beaker.engineering.redhat.com/jobs/3882404

Comment 11 errata-xmlrpc 2019-12-11 12:14:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4206