1769027 – [RHEL7][OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function

The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1769027 - [RHEL7][OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function

Summary: [RHEL7][OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Onl...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux Fast Datapath
Classification:	Red Hat
Component:	openvswitch2.12
Sub Component:
Version:	FDP 19.G
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	David Marchand
QA Contact:	Jean-Tsung Hsiao
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-11-05 18:44 UTC by Jean-Tsung Hsiao
Modified:	2019-12-11 12:15 UTC (History)
CC List:	8 users (show)
Fixed In Version:	openvswitch2.12-2.12.0-4.el7fdp
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1769392 (view as bug list)
Environment:
Last Closed:	2019-12-11 12:14:59 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:4206	0	None	None	None	2019-12-11 12:15:19 UTC

Description Jean-Tsung Hsiao 2019-11-05 18:44:38 UTC

Description of problem: [OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function

[root@netqe24 dev]# tail -f  /var/log/openvswitch/ovs-vswitchd.log -n 100
2019-11-05T18:21:12.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:12.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:12.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:12.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:13.118Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:13.118Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:13.430Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:13.430Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:15.526Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:15.526Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:15.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:15.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:15.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:15.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:15.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:15.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:15.877Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:15.877Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:15.877Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:15.877Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:15.877Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:15.877Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:15.877Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:15.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:15.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:15.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:15.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:16.112Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:16.112Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:16.425Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:16.425Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:18.525Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:18.525Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:18.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:18.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:18.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:18.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:18.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:18.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:18.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:18.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:18.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:18.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:18.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:18.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:18.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:18.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:18.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:18.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:18.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:19.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:19.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:19.424Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:19.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:21.532Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:21.532Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:21.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:21.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:21.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:21.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:21.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:21.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:21.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:21.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:21.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:21.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:21.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:21.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:21.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:21.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:21.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:21.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:21.906Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:22.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:22.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:22.424Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:22.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:24.523Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:24.523Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:24.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:24.877Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:24.877Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:24.877Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:24.877Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:24.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:24.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:24.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:24.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:24.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:24.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:24.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:24.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:24.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:24.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:24.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:24.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:25.116Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:25.116Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:25.428Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:25.428Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory

Version-Release number of selected component (if applicable):

[root@netqe24 dev]# rpm -q openvswitch2.12
openvswitch2.12-2.12.0-3.el8fdp.x86_64
[root@netqe24 dev]# uname -r
4.18.0-80.el8.x86_64
[root@netqe24 dev]#

How reproducible: Reproducible


Steps to Reproduce:
1. Run kernel/networking/ovs-dpdk-tunneling with mlx
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 David Marchand 2019-11-05 20:10:23 UTC

This is the same as #1672849 which was closed as a duplicate of #1711739.

Without the changes from this bz, dpdk will require access to physical addresses for the drivers which were not using a workaround in dpdk pci/mem subsystems (highjacking of a poorly documented IOVA_AS_VA flag).
The mlx drivers were part of the drivers affected until the backports from #1711739 fixed the situation for our downstream ovs-2.11.

Comment 2 Jean-Tsung Hsiao 2019-11-05 20:12:27 UTC

Same issue for mlx(CX-5 & CX-3) under Rhel-7.7.

Comment 3 Jean-Tsung Hsiao 2019-11-05 20:24:11 UTC

(In reply to David Marchand from comment #1)
> This is the same as #1672849 which was closed as a duplicate of #1711739.
> 
> Without the changes from this bz, dpdk will require access to physical
> addresses for the drivers which were not using a workaround in dpdk pci/mem
> subsystems (highjacking of a poorly documented IOVA_AS_VA flag).
> The mlx drivers were part of the drivers affected until the backports from
> #1711739 fixed the situation for our downstream ovs-2.11.

Definitely, a regression here --- from 2.11 to 2.12.

Comment 4 David Marchand 2019-11-05 20:35:51 UTC

Jean,

I created this scratch build with the missing backports.
Can you have a try?

http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/openvswitch2.12/2.12.0/4.el7fdn/x86_64/

Comment 5 Jean-Tsung Hsiao 2019-11-05 20:40:08 UTC

(In reply to David Marchand from comment #4)
> Jean,
> 
> I created this scratch build with the missing backports.
> Can you have a try?
> 
> http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/
> openvswitch2.12/2.12.0/4.el7fdn/x86_64/

Sure

Comment 6 Jean-Tsung Hsiao 2019-11-05 22:13:48 UTC

Hi David,
The scratch build passed: https://beaker.engineering.redhat.com/jobs/3877907
Thanks!
Jean

Comment 7 David Marchand 2019-11-06 05:35:38 UTC

Ok, thanks, I will push this to our fdn branch for rhel7 (using our ci).

Comment 9 Jean-Tsung Hsiao 2019-11-15 20:19:54 UTC

The fix has been verified with openvswitch2.12-2.12.0-4.el7fdp:

https://beaker.engineering.redhat.com/jobs/3882404

Comment 11 errata-xmlrpc 2019-12-11 12:14:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4206

Note You need to log in before you can comment on or make changes to this bug.