Bug 1769392

Summary:	[RHEL8] [OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	Timothy Redaelli <tredaelli>
Component:	openvswitch2.12	Assignee:	David Marchand <dmarchan>
Status:	CLOSED ERRATA	QA Contact:	Jean-Tsung Hsiao <jhsiao>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	FDP 19.G	CC:	aconole, ctrautma, dmarchan, fhallal, jhsiao, kfida, qding, ralongi
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openvswitch2.12-2.12.0-4.el8fdp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1769027	Environment:
Last Closed:	2019-12-11 12:06:53 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Timothy Redaelli 2019-11-06 14:30:17 UTC

+++ This bug was initially created as a clone of Bug #1769027 +++

Description of problem: [OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function

[root@netqe24 dev]# tail -f  /var/log/openvswitch/ovs-vswitchd.log -n 100
2019-11-05T18:21:12.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:12.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:12.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:12.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:13.118Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:13.118Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:13.430Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:13.430Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:15.526Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:15.526Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:15.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:15.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:15.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:15.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:15.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:15.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:15.877Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:15.877Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:15.877Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:15.877Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:15.877Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:15.877Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:15.877Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:15.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:15.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:15.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:15.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:16.112Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:16.112Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:16.425Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:16.425Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:18.525Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:18.525Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:18.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:18.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:18.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:18.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:18.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:18.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:18.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:18.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:18.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:18.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:18.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:18.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:18.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:18.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:18.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:18.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:18.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:19.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:19.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:19.424Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:19.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:21.532Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:21.532Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:21.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:21.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:21.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:21.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:21.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:21.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:21.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:21.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:21.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:21.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:21.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:21.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:21.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:21.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:21.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:21.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:21.906Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:22.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:22.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:22.424Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:22.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:24.523Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:24.523Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:24.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:24.877Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:24.877Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:24.877Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:24.877Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:24.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:24.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:24.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:24.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:24.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:24.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:24.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:24.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:24.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:24.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:24.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:24.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:25.116Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:25.116Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:25.428Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:25.428Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory

Version-Release number of selected component (if applicable):

[root@netqe24 dev]# rpm -q openvswitch2.12
openvswitch2.12-2.12.0-3.el8fdp.x86_64
[root@netqe24 dev]# uname -r
4.18.0-80.el8.x86_64
[root@netqe24 dev]#

How reproducible: Reproducible


Steps to Reproduce:
1. Run kernel/networking/ovs-dpdk-tunneling with mlx
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from David Marchand on 2019-11-05 21:10:23 CET ---

This is the same as #1672849 which was closed as a duplicate of #1711739.

Without the changes from this bz, dpdk will require access to physical addresses for the drivers which were not using a workaround in dpdk pci/mem subsystems (highjacking of a poorly documented IOVA_AS_VA flag).
The mlx drivers were part of the drivers affected until the backports from #1711739 fixed the situation for our downstream ovs-2.11.

--- Additional comment from Jean-Tsung Hsiao on 2019-11-05 21:12:27 CET ---

Same issue for mlx(CX-5 & CX-3) under Rhel-7.7.

--- Additional comment from Jean-Tsung Hsiao on 2019-11-05 21:24:11 CET ---

(In reply to David Marchand from comment #1)
> This is the same as #1672849 which was closed as a duplicate of #1711739.
> 
> Without the changes from this bz, dpdk will require access to physical
> addresses for the drivers which were not using a workaround in dpdk pci/mem
> subsystems (highjacking of a poorly documented IOVA_AS_VA flag).
> The mlx drivers were part of the drivers affected until the backports from
> #1711739 fixed the situation for our downstream ovs-2.11.

Definitely, a regression here --- from 2.11 to 2.12.

--- Additional comment from David Marchand on 2019-11-05 21:35:51 CET ---

Jean,

I created this scratch build with the missing backports.
Can you have a try?

http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/openvswitch2.12/2.12.0/4.el7fdn/x86_64/

--- Additional comment from Jean-Tsung Hsiao on 2019-11-05 21:40:08 CET ---

(In reply to David Marchand from comment #4)
> Jean,
> 
> I created this scratch build with the missing backports.
> Can you have a try?
> 
> http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/
> openvswitch2.12/2.12.0/4.el7fdn/x86_64/

Sure

--- Additional comment from Jean-Tsung Hsiao on 2019-11-05 23:13:48 CET ---

Hi David,
The scratch build passed: https://beaker.engineering.redhat.com/jobs/3877907
Thanks!
Jean

--- Additional comment from David Marchand on 2019-11-06 06:35:38 CET ---

Ok, thanks, I will push this to our fdn branch for rhel7 (using our ci).

Comment 2 Jean-Tsung Hsiao 2019-11-15 20:21:47 UTC

The fix has been verified with https://beaker.engineering.redhat.com/jobs/3882949:

https://beaker.engineering.redhat.com/jobs/3882949

Comment 4 errata-xmlrpc 2019-12-11 12:06:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4207