Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1769392

Summary:	[RHEL8] [OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	Timothy Redaelli <tredaelli>
Component:	openvswitch2.12	Assignee:	David Marchand <dmarchan>
Status:	CLOSED ERRATA	QA Contact:	Jean-Tsung Hsiao <jhsiao>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	FDP 19.G	CC:	aconole, ctrautma, dmarchan, fhallal, jhsiao, kfida, qding, ralongi
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openvswitch2.12-2.12.0-4.el8fdp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1769027	Environment:
Last Closed:	2019-12-11 12:06:53 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Timothy Redaelli 2019-11-06 14:30:17 UTC

+++ This bug was initially created as a clone of Bug #1769027 +++

Description of problem: [OVS2.12][mlx]Cannot obtain physical addresses: Permission denied. Only vfio will function

[root@netqe24 dev]# tail -f  /var/log/openvswitch/ovs-vswitchd.log -n 100
2019-11-05T18:21:12.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:12.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:12.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:12.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:13.118Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:13.118Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:13.430Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:13.430Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:15.526Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:15.526Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:15.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:15.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:15.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:15.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:15.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:15.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:15.877Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:15.877Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:15.877Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:15.877Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:15.877Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:15.877Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:15.877Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:15.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:15.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:15.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:15.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:16.112Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:16.112Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:16.425Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:16.425Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:18.525Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:18.525Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:18.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:18.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:18.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:18.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:18.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:18.876Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:18.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:18.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:18.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:18.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:18.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:18.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:18.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:18.881Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:18.881Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:18.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:18.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:19.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:19.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:19.424Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:19.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:21.532Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:21.532Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:21.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:21.876Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:21.876Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:21.876Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:21.876Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:21.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:21.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:21.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:21.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:21.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:21.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:21.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:21.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:21.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:21.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:21.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:21.906Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:22.113Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:22.113Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:22.424Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:22.424Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory
2019-11-05T18:21:24.523Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted))
2019-11-05T18:21:24.523Z|00003|daemon_unix|EMER|could not detach from foreground session
2019-11-05T18:21:24.868Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2019-11-05T18:21:24.877Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2019-11-05T18:21:24.877Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2019-11-05T18:21:24.877Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2019-11-05T18:21:24.877Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2019-11-05T18:21:24.877Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2019-11-05T18:21:24.878Z|00007|dpdk|INFO|Using DPDK 18.11.2
2019-11-05T18:21:24.878Z|00008|dpdk|INFO|DPDK Enabled - initializing...
2019-11-05T18:21:24.878Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2019-11-05T18:21:24.878Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled.
2019-11-05T18:21:24.878Z|00011|dpdk|INFO|POSTCOPY support for vhost-user-client disabled.
2019-11-05T18:21:24.878Z|00012|dpdk|INFO|Per port memory for DPDK devices disabled.
2019-11-05T18:21:24.878Z|00013|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x000f --socket-mem 4096,4096 --socket-limit 4096,4096.
2019-11-05T18:21:24.882Z|00014|dpdk|INFO|EAL: Detected 48 lcore(s)
2019-11-05T18:21:24.882Z|00015|dpdk|INFO|EAL: Detected 2 NUMA nodes
2019-11-05T18:21:24.883Z|00016|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket
2019-11-05T18:21:24.905Z|00017|dpdk|INFO|EAL: Probing VFIO support...
2019-11-05T18:21:25.116Z|00018|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied
2019-11-05T18:21:25.116Z|00019|dpdk|ERR|EAL: Cannot obtain physical addresses: Permission denied. Only vfio will function.
2019-11-05T18:21:25.428Z|00020|dpdk|ERR|EAL: Cannot init memory
2019-11-05T18:21:25.428Z|00021|dpdk|EMER|Unable to initialize DPDK: Cannot allocate memory

Version-Release number of selected component (if applicable):

[root@netqe24 dev]# rpm -q openvswitch2.12
openvswitch2.12-2.12.0-3.el8fdp.x86_64
[root@netqe24 dev]# uname -r
4.18.0-80.el8.x86_64
[root@netqe24 dev]#

How reproducible: Reproducible


Steps to Reproduce:
1. Run kernel/networking/ovs-dpdk-tunneling with mlx
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from David Marchand on 2019-11-05 21:10:23 CET ---

This is the same as #1672849 which was closed as a duplicate of #1711739.

Without the changes from this bz, dpdk will require access to physical addresses for the drivers which were not using a workaround in dpdk pci/mem subsystems (highjacking of a poorly documented IOVA_AS_VA flag).
The mlx drivers were part of the drivers affected until the backports from #1711739 fixed the situation for our downstream ovs-2.11.

--- Additional comment from Jean-Tsung Hsiao on 2019-11-05 21:12:27 CET ---

Same issue for mlx(CX-5 & CX-3) under Rhel-7.7.

--- Additional comment from Jean-Tsung Hsiao on 2019-11-05 21:24:11 CET ---

(In reply to David Marchand from comment #1)
> This is the same as #1672849 which was closed as a duplicate of #1711739.
> 
> Without the changes from this bz, dpdk will require access to physical
> addresses for the drivers which were not using a workaround in dpdk pci/mem
> subsystems (highjacking of a poorly documented IOVA_AS_VA flag).
> The mlx drivers were part of the drivers affected until the backports from
> #1711739 fixed the situation for our downstream ovs-2.11.

Definitely, a regression here --- from 2.11 to 2.12.

--- Additional comment from David Marchand on 2019-11-05 21:35:51 CET ---

Jean,

I created this scratch build with the missing backports.
Can you have a try?

http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/openvswitch2.12/2.12.0/4.el7fdn/x86_64/

--- Additional comment from Jean-Tsung Hsiao on 2019-11-05 21:40:08 CET ---

(In reply to David Marchand from comment #4)
> Jean,
> 
> I created this scratch build with the missing backports.
> Can you have a try?
> 
> http://brew-task-repos.usersys.redhat.com/repos/scratch/dmarchan/
> openvswitch2.12/2.12.0/4.el7fdn/x86_64/

Sure

--- Additional comment from Jean-Tsung Hsiao on 2019-11-05 23:13:48 CET ---

Hi David,
The scratch build passed: https://beaker.engineering.redhat.com/jobs/3877907
Thanks!
Jean

--- Additional comment from David Marchand on 2019-11-06 06:35:38 CET ---

Ok, thanks, I will push this to our fdn branch for rhel7 (using our ci).

Comment 2 Jean-Tsung Hsiao 2019-11-15 20:21:47 UTC

The fix has been verified with https://beaker.engineering.redhat.com/jobs/3882949:

https://beaker.engineering.redhat.com/jobs/3882949

Comment 4 errata-xmlrpc 2019-12-11 12:06:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4207