Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1531145

Summary: QLogic 41262H card fails to add to OVS DPDK with RAMROD error
Product: Red Hat Enterprise Linux 7 Reporter: Christian Trautman <ctrautma>
Component: openvswitchAssignee: Matteo Croce <mcroce>
Status: CLOSED ERRATA QA Contact: Christian Trautman <ctrautma>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.5CC: ameen.rahman, arahman, atragler, ctrautma, fbaudin, Harish.Patil, hpatil, jean-mickael.guerin, jeroen.van_bemmel, jhsiao, kfida, mcroce, ovs-qe, pvauter, qding, rasesh.mody, tredaelli
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openvswitch-2.9.0-31.el7fdn Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-15 13:53:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sosreport
none
QEDE PMD MTU fix none

Description Christian Trautman 2018-01-04 16:32:28 UTC
Created attachment 1377021 [details]
sosreport

Description of problem:
Qede 41262H card will not add port to OVS-DPDK. Reports RAMROD error.


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Make sure correct firmware file is in place /lib/firmware/qed/qed_init_values-8.30.12.0.bin
2. set Qlogic ISCSI and Fcoe ports to bind to driver none using driverctl
3. set Qlogic network ports to vfio-pci driver using driverctl
4. Add netdev bridge and try to add a Qlogic port as dpdk port to bridge

driverctl -v set-override 0000:05:00.0 vfio-pci
driverctl -v set-override 0000:05:00.1 vfio-pci
driverctl -v set-override 0000:05:00.2 none
driverctl -v set-override 0000:05:00.3 none
driverctl -v set-override 0000:05:00.4 none
driverctl -v set-override 0000:05:00.5 none

systemctl start openvswitch
PMDMASK=<appropriate PMD mask>

systemctl start openvswitch
ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024,1024"
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=$PMDMASK
ovs-vsctl add-port ovsbr0 dpdk0 -- set interface dpdk0 type=dpdk ofport_request=10 options:dpdk-devargs=0000:05:00.0

ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device '0000:05:00.0' to DPDK.  See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".


Actual results:
Log reports
Jan 04 11:18:08 netqe22.knqe.lab.eng.bos.redhat.com ovs-vswitchd[9369]: ovs|00052|dpdk|ERR|[QEDE PMD: (05:00.0:dpdk-port-0-0)]ecore_spq_block:Ramrod is stuck [CID ff000000 cmd 02 proto 04 echo 0004]
Jan 04 11:18:08 netqe22.knqe.lab.eng.bos.redhat.com ovs-vswitchd[9369]: ovs|00053|dpdk|ERR|[qede_hw_err_notify:292(05:00.0:dpdk-port-0-0)]HW error occurred [Ramrod Failure]
Jan 04 11:18:08 netqe22.knqe.lab.eng.bos.redhat.com ovs-vswitchd[9369]: ovs|00054|dpdk|ERR|[qede_activate_vport:469(05:00.0:dpdk-port-0)]Failed to update vport


Expected results:
Can add dpdk port for QLogic correctly.

Additional info:
driver: qede
version: 8.10.10.21
firmware-version: mfw 8.18.18.0 storm 8.20.0.0
expansion-rom-version: 
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes

openvswitch-2.9.0-0.2.20171212git6625e43.el7fdb.x86_64
dpdk-tools-17.11-2.el7fdb.x86_64
dpdk-17.11-2.el7fdb.x86_64
driverctl-0.95-1.el7fdparch.noarch

Will check 45000 series card today as well.

Comment 2 Christian Trautman 2018-01-04 16:51:17 UTC
Same error on QLogic 45000 25G card.

Comment 3 Rasesh Mody 2018-01-27 22:01:30 UTC
Created attachment 1387046 [details]
QEDE PMD MTU fix

The attached patch is a fix for QEDE PMD submitted to dpdk.org for inclusion in DPDK 18.02 and DPDK 17.11 stable. The patch is back ported to apply cleanly on DPDK 17.11 GA. Please apply the patch and re-test.

Comment 4 Ameen Rahman 2018-01-27 22:57:45 UTC
Hi Chris,

Could you please try this patch and let us know if this fix the issue you reported.

Thanks,
-Ameen

Comment 5 Matteo Croce 2018-02-05 19:17:30 UTC
I will build an RPM and test on proper hardware.

thanks
Matteo

Comment 6 Anita Tragler 2018-02-06 16:02:53 UTC
HI Ameen, Rasesh

Please post results for PVP/Vsperf tests for OVS-DPDK with QEDE with this patch including configuration steps.
We will in turn wait for rpm build in fast datapth channel for Chris and QE to test.
Thanks

Comment 8 Ameen Rahman 2018-02-07 03:38:32 UTC
Hi Anita,

We are running into issues with the PVP/VSPerf tests. Due to this we are not able to complete some tests. Rasesh is working with Chris on this.

Thanks,
-Ameen

Comment 10 Christian Trautman 2018-02-07 19:47:38 UTC
Verified using test build from Matteo. Could add ports successfully.

Comment 11 Matteo Croce 2018-02-08 00:14:14 UTC
Great!

I see that the patch are already merged in dpdk-next-net, will backport after they enter stable

Comment 12 Ameen Rahman 2018-02-09 23:58:59 UTC
Hi Matteo,

This patch is in the stable tree already.

http://dpdk.org/browse/dpdk-stable/commit/?h=17.11&id=40a6c11a287ef759464fdbe8061c7b7a3bb76498

Thanks.

Comment 13 Jean-Tsung Hsiao 2018-02-13 03:13:32 UTC
(In reply to Christian Trautman from comment #10)
> Verified using test build from Matteo. Could add ports successfully.

Also, verified for QL45212H 25 Gb adapter.

Comment 14 Ameen Rahman 2018-03-03 02:49:29 UTC
DPDK 17.11.1 release includes this fix.

Comment 15 Matteo Croce 2018-03-05 13:33:46 UTC
Got it Ameen

Comment 16 Timothy Redaelli 2018-05-03 15:00:18 UTC

*** This bug has been marked as a duplicate of bug 1494616 ***

Comment 17 Jeroen van Bemmel 2018-05-24 17:38:55 UTC
On what basis is bug 1494616 considered a duplicate? Seems like a different issue to me

Comment 18 Timothy Redaelli 2018-06-15 16:09:45 UTC
Sorry, I misread a comment on the linked bug (since the patch is the same for the 2 bugs).
Reopening.

Comment 20 Christian Trautman 2018-07-11 11:56:44 UTC
Verified using ovs version openvswitch-2.9.0-54.el7fdp.x86_64

dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:05:00.0 'FastLinQ QL41000 Series 10/25/40/50GbE Controller 8070' drv=vfio-pci unused=
0000:05:00.1 'FastLinQ QL41000 Series 10/25/40/50GbE Controller 8070' drv=vfio-pci unused=


ovs-vsctl show
8c9227cc-87cc-484d-9c38-32d80cd94f4c
    Bridge "ovsbr0"
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {dpdk-devargs="0000:05:00.0"}
        Port "dpdk1"
            Interface "dpdk1"
                type: dpdk
                options: {dpdk-devargs="0000:05:00.1"}
        Port "vhost0"
            Interface "vhost0"
                type: dpdkvhostuser
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "vhost1"
            Interface "vhost1"
                type: dpdkvhostuser
    ovs_version: "2.9.0"

Kernel version used -> 3.10.0-862.el7.x86_64 (7.5 release)

Comment 21 Timothy Redaelli 2018-08-10 13:45:33 UTC
The openvwitch component is delivered through the fast datapath channel, it is not documented in release notes.

Comment 23 errata-xmlrpc 2018-08-15 13:53:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2432