Bug 1478678 - DPDK CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF and queue_idx out of range issue with X710 (i40e PMD) cards and more than 64 cores per host
DPDK CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF and queue_idx out of range issue...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openvswitch-dpdk (Show other bugs)
10.0 (Newton)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Kevin Traynor
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-05 21:48 EDT by Andreas Karis
Modified: 2017-08-10 10:52 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-10 10:52:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Andreas Karis 2017-08-05 21:48:26 EDT
Description of problem:
DPDK CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF and queue_idx out of range issue with X710 (i40e PMD) cards and more than 64 cores per host

Version-Release number of selected component (if applicable):
OVS 2.6.1 with DPDK 16.11

Additional info:
We need to find a fix for this. E.g., on a customer system with 88 cores, the following error message will show up on PMD initialization:
~~~
Jul 28 16:15:53 netqe23.knqe.lab.eng.bos.redhat.com ovs-vswitchd[8125]: PMD: i40e_pf_get_vsi_by_qindex(): queue_idx out of range. VMDQ configured?
~~~

* we are hitting: `i40e_pf_get_vsi_by_qindex(): queue_idx out of range. VMDQ configured?` => according to Intel, need to set in this particular case: 
~~~
CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF=88
~~~
http://openvswitch.org/support/dist-docs-2.5/INSTALL.DPDK.md.html
"By default with DPDK 2.2, a maximum of 64 TX queues can be used with an Intel XL710 Network Interface on a platform with more than 64 logical cores. If a user attempts to add an XL710 interface as a DPDK port type to a system as described above, an error will be reported that initialization failed for the 65th queue. OVS will then roll back to the previous successful queue initialization and use that value as the total number of TX queues available with queue locking. If a user wishes to use more than 64 queues and avoid locking, then the CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF config parameter in DPDK must be increased to the desired number of queues. Both DPDK and OVS must be recompiled for this change to take effect."
Comment 4 Kevin Traynor 2017-08-10 08:23:30 EDT
OVS-DPDK is designed to cater for this case. In OVS 2.6 the num of tx queue's requested = num cores + 1.

In the case where there is 72 cores, OVS will attempt to setup 73 txq's on the i40e:
- txq's 0-63 will setup correctly
- txq 64 will fail and show the reported log message

PMD: i40e_pf_get_vsi_by_qindex(): queue_idx out of range. VMDQ configured?

Then OVS-DPDK will reconfigure the i40e, setup 64 queues and works with this amount.

2017-08-10T11:19:31Z|00027|dpdk|INFO|Interface dpdk0 txq(64) setup error: Input/output error
2017-08-10T11:19:31Z|00028|dpdk|INFO|Retrying setup with (rxq:1 txq:64)
Comment 6 Andreas Karis 2017-08-10 10:52:43 EDT
Closing

https://access.redhat.com/solutions/3147281

Note You need to log in before you can comment on or make changes to this bug.