Bug 2179024

Summary:	[dpdk] cannot use E810 nics in RHEL9
Product:	Red Hat Enterprise Linux 9	Reporter:	Robin Jarry <rjarry>
Component:	dpdk	Assignee:	Timothy Redaelli <tredaelli>
Status:	CLOSED ERRATA	QA Contact:	Hekai Wang <hewang>
Severity:	high	Docs Contact:
Priority:	high
Version:	9.2	CC:	dmarchan, jhsiao, lmiksik, mhou, ovs-qe, pvlasin, tredaelli
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	dpdk-22.11-3.el9_2	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2181250 (view as bug list)		Environment:
Last Closed:	2023-05-09 07:53:59 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2181250

Description Robin Jarry 2023-03-16 13:30:31 UTC

Description of problem:

dpdk-testpmd cannot run with E810 nics.

Version-Release number of selected component (if applicable):

dpdk-22.11-1.el9.x86_64

How reproducible:

With intel E810 adapters.

Steps to Reproduce:
1. try to run testpmd with E810 nics.

Actual results:

EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:3b:00.0 (socket 0)
EAL: libarchive not linked, /lib/firmware/intel/ice/ddp/ice.pkg.xz cannot be decompressed
ice_load_pkg(): failed to search file path
ice_dev_init(): Failed to load the DDP package,Use safe-mode-support=1 to enter Safe Mode
EAL: Releasing PCI mapped resource for 0000:3b:00.0
EAL: Calling pci_unmap_resource for 0000:3b:00.0 at 0x2200000000
EAL: Calling pci_unmap_resource for 0000:3b:00.0 at 0x2202000000
EAL: Requested device 0000:3b:00.0 cannot be used

Expected results:

The devices should be initialized without errors.

Additional info:

testpmd is not linked with libarchive. Probably missing a build requires: libarchive-devel.

Comment 1 Timothy Redaelli 2023-03-16 13:36:00 UTC

This is easily fixable by just enabling loading compressed firmware (BuildRequires: libarchive-devel).
We already did and tested that with DPDK inside OVS (on fast-datapath-rhel-9), so I'll ask for exception

Comment 3 Jean-Tsung Hsiao 2023-03-16 19:25:48 UTC

It is working for me:

[root@wsfd-advnetlab36 ddp]# !49
rpm -q dpdk
dpdk-22.11-1.el9.x86_64
[root@wsfd-advnetlab36 ddp]# dpdk-testpmd -a 0000:3b:00.0 -a 0000:3b:00.1 -- -i 
EAL: Detected CPU lcores: 40
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_i40e (8086:158b) device: 0000:3b:00.0 (socket 0)
EAL: Probe PCI driver: net_i40e (8086:158b) device: 0000:3b:00.1 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=459456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mb_pool_1>: n=459456, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: 3C:FD:FE:EA:F8:10
Configuring Port 1 (socket 0)
Port 1: 3C:FD:FE:EA:F8:11
Checking link statuses...
Done
testpmd> quit

Stopping port 0...
Stopping ports...
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
Port 1 is closed
Done

Bye...
[root@wsfd-advnetlab36 ddp]# cat /etc/yum.repos.d/beaker-BaseOS.repo 
[beaker-BaseOS]
name=beaker-BaseOS
baseurl=http://download.hosts.prod.psi.bos.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-20230228.28/compose/BaseOS/x86_64/os
enabled=1
gpgcheck=0
skip_if_unavailable=1
[root@wsfd-advnetlab36 ddp]# uname -r
5.14.0-283.el9.x86_64
[root@wsfd-advnetlab36 ddp]#

Comment 4 Jean-Tsung Hsiao 2023-03-16 19:44:02 UTC

Sorry, I was looking at wrong system and wrong NIC.

Comment 5 Robin Jarry 2023-03-16 20:02:40 UTC

Here is some additional info if needed:

+ lspci -vvvs 0000:3b:00.0
3b:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)
	Subsystem: Intel Corporation Ethernet Network Adapter E810-C-Q2
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 774
	NUMA node: 0
	IOMMU group: 59
	Region 0: Memory at ae000000 (64-bit, prefetchable) [size=32M]
	Region 3: Memory at b2010000 (64-bit, prefetchable) [size=64K]
	Expansion ROM at ab000000 [disabled] [size=1M]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=1024 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00008000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM not supported
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (downgraded), Width x8 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR-
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- LTR- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [e0] Vital Product Data
		Product Name: Intel(r) Ethernet Network Adapter E810-CQDA2
		Read-only fields:
			[V1] Vendor specific: Intel(r) Ethernet Network Adapter E810-CQDA2
			[PN] Part number: K91258-004
			[SN] Serial number: 40A6B718E060
			[V2] Vendor specific: 2020
			[RV] Reserved: checksum good, 0 byte(s) reserved
		End
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [150 v1] Device Serial Number 60-e0-18-ff-ff-b7-a6-40
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 128, Total VFs: 128, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 8, stride: 1, Device ID: 1889
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000b1000000 (64-bit, prefetchable)
		Region 3: Memory at 00000000b2220000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [1a0 v1] Transaction Processing Hints
		Device specific mode supported
		No steering table available
	Capabilities: [1b0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [1d0 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [200 v1] Data Link Feature <?>
	Capabilities: [210 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [250 v1] Lane Margining at the Receiver <?>
	Kernel driver in use: ice
	Kernel modules: ice

+ lspci -vvvs 0000:3b:00.1
3b:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)
	Subsystem: Intel Corporation Ethernet Network Adapter E810-C-Q2
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 774
	NUMA node: 0
	IOMMU group: 60
	Region 0: Memory at ac000000 (64-bit, prefetchable) [size=32M]
	Region 3: Memory at b2000000 (64-bit, prefetchable) [size=64K]
	Expansion ROM at ab100000 [disabled] [size=1M]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=1024 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00008000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM not supported
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (downgraded), Width x8 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR-
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- LTR- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [e0] Vital Product Data
		Product Name: Intel(r) Ethernet Network Adapter E810-CQDA2
		Read-only fields:
			[V1] Vendor specific: Intel(r) Ethernet Network Adapter E810-CQDA2
			[PN] Part number: K91258-004
			[SN] Serial number: 40A6B718E060
			[V2] Vendor specific: 2020
			[RV] Reserved: checksum good, 0 byte(s) reserved
		End
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [150 v1] Device Serial Number 60-e0-18-ff-ff-b7-a6-40
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 128, Total VFs: 128, Number of VFs: 0, Function Dependency Link: 01
		VF offset: 135, stride: 1, Device ID: 1889
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000b0000000 (64-bit, prefetchable)
		Region 3: Memory at 00000000b2020000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [1a0 v1] Transaction Processing Hints
		Device specific mode supported
		No steering table available
	Capabilities: [1b0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [200 v1] Data Link Feature <?>
	Kernel driver in use: ice
	Kernel modules: ice

+ ethtool -i ens1f0
driver: ice
version: 5.14.0-283.el9.x86_64
firmware-version: 4.20 0x80017785 1.3346.0
expansion-rom-version: 
bus-info: 0000:3b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
+ ethtool -i ens1f1
driver: ice
version: 5.14.0-283.el9.x86_64
firmware-version: 4.20 0x80017785 1.3346.0
expansion-rom-version: 
bus-info: 0000:3b:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
+ uname -r
5.14.0-283.el9.x86_64

Comment 8 Jean-Tsung Hsiao 2023-03-17 16:17:38 UTC

The following dpdk test build has been tested:

[root@wsfd-advnetlab10 ddp]# rpm -q dpdk
dpdk-22.11-2.el9_2.x86_64

dpdk-testpmd -l 2,4,28,6,30 -a 0000:3b:00.0 -a 0000:3b:00.1 --socket-mem=4096,4096 -- -i --nb-cores=4 --rxq=2 --txq=2 --rxd=1024 --txd=1024

With Trex traffic the binary-search throughtput rate is 80.7 Mpps(two ways) subject to 0.002% drop.

So, there is no regression.

Comment 17 errata-xmlrpc 2023-05-09 07:53:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (dpdk bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2419