Escalated to Bugzilla from IssueTracker
08:22 <grondo> What version of myri10ge driver is in RHEL5.2? 09:19 <neb> let me check 09:31 <neb> #define MYRI10GE_VERSION_STR "1.0.0" 09:32 <neb> which version do you need? what features do you need? RH is kind of hard to deal with re: version numbers 09:32 <neb> as you well know they often times backport fixes or features and don't update the version number 09:34 <neb> There appear to be at least 2 patches to that driver: 09:34 <neb> linux-2.6-net-netxen-initial-support-for-netxen-10gbe-nic.patch 09:34 <neb> linux-2.6-pci-pci-x-pci-express-read-control-interface.patch 09:54 <grondo> we'd like 1.4.2 09:54 <neb> OK I'll get that in the pipeline 09:54 <neb> can you give me some features in 1.4.2 that you need. 09:54 <neb> makes it easier to get it through the PM folk 09:56 <grondo> more efficient, faster 09:56 <grondo> loads FW via hotplug Can we get patches against 5.2's kernel ASAP so that we can start getting it working and flush out any bugs that might crop up during the 5.3 development cycle. Evidently, we already have hardware. This event sent from IssueTracker by jwest [SEG - Feature Request] issue 178049
Updating PM score.
To add support for Intel DCA (Direct Cache Access) (last change before 1.4.2 driver release) we will need to backport DCA driver too, which can be problematic. Other big problem is TX multiqueue, which is not supported with core network in RHEL5 kernel. Other things could be quite easy to backport, I mostly already done it.
(In reply to comment #1) > cycle. Evidently, we already have hardware. Where the hardware is? I can't find any system with myri10ge device on RHTS.
Created attachment 340305 [details] myri10ge update to 1.3.99-1.347 for kernel-2.6.18-138.el5 _NOT_TESTED_
(In reply to comment #5) > To add support for Intel DCA (Direct Cache Access) (last change before 1.4.2 > driver release) we will need to backport DCA driver too, which can be > problematic. We have already DCA backports posted to rhkernel-list: http://post-office.corp.redhat.com/archives/rhkernel-list/2009-April/msg00970.html If this changes will be added to rhel-5 kernel, we could add DCA support to the driver.
*** Bug 481630 has been marked as a duplicate of this bug. ***
As we still have no Miricom hardware in inventory, conditionally NACK the request.
Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.
Created attachment 486222 [details] 0001-PCI-make-pcie_get_readrq-visible-in-pci.h.patch
Created attachment 486223 [details] 0002-myri10ge-update-to-1.5.2.patch I'm going to post this patch to RH internal mailing list for review. Additional review and testing from Myricom are welcome.
BTW: I have only about 6Gbit/s throughput on my setup. Results was almost the same on backported patch, driver from myri.com site and upstream kernel. I tried to tune according to http://www.myri.com/serve/cache/511.html#linux to get some better speed, but without luck. I wonder what could bottleneck here, PCI-express bus?
I'm not sure exactly which NIC you have, but 6Gb/s sounds very much like you've got a Gen1 NIC in an x4 slot. What does lspci -vvv say? Alternately, the driver/firwmare does a DMA benchmark at driver load time. The results are displayed in ethtool -S in megabytes per second. Eg: %sudo ethtool -S eth2 | grep dma read_dma_bw_MBs: 1395 write_dma_bw_MBs: 1613 read_write_dma_bw_MBs: 2710 So for 10Gb/s operation, you need b/w above 1250MB/s
Yes, read only and write only dma bandwidth is limited on my system to about 6Gbits/s . Does putting cards on 8x or 16x PCI-e slot give me full speed? Here is full info: > [root@dhcp-31-100 ~]# ethtool -i eth3 > driver: myri10ge > version: 1.5.2-1.459 > firmware-version: 1.4.46 -- 2009/10/21 21:27:53 m > bus-info: 0000:20:00.0 > [root@dhcp-31-100 ~]# ethtool -S eth3 | grep dma > read_dma_bw_MBs: 832 > write_dma_bw_MBs: 777 > read_write_dma_bw_MBs: 1491 > [root@dhcp-31-100 ~]# lspci -vvv -d 14c1:0008 > 20:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (rev 01) > Subsystem: MYRICOM Inc. 10G-PCIE-8B > Physical Slot: 6 > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin A routed to IRQ 94 > Region 0: Memory at f3000000 (64-bit, prefetchable) [size=16M] > Region 2: Memory at f4100000 (64-bit, non-prefetchable) [size=1M] > Capabilities: [44] MSI: Enable+ Count=1/1 Maskable- 64bit+ > Address: 00000000fee0f00c Data: 41c9 > Capabilities: [54] Power Management version 3 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- > Capabilities: [5c] Express (v2) Endpoint, MSI 00 > DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ > DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- > MaxPayload 128 bytes, MaxReadReq 4096 bytes > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- > LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited > ClockPM- Surprise- LLActRep+ BwNot- > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt- > DevCap2: Completion Timeout: Range BCD, TimeoutDis+ > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- > LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -6dB > Capabilities: [a0] Vendor Specific Information <?> > Capabilities: [d0] MSI-X: Enable- Count=128 Masked- > Vector table: BAR=2 offset=000f0000 > PBA: BAR=2 offset=000f9000 > Capabilities: [e0] Vital Product Data > Product Name: 10G-PCIE-8B-S > Read-only fields: > [PN] Part number: 09-04228 > [SN] Serial number: 392282 > [RV] Reserved: checksum good, 0 byte(s) reserved > End > Capabilities: [100] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ > AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- > Capabilities: [1a8] Device Serial Number 00-60-dd-ff-ff-45-f7-41 > Kernel driver in use: myri10ge > Kernel modules: myri10ge
Yes, using a x8 (or x16) slot should give you full b/w
Looks very good now, 9.9 Gb/s in both directions.
Patches look good to me. Our test lab is in the middle of a re-org / spring cleaning, so I probably won't be able to test until at least tomorrow. If you can't wait, please make sure to test the driver with multiple rx queues (load driver with myri10ge_max_slices=8, this will require installing f/w RPM from http://www.myri.com/scs/download-Myri10GE-firmware-only.html)
We are not in hurry :-) Myri10ge_max_slices=8 works here (at least driver does not crash) with 1.4.52 firmware.
Our tests pass with this version of the driver. Thank you for doing the backport.
Patch(es) available in kernel-2.6.18-254.el5 You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-1065.html