This bug is opened to track the inclusion of the iWARP driver for Chelsio's next generation HW. The NIC driver is submitted upstream already, iw_cxgb4 will follow soon.
The iw_cxgb4 driver has been submitted upstream for review. The patch series can be found here: http://www.mail-archive.com/linux-rdma@vger.kernel.org/msg03209.html cover letter: [PATCH 00/10] iw_cxgb4 driver submission Steve Wise Fri, 02 Apr 2010 12:24:07 -0700 The following series adds the iw_cxgb4 rdma driver for Chelsio's new 1G and 10G T4 adapters. Please review and provide comments, and I'll repost an updated patch series as I incorporate your feedback. This series depends on the cxgb4 driver recently merged into: git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git You can find the final cxgb4 patch submission at: http://www.spinics.net/lists/netdev/msg125898.html The iw_cxgb4 patches are against the above net-2.6 tree as of cf16d7d. You can also pull these from: git://www.openfabrics.org/~swise/cxgb4.git iw_cxgb4 drivers/infiniband/Kconfig | 1 drivers/infiniband/Makefile | 1 drivers/infiniband/hw/cxgb4/Kconfig | 19 drivers/infiniband/hw/cxgb4/Makefile | 5 drivers/infiniband/hw/cxgb4/cm.c | 2360 +++++++++++++++++++++++++++++ drivers/infiniband/hw/cxgb4/cq.c | 885 +++++++++++ drivers/infiniband/hw/cxgb4/device.c | 520 ++++++ drivers/infiniband/hw/cxgb4/ev.c | 223 +++ drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 739 +++++++++ drivers/infiniband/hw/cxgb4/mem.c | 811 ++++++++++ drivers/infiniband/hw/cxgb4/provider.c | 521 ++++++ drivers/infiniband/hw/cxgb4/qp.c | 1491 ++++++++++++++++++ drivers/infiniband/hw/cxgb4/resource.c | 417 +++++ drivers/infiniband/hw/cxgb4/t4.h | 527 ++++++ drivers/infiniband/hw/cxgb4/t4fw_ri_api.h | 776 ++++++++++ drivers/infiniband/hw/cxgb4/user.h | 66 + 16 files changed, 9362 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/hw/cxgb4/Kconfig create mode 100644 drivers/infiniband/hw/cxgb4/Makefile create mode 100644 drivers/infiniband/hw/cxgb4/cm.c create mode 100644 drivers/infiniband/hw/cxgb4/cq.c create mode 100644 drivers/infiniband/hw/cxgb4/device.c create mode 100644 drivers/infiniband/hw/cxgb4/ev.c create mode 100644 drivers/infiniband/hw/cxgb4/iw_cxgb4.h create mode 100644 drivers/infiniband/hw/cxgb4/mem.c create mode 100644 drivers/infiniband/hw/cxgb4/provider.c create mode 100644 drivers/infiniband/hw/cxgb4/qp.c create mode 100644 drivers/infiniband/hw/cxgb4/resource.c create mode 100644 drivers/infiniband/hw/cxgb4/t4.h create mode 100644 drivers/infiniband/hw/cxgb4/t4fw_ri_api.h create mode 100644 drivers/infiniband/hw/cxgb4/user.h
In addition, libcxgb4 will need to be included to enable user mode RDMA. The libcxgb4 development git tree can be found at: git://www.openfabrics.org/~swise/libcxgb4.git I will be releasing 1.0.0 of this library soon. I'll update this feature when I release. Steve.
Created attachment 405071 [details] Upstream-under-review iw_cxgb4 driver backported to rhel5.5 This patch backports the current iw_cxgb4 patch series currently under review upstream to rhel5.5. It depends on the cxgb4 patch in feature 567446. Note: This patch adds idr_for_each() to the rhel5.5 kernel proper. It is needed by iw_cxgb4 and I just pulled in the upstream version. This is intended as an initial submission for inclusion of iw_cxgb4 into rhel-5.6. Once the upstream version is merged by Roland Dreier, I'll re-post a full patch of the final driver.
Created attachment 438993 [details] Latest iw_cxgb4 driver backported to 2.6.18-210 This patch backports the latest upstream iw_cxgb4 driver to linux-2.6.18-210. It depends on the cxgb4 driver/patch in feature 567446. Note: This patch adds idr_for_each() and debugfs_remove_recursive() to the kernel proper. It is needed by iw_cxgb4. This is intended as an latest up-to-date submission for inclusion of iw_cxgb4 into rhel-5.6.
You also want to pull the latest libcxgb4. Version 1.0.4. http://www.openfabrics.org/downloads/cxgb4/libcxgb4-1.0.4-md5sum.txt http://www.openfabrics.org/downloads/cxgb4/libcxgb4-1.0.4.tar.gz
Created attachment 451187 [details] Patch to add iw_cxgb4 to linux-2.6.18-225.el5 The attached patch is build against 2.6.18-225 found at: http://people.redhat.com/jwilson/el5/ It adds support for iw_cxgb4. The iw_cxgb4 code is a backport the upstream iw_cxgb4 code. If the patch is acceptable, can you please let us know the time line for its inclusion in the RHEL5.6 builds? Thanks, Steve
You also need the latest libcxgb4 from here: http://www.openfabrics.org/downloads/cxgb4/libcxgb4-1.1.0.tar.gz with md5sum: 8ab91446a9e3a991c03d3c4489a1311b libcxgb4-1.1.0.tar.gz Thanks, Steve.
Created attachment 479003 [details] refresh previous patch to TOT linux as of 2/15/2011 This patch plus the previous one posted brings iw_cxgb4 up to TOT linux. Please consider this for RHEL5.7. You also need the latest libcxgb4 package. Thanks Steve
I've just released version 1.1.1 of libcxgb4. It fixes 2 bugs from 1.1.0. Please include it when you include iw_cxgb4. Release 1.1.1 is available at: http://www.openfabrics.org/downloads/cxgb4/libcxgb4-1.1.1.tar.gz with md5sum: a7d7a432b26794cad4e997ebf63ae554 libcxgb4-1.1.1.tar.gz The development git tree is at: git://www.openfabrics.org/~swise/libcxgb4 Thanks, Steve.
Here are commit IDs for some upstream bug fixes that should be included. They are critical IMO: commit 767fbe8151d1a7cc8a69e52e354e4220a5e804fb Author: Steve Wise <swise> Date: Fri Mar 11 22:30:53 2011 +0000 RDMA/cxgb4: Dispatch FATAL event on EEH errors This at least kicks the user mode applications that are watching for device events. Signed-off-by: Steve Wise <swise> Signed-off-by: Roland Dreier <roland> commit ffc3f7487ff0b32500b319c770475383f6f6efab Author: Steve Wise <swise> Date: Fri Mar 11 22:30:42 2011 +0000 RDMA/cxgb4: Do CIDX_INC updates every 1/16 CQ depth CQE reaps Signed-off-by: Steve Wise <swise> Signed-off-by: Roland Dreier <roland> commit 294281373999e7fff393c04eb16092a8f00ad5aa Author: Steve Wise <swise> Date: Fri Mar 11 22:30:32 2011 +0000 RDMA/cxgb4: Remove db_drop_task Unloading iw_cxgb4 can crash due to the unload code trying to use db_drop_task, which is uninitialized. So remove this dead code. Signed-off-by: Steve Wise <swise> Signed-off-by: Roland Dreier <roland>
(In reply to comment #7) > You also need the latest libcxgb4 from here: > > http://www.openfabrics.org/downloads/cxgb4/libcxgb4-1.1.0.tar.gz > > with md5sum: > > 8ab91446a9e3a991c03d3c4489a1311b libcxgb4-1.1.0.tar.gz > > > Thanks, > > Steve. Steve or Divy, could you open a new bz for the libcxgb4 addition. it can't be bought in with this bz. please add that new bz # to this one after you have it open... thanks -Steve
BTW Steve, the same fixes that got pulled into RHEL6.1 for iw_cxgb4 should also be pulled into RHEL5.7. (In reply to comment #11) > (In reply to comment #7) > > You also need the latest libcxgb4 from here: > > > > http://www.openfabrics.org/downloads/cxgb4/libcxgb4-1.1.0.tar.gz > > > > with md5sum: > > > > 8ab91446a9e3a991c03d3c4489a1311b libcxgb4-1.1.0.tar.gz > > > > > > Thanks, > > > > Steve. > > Steve or Divy, > > could you open a new bz for the libcxgb4 addition. it can't be bought in with > this bz. please add that new bz # to this one after you have it open... thanks > > -Steve Hey Steve, I opened 693511 for including libcxgb4-1.1.1. Thanks, Steve Wise.
Steve, The commits added to the rhel6 stream should also be added to this RHEL5 stream. The following upstream bug fixes should be included: These three are in the refresh patch I posted in comment 8: db8b10167126d72829653690f57b9c7ca53c4d54 ac04c8c9a56bd326429b07ee5c0dcffd690fe57b 2a851f37a8ae27a0df4f77c8bc5b4420b020bf3b These three were posted in comment 10 and should be applied: 767fbe8151d1a7cc8a69e52e354e4220a5e804fb ffc3f7487ff0b32500b319c770475383f6f6efab 294281373999e7fff393c04eb16092a8f00ad5aa These 2 fix potential crashes, and are committed upstream in Roland Dreier's infiniband git tree: a2879f15a71d5d1797b006badd31648f5d9e4da7 92d7d8fd21a4649018eee869f5f7f8d3ecab5735 Hollar if this doesn't make sense. Steve.
(In reply to comment #6) > Created attachment 451187 [details] > Patch to add iw_cxgb4 to linux-2.6.18-225.el5 > > > The attached patch is build against 2.6.18-225 found at: > http://people.redhat.com/jwilson/el5/ > > It adds support for iw_cxgb4. The iw_cxgb4 code is a backport the upstream > iw_cxgb4 code. > > If the patch is acceptable, can you please let us know the time line for its > inclusion in the RHEL5.6 builds? > > Thanks, > > Steve Steve, could you provide commit ids for this patch. thanks -Steve
(In reply to comment #14) > (In reply to comment #6) > > Created attachment 451187 [details] > > Patch to add iw_cxgb4 to linux-2.6.18-225.el5 > > > > > > The attached patch is build against 2.6.18-225 found at: > > http://people.redhat.com/jwilson/el5/ > > > > It adds support for iw_cxgb4. The iw_cxgb4 code is a backport the upstream > > iw_cxgb4 code. > > > > If the patch is acceptable, can you please let us know the time line for its > > inclusion in the RHEL5.6 builds? > > > > Thanks, > > > > Steve > > Steve, > > could you provide commit ids for this patch. thanks > > -Steve Hey Steve, I basically took the upstream code as of ~9/2010 and ported it. Here is what Git tells me are the commits for drivers/infiniband/hw/cxgb4 from its birth until 10/2010: commit 3160977a6e66ea4c4b4f33010f5d04f0004b938c commit 8bbac892fb75d20fa274ca026e24faf00afbf9dd commit 40dbf6ee381008e471d3c4a332971247b7799744 commit 410ade4c26bdf256fea3246e968a12409eb08763 commit 98ae68b7ee6adb75ede42d84eae4032dbb122b81 commit 2f5b48c3ad84fda9efe35122b058ccffc0c2c7cf commit c6d7b26791a2aefdf97f2af1e93161ed05acd631 commit aadc4df3087a33ca7fc37f91a024e7b7efdafa75 commit 9e8d1fa3420f489da8a5da47c026511aa71fa50b commit 05fb9629473690e4be4112f22e1adeb1fe4ad733 commit 0e42c1f4303f3f8d5b2c257dc5488b0ad465097d commit 6ff0e343b3356897cef1f09452f93acb13703911 commit 7459486133dc726ff2edf0957d9cb5c954aedbc7 commit 13fecb83b410b147343e6c7b0427d244ef77b526 commit af93fb5dcc6b1cba5fd0861d349b3f9c93144bc0 commit c8e081a1bf11c5cbac5f2f9f53c040be61d7b29e commit 93fb72e443b9fcbef598faa05478883952edca77 commit 3cc08fc35db75b059118626c30b60b0f56583802 commit 3cfc2c42c1cbc8e238bb9c0612c0df4565e3a8b4 commit 6ba74014c1ab0e37af7de6f64b4eccbbae3cb9e7 commit a5f4a07820ebc60b21c984d893f48402c4b4a4a2 commit d790d4d583aeaed9fc6f8a9f4d9f8ce6b1c15c7f commit ca5a22028d0845dd6bcce0dce12a7beda315baf0 commit d4f1a5c6efabccd4b787a8b5907a5df9204ad2f6 commit 73d6fcad2ac84b6fad326d87dc1dd0b29aabbd34 commit d37ac31ddc24c1a0beed134278bc074c98812210 commit d3c814e8b2a094dc3bcbe6a0d93ec4824b26e86a commit ba6d39256bed87a0e8ee1770b5f7638bb3e0cfe4 commit 85963e4cbcf11c00b1d27ea0e0fcab8cb3d7a69b commit c1d7356c8572f3fe0445336d8e75914bdcadad59 commit 2c5934bfc5ffcbef3622d0bdbad93628d210012a commit 1973e8b8edea68d2408328d25b318ee7401293be commit b21ef16a8b956aee2fb3d7fc9d24a0b4dae2ae72 commit f38926aa1dc5fbf7dfc5f97a53377b2e796dedc3 commit f1bbbb6912662b9f6070c5bfc4ca9eb1f06a9d5b commit 732bee7af3102cad811fb047dee8d15966efe569 commit d8d1f30b95a635dbd610dcc5eb641aca8f4768cf commit acdc30b56abc0db7d409a13e9b6c72ea23b6f90d commit 30a6a62fc35208924ffa8a472b0af4552441cbff commit 2f1fb507eec22ee64b541ea8586a7365ede8be2e commit 4ab1eb9c8df620d5ed64fa1d1be683862b311685 commit f64b88433c27815f931d4d5ff7db7ac701fdc8c9 commit 25737bd4ca1b58e86efa9211c1717140e0d4910e commit 841dba9a5a0dbced4b4bc240f444d47f30ba6724 commit 7ec45b923446d484eb39434e18d354666426e606 commit 84172dee05cbce6ae791eac481ef4d8590cda791 commit 895cf5f3d6cb72825919a39ae48a41476a821e65 commit 1c01c5388306a4c2245b71da0cba22d521e897ae commit fd388ce677e7de9180a7d46d12c5162f76af64ac commit 9a6edb60ec10d86b1025a0cdad68fd89f1ddaf02 commit be4c9bad9d0edb6bc3bd8fffc2f98e0e2112da39 commit cfdda9d764362ab77b11a410bb928400e6520d57
The command I used to generate that list was: git log --summary --since=1/1/2000 --until 10/15/2010 drivers/infiniband/hw/cxgb4 In addition to those commits, there were backport changes.
(In reply to comment #16) > The command I used to generate that list was: > > git log --summary --since=1/1/2000 --until 10/15/2010 > drivers/infiniband/hw/cxgb4 > > In addition to those commits, there were backport changes. By the way, IMO you should take the uber-patch in comment 6 as the starting point. I don't really see a point in trying to use all these initial patches to bring the driver up to date, when this is the initial addition of this driver into the RHEL5 stream. My 2 centimes. :) Steve.
(In reply to comment #17) > (In reply to comment #16) > > The command I used to generate that list was: > > > > git log --summary --since=1/1/2000 --until 10/15/2010 > > drivers/infiniband/hw/cxgb4 > > > > In addition to those commits, there were backport changes. > > > By the way, IMO you should take the uber-patch in comment 6 as the starting > point. I don't really see a point in trying to use all these initial patches > to bring the driver up to date, when this is the initial addition of this > driver into the RHEL5 stream. > > My 2 centimes. :) > > Steve. Steve, I'm starting with the patch in comment 6. I should have test build with Neil's addition of cxgb4 and the addition iw_cxgb4 later today or tomorrow. -Steve
IBM and Chelsio are signed up to test and provide feedback, setting OtherQA.
------- Comment From sbest.com 2011-04-08 06:45 EDT------- Michael, could you test iw_cxgb4. This test kernel also has the changes to include cxgb4 that Neil did for https://bugzilla.redhat.com/show_bug.cgi?id=567446. could you also push this test kernel to Chelsio, so they can test also. test kernel is here http://pokgsa.ibm.com/~sbest/public/567449/ would like test results by Monday if possible.. thanks
posted to rh-kernel mailing list http://post-office.corp.redhat.com/archives/rhkernel-list/2011-April/msg00652.html
Patch(es) available in kernel-2.6.18-257.el5 You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed.
------- Comment From sbest.com 2011-06-04 05:31 EDT------- verified the driver is there and we will open new bzs if we find issues
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-1065.html