Bugzilla will be upgraded to version 5.0 on December 2, 2018. The outage period for the upgrade will start at 0:00 UTC and have a duration of 12 hours
Bug 590578 - RFE add suport for SRP storage (infiniband) to anaconda
RFE add suport for SRP storage (infiniband) to anaconda
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: anaconda (Show other bugs)
6.0
All Linux
low Severity medium
: rc
: ---
Assigned To: Ales Kozumplik
Release Test Team
: FutureFeature, Reopened
Depends On: 660686
Blocks: 647893
  Show dependency treegraph
 
Reported: 2010-05-10 04:34 EDT by Justin Clift
Modified: 2014-09-30 19:38 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 590597 (view as bug list)
Environment:
Last Closed: 2011-08-10 10:21:27 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Justin Clift 2010-05-10 04:34:39 EDT
Description of problem:

At present during installation of RHEL 6 (first public beta) there appears to be only two advanced block storage types available.

 + iSCSI
 + FCoE

As RHEL 6 itself supports additional types (SRP and GNBD come to mind), these should be added.

 + SRP would be conditional upon an Infiniband or iWARP capable 10GbE interface being detected, and would then require the loading of the ib_srp kernel module (already part of the RHEL 6 kernel package).

 + GNBD would likely be conditional upon a network interface being available, in a similar vein to iSCSI, and would require loading of the "gnbd" kernel module.
Comment 1 Hans de Goede 2010-05-10 04:44:56 EDT
Hi,

Thanks for the bug report.

(In reply to comment #0)
> Description of problem:
> 
> At present during installation of RHEL 6 (first public beta) there appears to
> be only two advanced block storage types available.
> 
>  + iSCSI
>  + FCoE
> 
> As RHEL 6 itself supports additional types (SRP and GNBD come to mind), these
> should be added.
> 
>  + SRP would be conditional upon an Infiniband or iWARP capable 10GbE interface
> being detected, and would then require the loading of the ib_srp kernel module
> (already part of the RHEL 6 kernel package).
> 

How can we see from userspace if such an interface has been detected, does it export any specific sysfs attributes for example ?

And is loading the kernel driver all that is need, will it then automatically configure itself and bring up any attached "disks" as scsi disks ?
If so how long does this configuration take ?

>  + GNBD would likely be conditional upon a network interface being available,
> in a similar vein to iSCSI, and would require loading of the "gnbd" kernel
> module.    

When you say "in a similar vein to iSCSI" does this mean that this nic needs to
be configured for IP traffic ?

And is loading the kernel driver all that is need, will it then automatically configure itself and bring up any attached "disks" as scsi disks ?
If so how long does this configuration take ?

Thanks & Regards,

Hans
Comment 2 Justin Clift 2010-05-10 05:08:08 EDT
(In reply to comment #1)
<snip>
> How can we see from userspace if such an interface has been detected, does it
> export any specific sysfs attributes for example ?

Good point.

This is from a RHEL 6 beta server (freshly installed), with an Infiniband card in it:

$ pwd
/sys/class/infiniband
$ ls -la
total 0
drwxr-xr-x.  2 root root 0 May 10 18:18 .
drwxr-xr-x. 45 root root 0 May 10 18:40 ..
lrwxrwxrwx.  1 root root 0 May 10 18:18 mthca0 -> ../../devices/pci0000:00/0000:00:03.0/0000:0a:00.0/infiniband/mthca0
$

This card uses the ib_mthca Infiniband driver, already part of the RHEL 6 kernel package.


> And is loading the kernel driver all that is need, will it then automatically
> configure itself and bring up any attached "disks" as scsi disks ?
> If so how long does this configuration take ?

Good question.  Once the ib_srp kernel module is loaded, it will then "see" any nodes presenting SRP storage to it.

However, there may need to be some work done using ibsrpdm and/or multipathing in order for this to be correctly mapped as available disk.

Probably best to ask Doug Ledford (Red Hat), the maintainer for the existing RHEL Infiniband and SRP packages (ie srptools), for his thoughts here.

Speed wise, configuration setup is very fast.  Sub-second here anyway.


> >  + GNBD would likely be conditional upon a network interface being available,
> > in a similar vein to iSCSI, and would require loading of the "gnbd" kernel
> > module.    
> 
> When you say "in a similar vein to iSCSI" does this mean that this nic needs to be configured for IP traffic ?

Unlike SRP (which doesn't need IP up and running), GNBD needs IP to be up and running in the same way that iSCSI does.  It's another IP based block storage protocol, used in clustering, expecially in regards to the old Red Hat Cluster Suite.


> And is loading the kernel driver all that is need, will it then automatically
> configure itself and bring up any attached "disks" as scsi disks ?
> If so how long does this configuration take ?

No, it needs to be configured with gnbd_import.

Actually, this is probably easier to take a look at:

  http://www.redhat.com/docs/manuals/csgfs/admin-guide/ch-gnbd.html
  11.1.2. Importing a GNBD on a Client
Comment 4 Hans de Goede 2010-05-10 05:43:15 EDT
I'm go to split this bug in 2 for tracking the 2 different requests it contains. I'm going to use this one for tracking SRP over infiniband support.
Comment 5 RHEL Product and Program Management 2010-05-10 06:48:14 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 6 RHEL Product and Program Management 2010-05-10 08:15:53 EDT
This feature request did not get resolved in time for Feature Freeze
for the current Red Hat Enterprise Linux release and has now been
denied. You may re-open your request by requesting your support
representative to propose it for the next release.
Comment 7 Justin Clift 2010-05-10 09:04:31 EDT
Ok, lets target this to 6.1 then?
Comment 8 Hans de Goede 2010-05-10 09:34:50 EDT
(In reply to comment #7)
> Ok, lets target this to 6.1 then?    

Yes.
Comment 9 Doug Ledford 2010-05-14 12:16:31 EDT
Hi Hans.  If you are going to target IB based installs, there are actually a number of things that should be considered: iSER, SRP, IPoIB, NFSoRDMA

So, going through that list:

IPoIB: This is just bringing up an IB interface with a TCP network address.  Currently we only support static IP configuration because the dhcp server we ship can't process IPoIB packets.  There is finally some work happening upstream to resolve the dhcp server issue (the problem is that our dhcp server really must be compiled to use the mmaped ring buffer method of packet gathering that the kernel supports for performance reasons, but this method uses too small of a sockaddr_ll struct and so we don't get the entire hardware address of IPoIB interfaces when doing things this way), but as it's not in place yet static configuration is the best we could do.

iSER: Same as iSCSI only it uses an IB interface instead of a TCP interface as the transport.  The fact that we support iSCSI already means this is actually pretty easy.  Once we enable the IB fabric and load the iSER module, it will appear simply as another iSCSI host adapter.

NFSoRDMA: This is fairly straight forward.  Once we support IB in general, we can do NFS mounts over the RDMA protocol.

SRP: This is actually the hardest.  The SRP protocol is fast and lean, but has its own unique discovery and attachment protocol.  In fact, the whole reason iSER exists is because the IB consortium decided that they didn't want to reinvent the wheel in terms of discovery and attachment protocols and usurped the iSCSI protocol with an RDMA transport and called it iSER.  It's the preferred way of doing things.  SRP was just the first generation remote disk protocol that they did way back in the day before they wisened up ;-)

So, that's the basic run down of the things it's possible to support.  Now, what you actually want to code up support for is another matter entirely...
Comment 10 Justin Clift 2010-05-14 12:28:32 EDT
Heh.  Interesting comment about SRP being the hardest. ;)

Thus far it's been a lot easier to manage than iSCSI.  Still need to do proper path failover testing here with it under multipath though, so you could be right.

From the point of view of adding it to anaconda though, the srptools package and friends should make it available to the system as if it's a local block storage device.

Thinking that's not (conceptually) so tricky?
Comment 11 Doug Ledford 2010-05-14 13:29:40 EDT
Hardest to add support for in Anaconda, not to use.  SRP is an island unto itself that wouldn't share code with other things.  iSER on the other hand is just another part of iSCSI and so even though iSCSI is harder, it's already done and iSER is basically free as an added transport.
Comment 12 Justin Clift 2010-05-14 13:36:54 EDT
Understood.  I see what you mean now.

iSCSI's already there, so there's a very minimal jump to have iSER work once IPoIB is in place, with similar thoughts for NFSoRDMA.

SRP on the other hand is approached differently, similar to Fibre Channel block storage.

Thanks Doug.
Comment 13 Hans de Goede 2010-06-01 08:00:06 EDT
Hi Doug,

Thanks for the long answer.

(In reply to comment #9)
> Hi Hans.  If you are going to target IB based installs, there are actually a
> number of things that should be considered: iSER, SRP, IPoIB, NFSoRDMA
> 
> So, going through that list:
> 
> IPoIB: This is just bringing up an IB interface with a TCP network address. 
> Currently we only support static IP configuration because the dhcp server we
> ship can't process IPoIB packets.  There is finally some work happening
> upstream to resolve the dhcp server issue (the problem is that our dhcp server
> really must be compiled to use the mmaped ring buffer method of packet
> gathering that the kernel supports for performance reasons, but this method
> uses too small of a sockaddr_ll struct and so we don't get the entire hardware
> address of IPoIB interfaces when doing things this way), but as it's not in
> place yet static configuration is the best we could do.
> 

Ok, as anaconda does all network configuration through the standard tools (through network manager to be precise) we should automatically support this once the tools do.

> iSER: Same as iSCSI only it uses an IB interface instead of a TCP interface as
> the transport.  The fact that we support iSCSI already means this is actually
> pretty easy.  Once we enable the IB fabric and load the iSER module, it will
> appear simply as another iSCSI host adapter.
> 

Ok, so once IP is configured on the IB interface the existing iscsi code should work.

> NFSoRDMA: This is fairly straight forward.  Once we support IB in general, we
> can do NFS mounts over the RDMA protocol.
> 

Will "mount" handle this transparently, or ... ?

> SRP: This is actually the hardest.  The SRP protocol is fast and lean, but has
> its own unique discovery and attachment protocol.  In fact, the whole reason
> iSER exists is because the IB consortium decided that they didn't want to
> reinvent the wheel in terms of discovery and attachment protocols and usurped
> the iSCSI protocol with an RDMA transport and called it iSER.  It's the
> preferred way of doing things.  SRP was just the first generation remote disk
> protocol that they did way back in the day before they wisened up ;-)
> 

Ok so lets say we would want to support SRP (I'm not saying we do but lets say we do). What would need to be done (ie cmdline commands) to bring up an SRP attached disk?

> So, that's the basic run down of the things it's possible to support.  Now,
> what you actually want to code up support for is another matter entirely...    

Well as described above I don't expect there to be much if any anaconda work
for IB as normal network adapter, not for iscsi/iser As for SRP, I would first like to learn some more and then we can consider adding SRP support to Fedora-14, from where it may or may not finds its way into a RHEL6-.x release.

Regards,

Hans
Comment 18 Ales Kozumplik 2011-01-12 04:45:41 EST
Doug,

I am assigned to this bug after Hans left. Is there some InfiniBand hardware available in Red Hat, or how do you test your changes?

Thanks.
Ales
Comment 20 Ales Kozumplik 2011-01-13 02:19:26 EST
Hi Justin,

That could in fact be everything I need! Who can I talk to to learn more?

Ales
Comment 24 Mike Khusid 2011-05-26 16:20:06 EDT
NOTE: Bugzilla is not a support tool

The Bugzilla interface at bugzilla.redhat.com is used internally by Red
Hat to process changes e.g. to Red Hat Enterprise Linux and related
products, as well as by the Fedora Community to develop the Fedora
Project.

It is publicly available and everyone with an email address is able to
create an account, file bugs, comment on bugs she or he has access to.
Not all bugs are public though and not all issues filed are handled in
the same way: it makes a huge difference who is behind a bug.

Red Hat does monitor Bugzilla entries, and it does review them for
inclusion in errata, etc.

Nevertheless, as noted on the login page, Bugzilla is not a Support
tool. It is an Engineering tool. It is used by Red Hat Engineering to
track issues and product changes, as well as to interact with Egineering
partners and other parties external to Red Hat on a technical level.

So while all changes to Red Hat Enterprise Linux will at a point go
through Bugzilla, this difference has a number of important consequences
for general product issues filed directly through Bugzilla by external
users without any privileged Engineering relationship:

    * Red Hat does NOT guarantee any response time or Service Level
Agreement (SLA) for Bugzilla entries. - A review might happen
immediately or after a time span of any length. The SLAs for Red Hat
Enterprise Linux provided by Red Hat Support can be found at:
https://www.redhat.com/support/policy/sla/production/

    * Not all comments are publicly visible. - Red Hat Support provides
customers with appropriate information excerpts and status updates from
that. Red Hat does not commit to provide detailed explanations, or
guidance in the context of Bugzilla. Therefore for example, Bugzilla
entries might be closed as it seems without any further explanation,
while Red Hat Support actually provides such explanation to customers
filing through the regular support channels.

    * Issues coming through the regular support process, will always be
prioritized higher than issues of similar impact and severity that are
being filed directly through Bugzilla (unless the later get linked to
customer support tickets). This means that they are more likely to be 
addressed and they are more likely to meet inclusion criteria 
consistent with the Red Hat Enterprise Linux life cycle policy:
http://www.redhat.com/security/updates/errata/

    * Work-arounds and Hotfixes if possible and appropriate are provided
by Red Hat Support and through the regular support process. - This means
that even before a permanent fix is made available through RHN, customers
who raised a high severity issue through Red Hat Support, are likely to
receive an interim solution.

Red Hat provides common Bugzilla access in order provide efficient
development community interaction and as much transparency as possible
to our customers. Our Engineers are encouraged to provide non-customer
specific and non-confidential information publicly as often as possible.

So while Red Hat considers issues directly entered into Bugzilla
valuable feedback - may it be as comments to existing Bugzilla entries
or by opening a new one; for customers encountering production issues,
Bugzilla is not the right channel.

Therefore we ask our customers to file requests important for their
production systems via our Support service. Only for those issues, we
can ensure a consistent communication. Information about our production
support process can be found at: http://www.redhat.com/support/process/

Bugzilla can always be used as a supportive channel to that
communication.

Note: If your customer is participating in the Academic program and has
chosen to run a Subscription without support, they consequently have no
access to Red Hat Support and thus no SLA. If you feel that this is
insufficient for your use case, you should consider contacting the Red
Hat Education specialist as described at:
http://www.redhat.com/solutions/education/academic/individual/
Comment 25 Mike Khusid 2011-05-26 16:20:59 EDT
There are no plans to add SRP storage to Red Hat Enterprise Linux installation.
Comment 26 RHEL Product and Program Management 2011-05-26 16:24:48 EDT
Product Management has reviewed and declined this request.  You may appeal this
decision by reopening this request.
Comment 27 Justin Clift 2011-05-26 17:03:30 EDT
Mike, we *already* support SRP storage in RHEL.

The modules are already there, as is the SRP daemon.

This request was for adding SRP support to Anaconda as well, not for adding SRP support to RHEL, which as mentioned, is already in place.
Comment 28 Mike Khusid 2011-05-26 17:20:26 EDT
(In reply to comment #27)
> Mike, we *already* support SRP storage in RHEL.
> 
> The modules are already there, as is the SRP daemon.
> 
> This request was for adding SRP support to Anaconda as well, not for adding SRP
> support to RHEL, which as mentioned, is already in place.

Clarification: word "installation" in comment 25 was referring to Anaconda.

Note You need to log in before you can comment on or make changes to this bug.