Bug 590578

Hi,

Thanks for the bug report.

(In reply to comment #0)
> Description of problem:
> 
> At present during installation of RHEL 6 (first public beta) there appears to
> be only two advanced block storage types available.
> 
>  + iSCSI
>  + FCoE
> 
> As RHEL 6 itself supports additional types (SRP and GNBD come to mind), these
> should be added.
> 
>  + SRP would be conditional upon an Infiniband or iWARP capable 10GbE interface
> being detected, and would then require the loading of the ib_srp kernel module
> (already part of the RHEL 6 kernel package).
> 

How can we see from userspace if such an interface has been detected, does it export any specific sysfs attributes for example ?

And is loading the kernel driver all that is need, will it then automatically configure itself and bring up any attached "disks" as scsi disks ?
If so how long does this configuration take ?

>  + GNBD would likely be conditional upon a network interface being available,
> in a similar vein to iSCSI, and would require loading of the "gnbd" kernel
> module.    

When you say "in a similar vein to iSCSI" does this mean that this nic needs to
be configured for IP traffic ?

And is loading the kernel driver all that is need, will it then automatically configure itself and bring up any attached "disks" as scsi disks ?
If so how long does this configuration take ?

Thanks & Regards,

Hans

(In reply to comment #1)
<snip>
> How can we see from userspace if such an interface has been detected, does it
> export any specific sysfs attributes for example ?

Good point.

This is from a RHEL 6 beta server (freshly installed), with an Infiniband card in it:

$ pwd
/sys/class/infiniband
$ ls -la
total 0
drwxr-xr-x.  2 root root 0 May 10 18:18 .
drwxr-xr-x. 45 root root 0 May 10 18:40 ..
lrwxrwxrwx.  1 root root 0 May 10 18:18 mthca0 -> ../../devices/pci0000:00/0000:00:03.0/0000:0a:00.0/infiniband/mthca0
$

This card uses the ib_mthca Infiniband driver, already part of the RHEL 6 kernel package.


> And is loading the kernel driver all that is need, will it then automatically
> configure itself and bring up any attached "disks" as scsi disks ?
> If so how long does this configuration take ?

Good question.  Once the ib_srp kernel module is loaded, it will then "see" any nodes presenting SRP storage to it.

However, there may need to be some work done using ibsrpdm and/or multipathing in order for this to be correctly mapped as available disk.

Probably best to ask Doug Ledford (Red Hat), the maintainer for the existing RHEL Infiniband and SRP packages (ie srptools), for his thoughts here.

Speed wise, configuration setup is very fast.  Sub-second here anyway.


> >  + GNBD would likely be conditional upon a network interface being available,
> > in a similar vein to iSCSI, and would require loading of the "gnbd" kernel
> > module.    
> 
> When you say "in a similar vein to iSCSI" does this mean that this nic needs to be configured for IP traffic ?

Unlike SRP (which doesn't need IP up and running), GNBD needs IP to be up and running in the same way that iSCSI does.  It's another IP based block storage protocol, used in clustering, expecially in regards to the old Red Hat Cluster Suite.


> And is loading the kernel driver all that is need, will it then automatically
> configure itself and bring up any attached "disks" as scsi disks ?
> If so how long does this configuration take ?

No, it needs to be configured with gnbd_import.

Actually, this is probably easier to take a look at:

  http://www.redhat.com/docs/manuals/csgfs/admin-guide/ch-gnbd.html
  11.1.2. Importing a GNBD on a Client

I'm go to split this bug in 2 for tracking the 2 different requests it contains. I'm going to use this one for tracking SRP over infiniband support.

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

This feature request did not get resolved in time for Feature Freeze
for the current Red Hat Enterprise Linux release and has now been
denied. You may re-open your request by requesting your support
representative to propose it for the next release.

Ok, lets target this to 6.1 then?

(In reply to comment #7)
> Ok, lets target this to 6.1 then?    

Yes.

Hi Hans.  If you are going to target IB based installs, there are actually a number of things that should be considered: iSER, SRP, IPoIB, NFSoRDMA

So, going through that list:

IPoIB: This is just bringing up an IB interface with a TCP network address.  Currently we only support static IP configuration because the dhcp server we ship can't process IPoIB packets.  There is finally some work happening upstream to resolve the dhcp server issue (the problem is that our dhcp server really must be compiled to use the mmaped ring buffer method of packet gathering that the kernel supports for performance reasons, but this method uses too small of a sockaddr_ll struct and so we don't get the entire hardware address of IPoIB interfaces when doing things this way), but as it's not in place yet static configuration is the best we could do.

iSER: Same as iSCSI only it uses an IB interface instead of a TCP interface as the transport.  The fact that we support iSCSI already means this is actually pretty easy.  Once we enable the IB fabric and load the iSER module, it will appear simply as another iSCSI host adapter.

NFSoRDMA: This is fairly straight forward.  Once we support IB in general, we can do NFS mounts over the RDMA protocol.

SRP: This is actually the hardest.  The SRP protocol is fast and lean, but has its own unique discovery and attachment protocol.  In fact, the whole reason iSER exists is because the IB consortium decided that they didn't want to reinvent the wheel in terms of discovery and attachment protocols and usurped the iSCSI protocol with an RDMA transport and called it iSER.  It's the preferred way of doing things.  SRP was just the first generation remote disk protocol that they did way back in the day before they wisened up ;-)

So, that's the basic run down of the things it's possible to support.  Now, what you actually want to code up support for is another matter entirely...

Heh.  Interesting comment about SRP being the hardest. ;)

Thus far it's been a lot easier to manage than iSCSI.  Still need to do proper path failover testing here with it under multipath though, so you could be right.

From the point of view of adding it to anaconda though, the srptools package and friends should make it available to the system as if it's a local block storage device.

Thinking that's not (conceptually) so tricky?

Hardest to add support for in Anaconda, not to use.  SRP is an island unto itself that wouldn't share code with other things.  iSER on the other hand is just another part of iSCSI and so even though iSCSI is harder, it's already done and iSER is basically free as an added transport.

Understood.  I see what you mean now.

iSCSI's already there, so there's a very minimal jump to have iSER work once IPoIB is in place, with similar thoughts for NFSoRDMA.

SRP on the other hand is approached differently, similar to Fibre Channel block storage.

Thanks Doug.

Hi Doug,

Thanks for the long answer.

(In reply to comment #9)
> Hi Hans.  If you are going to target IB based installs, there are actually a
> number of things that should be considered: iSER, SRP, IPoIB, NFSoRDMA
> 
> So, going through that list:
> 
> IPoIB: This is just bringing up an IB interface with a TCP network address. 
> Currently we only support static IP configuration because the dhcp server we
> ship can't process IPoIB packets.  There is finally some work happening
> upstream to resolve the dhcp server issue (the problem is that our dhcp server
> really must be compiled to use the mmaped ring buffer method of packet
> gathering that the kernel supports for performance reasons, but this method
> uses too small of a sockaddr_ll struct and so we don't get the entire hardware
> address of IPoIB interfaces when doing things this way), but as it's not in
> place yet static configuration is the best we could do.
> 

Ok, as anaconda does all network configuration through the standard tools (through network manager to be precise) we should automatically support this once the tools do.

> iSER: Same as iSCSI only it uses an IB interface instead of a TCP interface as
> the transport.  The fact that we support iSCSI already means this is actually
> pretty easy.  Once we enable the IB fabric and load the iSER module, it will
> appear simply as another iSCSI host adapter.
> 

Ok, so once IP is configured on the IB interface the existing iscsi code should work.

> NFSoRDMA: This is fairly straight forward.  Once we support IB in general, we
> can do NFS mounts over the RDMA protocol.
> 

Will "mount" handle this transparently, or ... ?

> SRP: This is actually the hardest.  The SRP protocol is fast and lean, but has
> its own unique discovery and attachment protocol.  In fact, the whole reason
> iSER exists is because the IB consortium decided that they didn't want to
> reinvent the wheel in terms of discovery and attachment protocols and usurped
> the iSCSI protocol with an RDMA transport and called it iSER.  It's the
> preferred way of doing things.  SRP was just the first generation remote disk
> protocol that they did way back in the day before they wisened up ;-)
> 

Ok so lets say we would want to support SRP (I'm not saying we do but lets say we do). What would need to be done (ie cmdline commands) to bring up an SRP attached disk?

> So, that's the basic run down of the things it's possible to support.  Now,
> what you actually want to code up support for is another matter entirely...    

Well as described above I don't expect there to be much if any anaconda work
for IB as normal network adapter, not for iscsi/iser As for SRP, I would first like to learn some more and then we can consider adding SRP support to Fedora-14, from where it may or may not finds its way into a RHEL6-.x release.

Regards,

Hans

Doug,

I am assigned to this bug after Hans left. Is there some InfiniBand hardware available in Red Hat, or how do you test your changes?

Thanks.
Ales

Hi Justin,

That could in fact be everything I need! Who can I talk to to learn more?

Ales

NOTE: Bugzilla is not a support tool

The Bugzilla interface at bugzilla.redhat.com is used internally by Red
Hat to process changes e.g. to Red Hat Enterprise Linux and related
products, as well as by the Fedora Community to develop the Fedora
Project.

It is publicly available and everyone with an email address is able to
create an account, file bugs, comment on bugs she or he has access to.
Not all bugs are public though and not all issues filed are handled in
the same way: it makes a huge difference who is behind a bug.

Red Hat does monitor Bugzilla entries, and it does review them for
inclusion in errata, etc.

Nevertheless, as noted on the login page, Bugzilla is not a Support
tool. It is an Engineering tool. It is used by Red Hat Engineering to
track issues and product changes, as well as to interact with Egineering
partners and other parties external to Red Hat on a technical level.

So while all changes to Red Hat Enterprise Linux will at a point go
through Bugzilla, this difference has a number of important consequences
for general product issues filed directly through Bugzilla by external
users without any privileged Engineering relationship:

    * Red Hat does NOT guarantee any response time or Service Level
Agreement (SLA) for Bugzilla entries. - A review might happen
immediately or after a time span of any length. The SLAs for Red Hat
Enterprise Linux provided by Red Hat Support can be found at:
https://www.redhat.com/support/policy/sla/production/

    * Not all comments are publicly visible. - Red Hat Support provides
customers with appropriate information excerpts and status updates from
that. Red Hat does not commit to provide detailed explanations, or
guidance in the context of Bugzilla. Therefore for example, Bugzilla
entries might be closed as it seems without any further explanation,
while Red Hat Support actually provides such explanation to customers
filing through the regular support channels.

    * Issues coming through the regular support process, will always be
prioritized higher than issues of similar impact and severity that are
being filed directly through Bugzilla (unless the later get linked to
customer support tickets). This means that they are more likely to be 
addressed and they are more likely to meet inclusion criteria 
consistent with the Red Hat Enterprise Linux life cycle policy:
http://www.redhat.com/security/updates/errata/

    * Work-arounds and Hotfixes if possible and appropriate are provided
by Red Hat Support and through the regular support process. - This means
that even before a permanent fix is made available through RHN, customers
who raised a high severity issue through Red Hat Support, are likely to
receive an interim solution.

Red Hat provides common Bugzilla access in order provide efficient
development community interaction and as much transparency as possible
to our customers. Our Engineers are encouraged to provide non-customer
specific and non-confidential information publicly as often as possible.

So while Red Hat considers issues directly entered into Bugzilla
valuable feedback - may it be as comments to existing Bugzilla entries
or by opening a new one; for customers encountering production issues,
Bugzilla is not the right channel.

Therefore we ask our customers to file requests important for their
production systems via our Support service. Only for those issues, we
can ensure a consistent communication. Information about our production
support process can be found at: http://www.redhat.com/support/process/

Bugzilla can always be used as a supportive channel to that
communication.

Note: If your customer is participating in the Academic program and has
chosen to run a Subscription without support, they consequently have no
access to Red Hat Support and thus no SLA. If you feel that this is
insufficient for your use case, you should consider contacting the Red
Hat Education specialist as described at:
http://www.redhat.com/solutions/education/academic/individual/

There are no plans to add SRP storage to Red Hat Enterprise Linux installation.

Product Management has reviewed and declined this request.  You may appeal this
decision by reopening this request.

Mike, we *already* support SRP storage in RHEL.

The modules are already there, as is the SRP daemon.

This request was for adding SRP support to Anaconda as well, not for adding SRP support to RHEL, which as mentioned, is already in place.

(In reply to comment #27)
> Mike, we *already* support SRP storage in RHEL.
> 
> The modules are already there, as is the SRP daemon.
> 
> This request was for adding SRP support to Anaconda as well, not for adding SRP
> support to RHEL, which as mentioned, is already in place.

Clarification: word "installation" in comment 25 was referring to Anaconda.