Bug 736754

Jacquelynn,

are you looking for more info then covered below? If you're, I'd need
the details and I'll also need a couple of hosts to play with.

On the server:

1. install rdma rpm and its prereqs
2. chkconfig --add nfs-rdma (steved - is if yours of 
   ledford's to make it automagic?)
3. make sure NFSoRDMA_LOAD is set to yes in /etc/rdma/rdma.conf

On the client

1. install rdma rpm and its prereqs
2. use 'rdma' option when mounting, e.g.

   mount -t nfs -o rdma dumpy:/bitbucket /mnt/bits

(In reply to comment #6)
> Jacquelynn,
> 
> are you looking for more info then covered below? If you're, I'd need
> the details and I'll also need a couple of hosts to play with.
> 
> On the server:
> 
> 1. install rdma rpm and its prereqs
> 2. chkconfig --add nfs-rdma (steved - is if yours of 
>    ledford's to make it automagic?)
> 3. make sure NFSoRDMA_LOAD is set to yes in /etc/rdma/rdma.conf
This all new to me... 

The "Check RDMA and NFS Setup' and 'NFS/RDMA Setup' sections in:
   http://kernel.org/doc/Documentation/filesystems/nfs/nfs-rdma.txt
are what I used to set up my NFSoRDMA testing on both the 

Doug, are those scripts supported, tested and or documented?
I see they rdma is also in Fedora, but have not been converted 
to uses systemd.

Also note, to enable RDMA support on the server side and have all
the correct modules loaded, uncomment RDMA_PORT in /etc/sysconfig/nfs
I added this to the RHEL6.1 nfs-utils.

So it appears there is two ways to bring up the server... We
probably should consolidate this into one...  Thoughts?

I'm going to be rebuilding the rdma package before snap4 user space deadline I think, so I can get any needed changes in.  However, all my startup script really does is enable the extra rdma port and load the rdma related transfer modules.  One thing I'm concerned about though is changing configuration methods mid stream.  People using one method or the other might get confused when it goes away unexpectedly.

I forgot to answer Steve's question from comment #7.  As for any file in the rdma package, I own and am the upstream for that package and I support all those files.  Now, that's not to say I think my nfs-rdma script has to take precedence over anything else, I just added it because back when I added it, there wasn't another option to automatically bring up rdma support at each boot.

As for documentation, the nfs-rdma stuff has not been documented, although the /etc/rdma/rdma.conf file and its role in rdma system configuration has been documented on our knowledge base.  If people go poking around in that file looking at other options because of docs on the knowledge base, then they are likely to see the NFSoRDMA option and figure the rest out for themselves (although they might not get that they need to enable the initscript as well).

So, I would say partially documented and easy for people to put two and two together.

And they are tested, although as you point out in Fedora I have yet to convert them to systemd based services.

Pity that the left foot didn't know about the actions of the right hand.

To me Doug's script looks more polished and more likely to
make things work: to have NFS/RDMA server up and running one
would need rdma_cm and ipoib loaded - svcrdma has dependency on rmda_cm
but not on ipoib, Doug's nfs-rdma has dependency on $rdma which is loading
ipoib if configured which seems to be a more 'logical' approach.

I'd say we "de-emphasise" RDMA_PORT in /etc/sysconfig/nfs in preference
to nfs-rdma.

(In reply to comment #11)
> Pity that the left foot didn't know about the actions of the right hand.
:-)

> 
> To me Doug's script looks more polished and more likely to
> make things work: to have NFS/RDMA server up and running one
> would need rdma_cm and ipoib loaded - svcrdma has dependency on rmda_cm
> but not on ipoib, Doug's nfs-rdma has dependency on $rdma which is loading
> ipoib if configured which seems to be a more 'logical' approach.
> 
> I'd say we "de-emphasise" RDMA_PORT in /etc/sysconfig/nfs in preference
> to nfs-rdma.
I agree for RHEL6 but I think these scripts should be rolled into
nfs-utils for RHEL7 since 1) everything changes with systemd anyways
2) Having to install/enable/start two packages just to get NFSoRDMA
support in the server is not ideal...

So Jacquelynn what is needed to move forward with this?

Actually, I'd do it the other way around Steve (meaning instead of rolling the nfs-rdma script into nfs-utils, I'd leave it separate).  Here's my reasoning:

The rdma stack is not simple, and requires a rather complex script to bring up properly (see /etc/init.d/rdma).  It also has lots of options.  And most of it is totally unrelated to nfs, so there's no way rolling this into nfs-utils makes any sense.

The rdma portion of nfs depends on the rdma stack being up and in a usable condition in order for rdma support to actually work.  From a packaging standpoint, that means you really want the nfs-rdma support to have a requirement on the base rdma support.  But, for any non-rdma installation, the entire rdma stack is a waste of time.  So it would be best if we could install base nfs support without requiring the rdma stack, but if we *do* include nfs-rdma support, then we want a hard requirement on the rdma package.

What this means is that we really want the nfs-rdma support in its own package.  Whether it is built as a subpackage of nfs-utils and is called nfs-utils-rdma and Requires: rdma in just the nfs-utils-rdma portion of the spec file or whether it is in the rdma package directly doesn't matter to me.  But if it gets rolled into the base nfs-utils package then you need to also add a Requires: rdma to the base nfs-utils package and we *don't* want that.

So, personally, I think it being in the base rdma package makes perfect sense.  It means that nfs-utils doesn't have an unnecessary dependency on the rdma package, and it means that you will always have the rdma package installed when you enable nfs-rdma.  Really, the rdma support in nfs is a subcomponent of the overall nfs stack, and an only somewhat used one at that, so I would treat it as such instead of treating it as though it belongs in the base nfs implementation (analogous to the fact that ISDN is a subcomponent of dialup networking and is therefore separated out since only a select few users need it).

(In reply to comment #13)
> Actually, I'd do it the other way around Steve (meaning instead of rolling the
> nfs-rdma script into nfs-utils, I'd leave it separate).  Here's my reasoning:
> 
> The rdma stack is not simple, and requires a rather complex script to bring up
> properly (see /etc/init.d/rdma).  It also has lots of options.  And most of it
> is totally unrelated to nfs, so there's no way rolling this into nfs-utils
> makes any sense.
Right.... I was just thinking about the nfs-rdma script so two
scripts do not have to be run to start the nfs server.

> 
> The rdma portion of nfs depends on the rdma stack being up and in a usable
> condition in order for rdma support to actually work.  From a packaging
> standpoint, that means you really want the nfs-rdma support to have a
> requirement on the base rdma support.  But, for any non-rdma installation, the
> entire rdma stack is a waste of time.  So it would be best if we could install
> base nfs support without requiring the rdma stack, but if we *do* include
> nfs-rdma support, then we want a hard requirement on the rdma package.
In the systemd world this will be take care with an "Requires" and "After" 
requirement.If foresee a systemd service called nfs-rdma-server.service.
In that service there would the following 2 line:
    Requires="rdma nfs-server.service" 
    After="rdma nfs-server.service" 
which would cause both the rdma and NFS server to be installed and
up and running.

 
> 
> What this means is that we really want the nfs-rdma support in its own package.
>  Whether it is built as a subpackage of nfs-utils and is called nfs-utils-rdma
> and Requires: rdma in just the nfs-utils-rdma portion of the spec file or
> whether it is in the rdma package directly doesn't matter to me. 
I guess I do see the wisdom of making it its own package. Plus
making it a sub package of nfs-utils would allow me to replace
the current RDMA support in /etc/sysconfig/nfs (which would be a 
good thing IMHO)... 

> But if it
> gets rolled into the base nfs-utils package then you need to also add a
> Requires: rdma to the base nfs-utils package and we *don't* want that.
You can could put a Requires: on a sub packages. Something similar to:

%package rdma
Summary: NFS over RDMA server support
Group:  System Environment/Base
Requires:   %{name} = %{version}-%{release}
Requires(rdma): rdma

> 
> So, personally, I think it being in the base rdma package makes perfect sense. 
> It means that nfs-utils doesn't have an unnecessary dependency on the rdma
> package, and it means that you will always have the rdma package installed when
> you enable nfs-rdma. 
The requirement of rdma  would be on the nfs-rdma-utils package not 
the normal nfs-utils package.
 
> Really, the rdma support in nfs is a subcomponent of the
> overall nfs stack, and an only somewhat used one at that, so I would treat it
> as such instead of treating it as though it belongs in the base nfs
> implementation (analogous to the fact that ISDN is a subcomponent of dialup
> networking and is therefore separated out since only a select few users need
> it).
From a support standpoint of view, I'm going to have to disagree... 
Have in that script part of the nfs-utils package will make it easier
for things to change. For example, the default port nfs-rdma is
different that the port used in the kernel doc (in Comment 7). Now
I agree this is a nit, but I think it proves my point. When the NFS
community makes changes in this area, it would be much easier for 
me to make the change if it was n nfs-utils then bothering you 
to make it.. 

So in summary, I think we both agree it the NFSoRDMA support should 
be its own package but we don't agree on which package should be a 
part of.... which we can hash out later... 

So Jacquelynn, what this means to you is, you need to document the
functionality of the /etc/rc.d/init.d/nfs-rdma init script since, 
regardless of who ones it, the functionality will not change...

*** Bug 730158 has been marked as a duplicate of this bug. ***