RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 463530 - [LTC 6.0 FEAT] 201026:IPv6 support in NFS - kernel
Summary: [LTC 6.0 FEAT] 201026:IPv6 support in NFS - kernel
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.0
Hardware: All
OS: All
high
high
Target Milestone: alpha
: 6.0
Assignee: Jeff Layton
QA Contact: Filesystem QE
URL:
Whiteboard:
: 467711 470030 479349 (view as bug list)
Depends On:
Blocks: 217209 356741 554559
TreeView+ depends on / blocked
 
Reported: 2008-09-23 20:31 UTC by IBM Bug Proxy
Modified: 2011-12-10 06:43 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-07-12 12:36:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 47948 0 None None None Never

Description IBM Bug Proxy 2008-09-23 20:31:06 UTC
=Comment: #1=================================================
Emily J. Ratliff <emilyr.com> - 2008-09-17 11:13 EDT
1. Feature Overview:
Feature Id:	[201026]
a. Name of Feature:	IPv6 support in NFS - kernel
b. Feature Description
To add the support of IPv6 in NFS.  Several components: 1) The client side IPv6
support should be completed in 2.6.21. 2) The server side IPv6 support is in
progress. 3) The NFS commands/tools/utils packages is already available since 1.0.10

2. Feature Details:
Sponsor:	LTC
Architectures:
x86
x86_64
ppc64
s390 native
s390 compat
s390x

Arch Specificity: Purely Common Code
Delivery Mechanism: 
Category:	Remote Filesystems
Request Type:	Kernel - Enhancement from IBM
d. Upstream Acceptance:	In Progress
Sponsor Priority	1
f. Severity: High
IBM Confidential:	no
Code Contribution:	3rd party code
g. Component Version Target:	server support in progress, other parts are
upstream:  2.6.21 -> client support  nfs-utils 1.0.10 -> userspace tools (in a
separate request)

3. Business Case
IPv6 support in NFS is needed for several customers and is required for some
government contracts (all DoD contracts).

Benefits
NFS over IPv6 would provide more flexibility to customers to configure their
networks to meet their needs.

4. Primary contact at Red Hat: 
John Jarvis
jjarvis

5. Primary contacts at Partner:
Project Management Contact:
Sarah Wright, sarahw.com, 503-578-5145

Technical contact(s):
Frank Filz, ffilz.com
Varun Chandramohan, varuncha.com

IBM Manager:
Jeffrey Heroux, heroux.com

Comment 1 Bill Nottingham 2008-10-02 19:23:55 UTC
We should have the client and utilities handled fine in RHEL 6. What's the status of the server support?

Comment 2 IBM Bug Proxy 2008-10-03 03:16:53 UTC
The server support is still not ready as of 2.6.27-rc6. I have been able to get a successful mount over ipv6 but with lots of undesired hacks. The main kernel patches i used are http://nfsv4.bullopensource.org/patches/ipv6-server/2.6.27-rc3/patch1_rpcbindv4 and http://nfsv4.bullopensource.org/patches/ipv6-server/2.6.27-rc3/patch2_ipv6overrpcbindv4. They both seems to work fine. But they are not upstream yet.

Comment 3 Jeff Layton 2008-10-28 15:38:30 UTC
Looks like I've been tasked with this. Still haven't looked over the current state yet, but I've been loosely following the patches on the mailing lists.

Comment 4 Jeff Layton 2008-11-05 11:45:49 UTC
Since it's not clear what should happen to these RHEL6 feature requests from a procedure standpoint, I'm going to toss this back to the kernel-mgr queue and open a separate rawhide BZ to track it.

Comment 5 Jeff Layton 2008-11-05 11:49:47 UTC
The BZ opened for rawhide is bug 470030

Comment 6 IBM Bug Proxy 2009-01-28 19:01:16 UTC
Varun is no longer working on NFS over IPv6

Comment 7 IBM Bug Proxy 2009-02-05 20:51:59 UTC
Here is an update from Chuck Lever on the status of NFS support fpor IPv6:

On Jan 28, 2009, at Jan 28, 2009, 2:34 PM, Frank S Filz wrote:
> I'm trying to determine the current status of IPv6 support for NFS.
> I understand support is nearly complete but a few pieces may not be
> quite complete.

You should be able to mount NFSv4 servers from the Linux NFS client in
2.6.28, using the mount.nfs command from the latest nfs-utils.
Callback over IPv6 is supported, and maybe referral too.

With 2.6.30, IPv6 will be supported in all the kernel NFS client and
server components (rpcbind and mountd client, NLM, NSM, NFS client,
and NFSD), though I wager it will need some shake-down time as user
space support for IPv6 is still lagging.

Distributions need to carry the new libtirpc and replace their
portmapper with rpcbind.  Fedora, so far, is the only one that is
doing this, although I think RHEL 6 and the next release of OpenSUSE
will have it too.

The next upstream release of nfs-utils will have showmount and
complete [u]mount.nfs support for IPv6 (necessary for mounting NFSv2/
v3 servers over IPv6, but not sufficient).

I have sm-notify support done, and am working on rpc.statd now for
testing at Connectathon 2009.  This will complete client side support
for NFSv2/v3 over IPv6, and is also necessary but not sufficient for
server side NFSv2/v3 over IPv6.

The next piece is rpc.mountd and exportfs to complete server side IPv6
support, as NFS server access control is IP address-based.  Bull had
something here, but I haven't seen that work, and they've been defunded.

We haven't looked at Kerberos (GSSAPI) yet, but I don't think there
will be significant issues there.  Jeff (cc added) might be looking at
this.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Comment 8 IBM Bug Proxy 2009-03-02 17:11:27 UTC
As mentioned above, upstream submission/acceptance is in progress. Moving this along to submitted state.

Comment 9 Jeff Layton 2009-04-02 12:18:02 UTC
*** Bug 479349 has been marked as a duplicate of this bug. ***

Comment 10 Chuck Lever 2009-04-02 16:36:01 UTC
Here are some updates and corrections.

We are using

  http://wiki.linux-nfs.org/wiki/index.php/Ipv6PlanningDocument

as a working document for upstream IPv6 NFS support.

2.6.30 should have complete client side support for all versions of NFS on IPv6.  This includes the NFSv4 callback service and support for NFSv2/v3 locking and NSM.  Jeff's done a little testing that demonstrates GSS authentication support in the kernel is already working for IPv6.

Steve Dickson has included libtirpc and rpcbind in Fedora user space since Fedora 9.

nfs-utils 1.1.5 (not 1.0.10) has full mount.nfs support for mounting IPv6 servers.  This includes mount, unmount, and v2/v3 version fallback (version and transport protocol determined by negotiation, as with IPv4).

I have a prototype of rpc.statd and the sm-notify command that support IPv6, which are required for NFSv2/v3 lock recovery.  These are available at git.linux-nfs.org.

Jeff is about to provide IPv6 support for gssd, which will allow clients to mount IPv6 servers with krb5 authentication.  I'm guessing that I will tuck those patches in my git repo for people to test.

Groupe Bull is no longer working on NFS/IPv6, but I have a git repository at git.linux-nfs.org containing server-side IPv6 patches.  Bull has tested these on previous kernels, and they basically work.  We do not have any server-side user space support for these yet.  In addition to rpc.statd/sm-notify, this would include IPv6 support in rpc.mountd, support for IPv6 addresses in the exportfs command, and support for IPv6 in rpc.svcgssd.

We still need to address issues with IPv6 support for LDAP lookups in rpc.idmapd.  We don't know if link-local address support should be required: universal address strings, used by NFSv4 and rpcbindv4, do not support scope IDs, for example.  TCP wrappers do not currently support IPv6.

Naturally we require a hardened O/S implementation of IPv6 for NFS/IPv6 to work.  IPv6 support in Linux networking infrastructure (especially user space utilities and firewalls) needs to be thoroughly vetted.

Finally we have been uncovering plenty of issues due to incompatibilities between 32- and 64-bit hardware/compilers, and due to more stringent alignment issues on 64-bit architectures.  As we build out our test harnesses we will be knocking these off one-by-one.

Comment 11 Chuck Lever 2009-04-02 16:41:06 UTC
I just noticed that SGI has open-sourced its NFS testing suites, including a tester for portmap/rpcbind, and tests that exercise the user space RPC libraries.

Comment 12 Jeff Layton 2009-04-09 15:19:38 UTC
Thanks for the excellent writeup, Chuck. Here's a bit of an update:

Earlier this week, I sent a patchset for rpc.gssd to Steve and asked him to commit it to upstream nfs-utils. Thanks to Chuck for the excellent review and help.

For the client side, I believe all we're really waiting on at this point is rpc.statd. Chuck has decided to take a crack at reimplementing it basically from scratch. His ballpark estimate is ~2 months before he'll have it done.

The existing rpc.statd is very problematic. Most of these problems don't have anything to do with IPv6, but since it'll have to be overhauled for that anyway it makes sense to try and address some of the other problems as well.

The remaining pieces are server-side. I've started looking over the Bull patches. They're not really suitable as-is, but can help serve to give
us some ideas about what needs to be changed.

From a cursory look, I don't expect that we'll need a lot of changes at the RPC level. That's good since RPC-level changes are more likely to be problematic for kabi.

Most of the server-side changes I think we'll need are in knfsd itself and in the userspace tools. I'd like to see if we can clean up the knfsd interfaces so that we don't rely on nfsctl() anymore. I think that was part of the idea behind adding the nfsd pseudo-fs, but I don't think that its interface is complete.

Comment 13 Chuck Lever 2009-04-09 15:36:10 UTC
(In reply to comment #12)
> From a cursory look, I don't expect that we'll need a lot of changes at the RPC
> level. That's good since RPC-level changes are more likely to be problematic
> for kabi.
> 
> Most of the server-side changes I think we'll need are in knfsd itself and in
> the userspace tools. I'd like to see if we can clean up the knfsd interfaces so
> that we don't rely on nfsctl() anymore. I think that was part of the idea
> behind adding the nfsd pseudo-fs, but I don't think that its interface is
> complete.  

The server-side kernel pieces are already written, and have had some upstream review.  You can find them in my kernel git repository on linux-nfs.org.  These feature a clean-up of the nfsctl code.  It's worth noting that the existing nfsctl API doesn't change much or at all to support IPv6.

I have held back on submitting these for a couple of reasons.

First, we can't test them until we have server-side IPv6 support in user space.  These have seen a little testing by Bull with their user space prototype.

Second, the server-side kernel RPC support that is already in mainline has seen some changes now that it's being used by the client side (lockd and NFSv4 callback).  I think that churn should be mostly done now.

However, if you have kABI concerns, it's worth taking a look at these now.

Comment 14 Chuck Lever 2009-04-09 15:50:00 UTC
(In reply to comment #12)
> For the client side, I believe all we're really waiting on at this point is
> rpc.statd. Chuck has decided to take a crack at reimplementing it basically
> from scratch. His ballpark estimate is ~2 months before he'll have it done.
> 
> The existing rpc.statd is very problematic. Most of these problems don't have
> anything to do with IPv6, but since it'll have to be overhauled for that anyway
> it makes sense to try and address some of the other problems as well.

Filling in some technical detail:

The current statd implementation dates to 1995.  It has seen 15 years of expedient changes (bug fixes) with little other attention.  In 2006 Olaf Kirch separated the SM_NOTIFY functionality because he was trying to introduce kernel support for NSM listeners, but kernel NSM support never caught and has been abandoned.

Both statd and sm-notify open-code an RPC scheduler used to send NLM callbacks and reboot notifications.  The NSM listeners in statd are folded into this ad-hoc scheduler.  It open-codes PMAP_GETPORT as well.  A big advantage is that a single socket can be used for multiple RPC programs, but the downside is it is complex and fragile code, and supports on UDP transports.

For IPv6 support, I originally open-coded support for RPCB_GETADDR and fit it into the current model.  However because of the age of the code, a number of problems unrelated to IPv6 became readily apparent.  In addition, the NSM protocol itself is inadequate for typical modern deployments, and we want to be in a position to replace loopback RPC communication with a richer API (possibly using rpc_pipefs, as does gssd and idmapd).

Jeff and others tested the original prototype successfully at the 2009 Connectathon event.  Since then I've decided to re-implement statd and sm-notify from the ground up, making full use of TI-RPC to replace the open-coded RPC logic in the current implementation.  Progress has been good.

The re-implementation will be backwards compatible with current kernels, but will require TI-RPC in user space.

Comment 15 Jeff Layton 2009-05-04 11:48:58 UTC
*** Bug 470030 has been marked as a duplicate of this bug. ***

Comment 16 Jeff Layton 2009-05-04 12:58:50 UTC
(In reply to comment #13)
> The server-side kernel pieces are already written, and have had some upstream
> review.  You can find them in my kernel git repository on linux-nfs.org.  These
> feature a clean-up of the nfsctl code.  It's worth noting that the existing
> nfsctl API doesn't change much or at all to support IPv6.
> 

Sorry for the long delay, I've been sidetracked with other (mostly CIFS-related) concerns for the last few weeks. I'm ready to start working on the server-side stuff in earnest now...

I've been giving the patches in your tree a once-over. Am I correct that most of them are now in Bruce's tree? I see where you've added support for some of the things we'll need for IPv6 support, but the kernel code doesn't appear to be complete. In particular it doesn't look like nfsd creates PF_INET6 listeners yet. It doesn't look too hard to add that in, so maybe I'll hack something together for the purposes of working on mountd/exportfs.

Also, isn't the nfsctl syscall considered deprecated? Shouldn't we consider adding a way to turn different protocols on and off via nfsdfs (/proc/fs/nfsd)?

One idea...on my (unpatched) box:

$ cat /proc/fs/nfsd/portlist
tcp 2049
udp 2049

...maybe we should extend it so that it shows something like:

tcp 2049
udp 2049
tcp6 2049
udp6 2049

...and make it so that writing similar values to it enable and disable the right family and xprt. Another possibility is to add a separate field to this file for the address family, and make it so that that if that field doesn't exist that PF_INET is assumed.

Thoughts?

Comment 17 Chuck Lever 2009-05-04 14:41:14 UTC
(In reply to comment #16)
> (In reply to comment #13)
> > The server-side kernel pieces are already written, and have had some upstream
> > review.  You can find them in my kernel git repository on linux-nfs.org.  These
> > feature a clean-up of the nfsctl code.  It's worth noting that the existing
> > nfsctl API doesn't change much or at all to support IPv6.
> > 
> 
> Sorry for the long delay, I've been sidetracked with other (mostly
> CIFS-related) concerns for the last few weeks. I'm ready to start working on
> the server-side stuff in earnest now...
> 
> I've been giving the patches in your tree a once-over. Am I correct that most
> of them are now in Bruce's tree?

At this point, the final IPv6 NFSD support is still in my repo and not in 2.6.30-rc or 2.6.31.  We've decided to keep those final patches out until we have user space ready for testing.

> I see where you've added support for some of
> the things we'll need for IPv6 support, but the kernel code doesn't appear to
> be complete. In particular it doesn't look like nfsd creates PF_INET6 listeners
> yet. It doesn't look too hard to add that in, so maybe I'll hack something
> together for the purposes of working on mountd/exportfs.
> 
> Also, isn't the nfsctl syscall considered deprecated? Shouldn't we consider
> adding a way to turn different protocols on and off via nfsdfs (/proc/fs/nfsd)?
> 
> One idea...on my (unpatched) box:
> 
> $ cat /proc/fs/nfsd/portlist
> tcp 2049
> udp 2049
> 
> ...maybe we should extend it so that it shows something like:
> 
> tcp 2049
> udp 2049
> tcp6 2049
> udp6 2049
> 
> ...and make it so that writing similar values to it enable and disable the
> right family and xprt. Another possibility is to add a separate field to this
> file for the address family, and make it so that that if that field doesn't
> exist that PF_INET is assumed.

All this should already be in my tree.

The legacy nfsctl interface won't ever support IPv6 because the API was incorrectly defined.  The passed-in binary blobs include fields of type "struct sockaddr." This type is the size of "struct sockaddr_in" on Linux, so it won't hold an IPv6 address.  There doesn't appear to be way of versioning these blobs.

Comment 18 Jeff Layton 2009-05-04 17:52:21 UTC
Ahh yes...forgot that you're leaving the master branch pristine and are working with date tags. Checked out the cel-ipv6-04132009 and it builds. A couple of minor problems so far...

When I started up nfs, it didn't start the listeners for IPv6. I assume this is due to the problem you mentioned with the nfsctl syscall. We may need to consider rewriting the nfsd binary in nfs_utils to use nfsdfs instead of nfsctl. The program is already pretty tiny, so I don't think it'll be too tough. In principle, it could even be a shell script...

Anyway, I removed the ports that the nfsd program started up, and did:

# echo 'tcp 2049' > /proc/fs/nfsd/portlist
# echo 'udp 2049' > /proc/fs/nfsd/portlist

...then:

# cat /proc/fs/nfsd/portlist
udp 2049
udp 2049
tcp 2049
tcp 2049

...so we see 2 entries for each transport type, which makes sense since we have 2 sockets for each. I think though it would be more intuitive to shoot for an output in this case like:

# cat /proc/fs/nfsd/portlist
udp 2049
udp6 2049
tcp 2049
tcp6 2049

...but we could also consider adding a new field maybe:

# cat /proc/fs/nfsd/portlist
udp 2049
udp 2049 6
tcp 2049
tcp 2049 6


...not sure which one is worse from the perspective of a change in user-level interfaces.

Chuck, can you clarify how you envision this working?

Comment 19 Jeff Layton 2009-05-04 17:59:18 UTC
Another problem too...removing and adding interfaces via /proc/fs/nfsd/portlist
also seems to have no effect on rpcbind registrations.

Comment 20 Chuck Lever 2009-05-04 19:05:55 UTC
(In reply to comment #19)
> Another problem too...removing and adding interfaces via /proc/fs/nfsd/portlist
> also seems to have no effect on rpcbind registrations.  

Are you sure NFSD has AF_INET6 listeners?

I should note that none of this code is tested, as I said before, except for the NFSv4-only testing that Bull did a while back with their own user space.  Since then I've had to rewrite svc_create_xprt() to handle someone possibly unloading ipv6.ko, so the latest version of this code is untested.

That said:

> When I started up nfs, it didn't start the listeners for IPv6. I assume this is
> due to the problem you mentioned with the nfsctl syscall. We may need to
> consider rewriting the nfsd binary in nfs_utils to use nfsdfs instead of
> nfsctl. The program is already pretty tiny, so I don't think it'll be too
> tough. In principle, it could even be a shell script...

NFSD should attempt to start AF_INET6 listeners automatically if CONFIG_IPV6 or CONFIG_IPV6_MODULE are set.  My version of nfsd_init_socks() does this, but perhaps the 4/13 tag doesn't have that patch.

> # cat /proc/fs/nfsd/portlist
> udp 2049
> udp 2049
> tcp 2049
> tcp 2049

The xcl_name field is displayed here.  We need to decide whether we want to see netids here, whether we want to see a standard protocol name and an address family, or something else.  My initial suggestion is to display netids here, and that will likely require some jiggling of the server-side transport switch.

Comment 21 Chuck Lever 2009-05-04 22:27:06 UTC
Looking at this briefly, I think Fedora's rpc.nfsd is creating sockets in user space and passing them to the kernel NFSD by file descriptor.  From what I can see the automatic listener creation logic in nfsd_init_socks() is entirely bypassed in this arrangement.

You might try changing rpc.nfsd to create PF_INET6 listener sockets and passing them in via portlist.

Comment 22 Jeff Layton 2009-05-05 13:44:35 UTC
No, the version I'm using is the one in rawhide and that one uses nfsctl() to start it up.  When I echo the right lines into "portlist", ipv6 listeners are definitely started. I had a look at the differences between the 04/13 tag and your latest tag and didn't see a lot of changes there, but I may have missed something.

I think we'll probably want to rewrite rpc.nfsd to use the files in /proc/fs/nfsd if they exist (that's a conversion that's long overdue anyway). I'll have a look at that when I have a chance.

I'm not exactly clear on the best way yet to fix portlist. Using netid-like strings for the transport seems like the right thing to do, but we have to be cognizant of the fact that the netid's in userspace and kernel might be different. If you have more detailed thoughts on this though, I'd welcome them.

In the meantime, I started playing with attempting mounts on a kernel with your cel-ipv6-04132009 patches:

# cat /proc/net/rpc/auth.unix.ip/content 
#class IP domain
nfsd 0.0.0.0 -test-client-
# nfsd feed:0000:0000:0000:0000:0000:0000:0022 -no-domain-

...so the main hurdle at this point is to make mountd respond appropriately to upcalls in that format.

That though also means that we need to allow people to specify IPv6 addresses in /etc/exports. I'll pull up a solaris machine in the near future and have a look at how they do their IPv6 address specifications for exports.

Comment 23 Chuck Lever 2009-05-05 15:29:08 UTC
(In reply to comment #22)
> No, the version I'm using is the one in rawhide and that one uses nfsctl() to
> start it up.  When I echo the right lines into "portlist", ipv6 listeners are
> definitely started. I had a look at the differences between the 04/13 tag and
> your latest tag and didn't see a lot of changes there, but I may have missed
> something.

Is rawhide's rpc.nfsd different than what's in Fedora 10?  My tracing does not show that nfsd_init_socks() is ever called on my system.  The only explanation I have for that is that rpc.nfsd is starting each listener separately.

> I think we'll probably want to rewrite rpc.nfsd to use the files in
> /proc/fs/nfsd if they exist (that's a conversion that's long overdue anyway).
> I'll have a look at that when I have a chance.

Originally this design was going to use a single listener for both AF_INET and AF_INET6 traffic.  So, one listener for UDP and one listener for TCP, but both would handle all protocol families.  Because of the requirement to operate in kernels that have IPv6 built in, but ipv6.ko is not loadable at run-time, that design was abandoned recently in favor of separate listeners for each protocol and protocol family.  This API has not been updated with those changes, as I wanted to see the new "separate listeners" design go upstream first (to vindicate the design choices there).

Passing in socket file descriptors might be the best way to proceed.  A socket provides a protocol family (inet/inet6), a transport protocol (UDP datagrams or TCP streams), a listener bind address (today it's always 0.0.0.0, but can be anything if we want to get wild with support for multi-homed hosts) and a listener port number (usually 2049) all in a single integer.

That way rpc.nfsd has explicit control over the properties of all listeners.

> I'm not exactly clear on the best way yet to fix portlist. Using netid-like
> strings for the transport seems like the right thing to do, but we have to be
> cognizant of the fact that the netid's in userspace and kernel might be
> different.

This API was augmented to support RDMA transports in the absense of any netid support because a netid string for RDMA hadn't been allocated by the appropriate standards body when server-side RDMA support was ready to go upstream.

As you point out, the kernel's netid support is not dependent in any way on the contents of /etc/netconfig, so it might be best, instead of using netids, to spell out the characteristics of each listener like this:

"protocol" "port" "protocol family" "bindaddr"

We could have something like this:

udp 2049 inet 192.168.0.0
udp 2049 inet6

Meaning we have an ANYADDR AF_INET6 listener, and a listener for AF_INET traffic on 192.168.0/24.

rpc.nfsd can also make some NFS versions available and not others.  It seems to me we might want to allow NFSv4.x on TCP, but not on UDP, for example.  Does the current API make that possible?

In any event, I don't have a clear picture of backwards compatibility requirements for this API, nor how all of the /proc files are supposed to work together.  I think we should take this design discussion to linux-nfs.org so that Bruce, Tom, and others can participate.

> In the meantime, I started playing with attempting mounts on a kernel with your
> cel-ipv6-04132009 patches:
> 
> # cat /proc/net/rpc/auth.unix.ip/content 
> #class IP domain
> nfsd 0.0.0.0 -test-client-
> # nfsd feed:0000:0000:0000:0000:0000:0000:0022 -no-domain-
> 
> ...so the main hurdle at this point is to make mountd respond appropriately to
> upcalls in that format.
> 
> That though also means that we need to allow people to specify IPv6 addresses
> in /etc/exports. I'll pull up a solaris machine in the near future and have a
> look at how they do their IPv6 address specifications for exports.  

Yes, all as expected.  You should see if the Bull web site has published versions of their server-side user space.

Comment 24 Jeff Layton 2009-05-05 19:49:01 UTC
(In reply to comment #23)
> Is rawhide's rpc.nfsd different than what's in Fedora 10?  My tracing does not
> show that nfsd_init_socks() is ever called on my system.  The only explanation
> I have for that is that rpc.nfsd is starting each listener separately.
> 

No, I'm pretty sure it's the same. rpc.nfsd has been relatively unchanged for years.

My tracing showed that nfsd_init_socks is called, but it happens after some entries have already been placed on sv_permsocks. It then does this, which makes it a no-op.

        if (!list_empty(&nfsd_serv->sv_permsocks))
                return 0;

Still looking at this and also trying to determine why nfsd is calling nfsctl() at all. It looks like it should be using the new interfaces if I'm reading nfssvc.c correctly.

> 
> Originally this design was going to use a single listener for both AF_INET and
> AF_INET6 traffic.  So, one listener for UDP and one listener for TCP, but both
> would handle all protocol families.  Because of the requirement to operate in
> kernels that have IPv6 built in, but ipv6.ko is not loadable at run-time, that
> design was abandoned recently in favor of separate listeners for each protocol
> and protocol family.  This API has not been updated with those changes, as I
> wanted to see the new "separate listeners" design go upstream first (to
> vindicate the design choices there).
> 
> Passing in socket file descriptors might be the best way to proceed.  A socket
> provides a protocol family (inet/inet6), a transport protocol (UDP datagrams or
> TCP streams), a listener bind address (today it's always 0.0.0.0, but can be
> anything if we want to get wild with support for multi-homed hosts) and a
> listener port number (usually 2049) all in a single integer.
> 
> That way rpc.nfsd has explicit control over the properties of all listeners.
> 

Ahh ok. That makes sense. I'll admit though that I'm not crazy about passing in sockets like that. Since we'll want to report back the info via a text based interface, it makes sense that we should allow changes to that info via a similar method.

> 
> This API was augmented to support RDMA transports in the absense of any netid
> support because a netid string for RDMA hadn't been allocated by the
> appropriate standards body when server-side RDMA support was ready to go
> upstream.
> 
> As you point out, the kernel's netid support is not dependent in any way on the
> contents of /etc/netconfig, so it might be best, instead of using netids, to
> spell out the characteristics of each listener like this:
> 
> "protocol" "port" "protocol family" "bindaddr"
> 
> We could have something like this:
> 
> udp 2049 inet 192.168.0.0
> udp 2049 inet6
> 
> Meaning we have an ANYADDR AF_INET6 listener, and a listener for AF_INET
> traffic on 192.168.0/24.
> 
> rpc.nfsd can also make some NFS versions available and not others.  It seems to
> me we might want to allow NFSv4.x on TCP, but not on UDP, for example.  Does
> the current API make that possible?
> 

Not sure. I'll have a look at that though. It seems sensible to allow that sort of granularity.

> In any event, I don't have a clear picture of backwards compatibility
> requirements for this API, nor how all of the /proc files are supposed to work
> together.  I think we should take this design discussion to
> linux-nfs.org so that Bruce, Tom, and others can participate.
> 

I'll see about writing something up for list consumption in the next day or two. My guess is that we'll probably need to allow backward compatibility with these interfaces (i.e. they need to work with older nfs-utils). Whether that precludes adding new fields to the ends of the lines, I'm not sure.

Before I do any of that I want to get a better idea of why the rpc.nfsd binary doesn't seem to be doing what I expect.

> 
> Yes, all as expected.  You should see if the Bull web site has published
> versions of their server-side user space.  
>

Yep, I wasn't actually expecting all of this to work. At this point, I'm just trying to survey what you already have done so far and seeing where I can pitch in. It seems like Bull had a (rather large) userspace patch at one point. I'll have a look and see if any of it is usable.

Comment 25 Jeff Layton 2009-05-05 20:03:22 UTC
Ahh ok, I see what's happening...

When nfsd is first run on the machine, /proc/fs/nfsd isn't yet mounted. On fedora at least, that happens via a modprobe "install" routine. Once nfsd.ko has been plugged in, the filesystem is present and subsequent runs of rpc.nfsd use the new interfaces. With the new interfaces, you're correct and the program opens sockets and feeds the fd's into the portlist file.

I wonder whether it might be reasonable to have rpc.nfsd call "modprobe nfsd" prior to doing anything? If that's making too many distro-specific assumptions, then we should probably add such a call to the nfs init script for Fedora.

Comment 26 Chuck Lever 2009-05-06 16:14:14 UTC
(In reply to comment #25)
> Ahh ok, I see what's happening...
> 
> When nfsd is first run on the machine, /proc/fs/nfsd isn't yet mounted. On
> fedora at least, that happens via a modprobe "install" routine. Once nfsd.ko
> has been plugged in, the filesystem is present and subsequent runs of rpc.nfsd
> use the new interfaces. With the new interfaces, you're correct and the program
> opens sockets and feeds the fd's into the portlist file.
> 
> I wonder whether it might be reasonable to have rpc.nfsd call "modprobe nfsd"
> prior to doing anything? If that's making too many distro-specific assumptions,
> then we should probably add such a call to the nfs init script for Fedora.  

So, this is one of the main problems we have in this area: our system boot time initialization is order dependent and the dependencies aren't really documented anywhere.  It's very similar to lockd/statd start up ordering.  My preference is to reshape the API so that start up order is not consequential.  Not always possible.

You should probably look for an expedient solution here with rpc.nfsd and consider a complete overhaul as a longer term project, once we have a stronger sense of how this needs to fit together.

Comment 27 Jeff Layton 2009-05-06 20:56:49 UTC
For now I'm going to focus on getting the actual startup of knfsd cleaned up and correct. Once we have that settled then I'll plan to look at mountd and exportfs (and rpc.svcgssd).

It turns out that fedora's init script already modprobes nfsd.ko before starting nfsd. I got confused when playing with it by hand so that I could strace it. So the "legacy" nfsctl() interface is really never used. I think that piece of it is OK as is (so far).

Today, I've been playing some with rpc.nfsd. The patches from Bull pretty much make use of IPv4 in IPv6 mapped addrs everywhere. When the host is IPv6-capable, it by default opens a socket for in6addr_any and sends that info to the kernel.

One immediate problem -- that doesn't make any the kernel register any IPv4 ports. We'll either need to change this or try to pass in separate IPv4 and IPv6 sockets. Doing the former seems lame (what if you really do want an IPv6 only nfs server). I'm not sure if the latter will mean port conflicts. Maybe there's a socket option that can prevent that problem.

Comment 28 Chuck Lever 2009-05-06 21:05:57 UTC
(In reply to comment #27)
> For now I'm going to focus on getting the actual startup of knfsd cleaned up
> and correct. Once we have that settled then I'll plan to look at mountd and
> exportfs (and rpc.svcgssd).

statd is going well... I still plan to do rpc.mountd, and will probably get to it soon.

> It turns out that fedora's init script already modprobes nfsd.ko before
> starting nfsd. I got confused when playing with it by hand so that I could
> strace it. So the "legacy" nfsctl() interface is really never used. I think
> that piece of it is OK as is (so far).
> 
> Today, I've been playing some with rpc.nfsd. The patches from Bull pretty much
> make use of IPv4 in IPv6 mapped addrs everywhere. When the host is
> IPv6-capable, it by default opens a socket for in6addr_any and sends that info
> to the kernel.
> 
> One immediate problem -- that doesn't make any the kernel register any IPv4
> ports. We'll either need to change this or try to pass in separate IPv4 and
> IPv6 sockets. Doing the former seems lame (what if you really do want an IPv6
> only nfs server). I'm not sure if the latter will mean port conflicts. Maybe
> there's a socket option that can prevent that problem. 

TI-RPC uses "IPV6_ONLY" sockets.  The kernel now does too.  So we need to start transports for each address family and each transport protocol separately.

That almost certainly means that mapped addresses are reasonable only for the kernel's svc_auth code, and nowhere else.

Comment 29 Jeff Layton 2009-07-23 11:33:14 UTC
Status update:

We're targeting client side support for NFS over IPv6 for Fedora 12. 

    https://fedoraproject.org/wiki/Features/NFSClientIPv6

...TI-RPC support has already been enabled in F12's nfs-utils. The next step is to turn on IPv6 support.

All of the client side pieces are in place for NFSv4. For NFSv2/3 all of them are in place except for Chuck's rewrite of rpc.statd. I've been testing his rewrite and it seems to work well enough AFAICT. I'd like to see it go upstream soon.

He's on vacation for a couple of weeks, but when he returns we'll have to make a go/no-go decision on NFSv2/3 for F12. If it's not going to be ready in time, it will probably mean a mount.nfs patch of some sort that enables IPv6 support for NFSv4 but not for NFSv2/3.

I'd really like to avoid that however. It means diverging from upstream for one thing. Steve D. will be hard to sell on that point -- he may even NAK it.

Comment 30 IBM Bug Proxy 2009-09-10 04:00:47 UTC
------- Comment From lxie.com 2009-09-09 23:51 EDT-------
Red Hat,

What's the current status on this? will this be in alpha2?

> Status update:
> We're targeting client side support for NFS over IPv6 for Fedora 12.
> https://fedoraproject.org/wiki/Features/NFSClientIPv6
> ...TI-RPC support has already been enabled in F12's nfs-utils. The next step is
> to turn on IPv6 support.
> All of the client side pieces are in place for NFSv4. For NFSv2/3 all of them
> are in place except for Chuck's rewrite of rpc.statd. I've been testing his
> rewrite and it seems to work well enough AFAICT. I'd like to see it go upstream
> soon.
> He's on vacation for a couple of weeks, but when he returns we'll have to make
> a go/no-go decision on NFSv2/3 for F12. If it's not going to be ready in time,
> it will probably mean a mount.nfs patch of some sort that enables IPv6 support
> for NFSv4 but not for NFSv2/3.
> I'd really like to avoid that however. It means diverging from upstream for one
> thing. Steve D. will be hard to sell on that point -- he may even NAK it.

Comment 31 Jeff Layton 2009-09-10 11:42:33 UTC
The statd changes are still stalled upstream. It's doubtful that this will make F12, unfortunately. Hopefully we can come to some resolution there soon and get it into F13.

Comment 32 IBM Bug Proxy 2009-09-15 17:40:41 UTC
------- Comment From ffilz.com 2009-09-15 13:35 EDT-------
It looks like the proposed updated statd will not be accepted upstream an a re-work is necessary. What impact does that have?

Also, the planning document on the wiki:


hasn't been updated since May, is it possible for that to be updated to reflect the current state of things?

Comment 33 Chuck Lever 2009-09-15 18:22:26 UTC
(In reply to comment #32)
> ------- Comment From ffilz.com 2009-09-15 13:35 EDT-------
> It looks like the proposed updated statd will not be accepted upstream and a
> re-work is necessary. What impact does that have?

F13 is still the target, in my opinion, but it will depend on upstream review of the re-work.  We still don't have multi-homed NLM support upstream or in prototype, which IETF is now considering as a requirement for full IPv6 support for NFSv2/v3.

> Also, the planning document on the wiki:
> 
> hasn't been updated since May, is it possible for that to be updated to reflect
> the current state of things?  

I made some adjustments.

Comment 34 Jeff Layton 2009-09-17 15:33:55 UTC
*** Bug 467711 has been marked as a duplicate of this bug. ***

Comment 39 IBM Bug Proxy 2009-11-17 21:30:45 UTC
Frank could you post the status from your eSDT presentation into this bug to make it clear Jeff Layton since he wasn't on the call?

Thanks, Emily

Comment 40 Stephanie Glass 2009-11-18 13:49:25 UTC
Testing mirroring to new bug on IBM side

Comment 41 IBM Bug Proxy 2009-11-18 14:01:02 UTC
------- Comment From sglass.com 2009-11-18 08:58 EDT-------
Please ignore

Testing this comment gets over to Red Hat bug

Comment 42 Stephanie Glass 2009-11-18 15:02:27 UTC
2nd test, please ignore...

Want to see if this makes it to LTC bug

Comment 43 Chris Ward 2009-11-25 10:07:22 UTC
@Reporters

We need to confirm that there is third-party commitment to 
test for the resolution of this request during the RHEL 6.0 
Beta Test Phase before we can approve it for acceptance 
into the release.

In order to avoid any unnecessary delays, please post a 
confirmation as soon as possible, including the contact 
information for testing engineers.

Any additional information about alternative testing variations we 
could use to reproduce this issue in-house would be appreciated.

Comment 44 John Jarvis 2009-11-25 14:26:34 UTC
IBM is signed up to test and provided feedback.

Comment 48 IBM Bug Proxy 2010-02-06 00:41:34 UTC
------- Comment From ffilz.com 2010-02-05 19:38 EDT-------
Most recent status from Chuck Lever:

On Dec 9, 2009, at 11:13 AM, Frank S Filz wrote:
> What is the latest state of IPv6 support for NFS?

I assume you are interested in upstream, and not a specific
distribution.  As always, you can look at:

http://wiki.linux-nfs.org/wiki/index.php/Ipv6PlanningDocument

Or search the archives of linux-nfs.org.

Steve and I have agreed on a plan to integrate my IPv6 patches into
statd and sm-notify over time.

"proto=netid" support is going into 2.6.33, and mount.nfs changes to
support this were submitted this week.

We're expecting to have F13 alpha (or equivalent) ready for a full
client-side test at Connectathon 2010 in February.  This will give us,
among other things, an indication of what client related pieces are
still not ready for distributions.  My guess is we will find details,
but nothing significant.

We are waiting for wider adoption by distributions of rpcbind and
libtirpc.  There seems to be reticence.  I'm exploring a few options
to make the transition easier.

I have a mountd/exportfs prototype that supports IPv6 available in my
git repo on linux-nfs.org.  This includes support for IPv6 in nfs-
utils' tcp-wrapper shim.  I've already received at least one bug
report, suggesting that people are trying it, and it is mostly
working.  It still needs a lot of clean up.  In the meantime we are
going to start using it to test the remaining few kernel patches that
are required for IPv6 server-side support.

I still need to look at multi-homed support for NLM, which will be
needed for systems that have both IPv4 and IPv6 addresses.  We're
still trying to understand what full support for link-local and
deprecated site-local addresses will mean, if it is possible.  Native
IPv6 still seems a ways off, given the state of the rest of the Linux
network stack.  Support for IPv6 for NFS migration/replication should
be explored, but I'm guessing that is mostly ready.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Comment 49 IBM Bug Proxy 2010-05-28 16:49:29 UTC
------- Comment From ffilz.com 2010-05-28 12:35 EDT-------
This feature has been verified. Only NFS v4 was tested since we didn't have a IPv6 capable NFS v3 server.

Comment 50 Chris Ward 2010-07-12 12:36:00 UTC
Red Hat Enterprise Linux Beta 2 is now available and should resolve
the problem described in this bug report. One or more confirmation reports have already been received confirming this. This report is therefore being closed with a resolution of CURRENTRELEASE. You may reopen this bug report if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.