22 is -EINVAL.  A network trace might be helpful (tcpdump -s0 -wtmp.pcap, then try the mount, then kill tcpdump and attach the tmp.pcap to this bug and/or take a look at it in wireshark).

"In my office, I have a NFS server with shared files. With Fedora 21 I don't have any problems to establish a connection with it and to mount the directory.

After fedup from 21 to 22, and any change in configuration, I can't do that."

I'm having trouble sorting out client and server here: is it the NFS server you're upgrading, or the NFS client?  If it's the client, what is your NFS server?  (Fedora, or something else?)

> 22 is -EINVAL.  A network trace might be helpful (tcpdump -s0 -wtmp.pcap, then try the mount, then kill tcpdump and attach the tmp.pcap to this bug and/or take a look at it in wireshark).

Ok, I will test tomorrow.

> I'm having trouble sorting out client and server here: is it the NFS server you're upgrading, or the NFS client?  If it's the client, what is your NFS server?  (Fedora, or something else?)

It's my workstation only, the client side. The server is based on Debian (probably previous release stable, I'm not sure).

Apparently, I am not alone : http://forums.fedoraforum.org/showthread.php?p=1734382

Created attachment 1035052 [details]
TCPDump result after mount -a

(In reply to Couret Charles-Antoine from comment #4)
> Created attachment 1035052 [details]
> TCPDump result after mount -a

V4 NULL Call (Reply In 222)
V4 NULL Reply (Call In 220)
V4 Call (Reply In 225) EXCHANGE_ID
V4 Reply (Call In 224) Status: NFS4ERR_MINOR_VERS_MISMATCH
V4 NULL Call (Reply In 234)
V4 NULL Reply (Call In 232)
V4 Call (Reply In 237) EXCHANGE_ID
V4 Reply (Call In 236) EXCHANGE_ID Status: NFS4ERR_INVAL

I wonder if the VERS_MISMATCH is causing the INVAL?

(In reply to Steve Dickson from comment #5)
> (In reply to Couret Charles-Antoine from comment #4)
> > Created attachment 1035052 [details]
> > TCPDump result after mount -a
> 
> V4 NULL Call (Reply In 222)
> V4 NULL Reply (Call In 220)
> V4 Call (Reply In 225) EXCHANGE_ID
> V4 Reply (Call In 224) Status: NFS4ERR_MINOR_VERS_MISMATCH
> V4 NULL Call (Reply In 234)
> V4 NULL Reply (Call In 232)
> V4 Call (Reply In 237) EXCHANGE_ID
> V4 Reply (Call In 236) EXCHANGE_ID Status: NFS4ERR_INVAL
> 
> I wonder if the VERS_MISMATCH is causing the INVAL?

Note the first EXCHANGE_ID is with minorversion 2, and the second with minorversion 1.  The EXCHANGE_ID's otherwise look the same.

Almost certainly a server bug (is this a Solaris server?), probably a dup of bug 1226387.  Please report to the server vendor and see 1226387 for workarounds (basically, mount with vers=4.0 to broken around the server's broken version negotiation).

I see the same problem here, 

# mount -v myserver:/some/dir /mnt/
mount.nfs: mount(2): Protocol not supported
mount.nfs: mount(2): Input/output error
mount.nfs: mount system call failed
mount.nfs: timeout set for Wed Jun 17 16:15:44 2015
mount.nfs: trying text-based options 'vers=4.2,addr=10.0.0.189,clientaddr=10.0.0.143'
mount.nfs: trying text-based options 'vers=4.1,addr=10.0.0.189,clientaddr=10.0.0.143'


Works with this in /etc/nfsmount.conf ():

# Workaround for nfs mount negotiation stopping at v4.1
[ Server "myserver" ]
Defaultvers=4

BTW, my server that gives problems is an old Fedora :-(
This is my blind and stupid fix to nfs-utils:

--- utils/mount/stropts.c~	2015-06-17 16:28:16.413539978 +0200
+++ utils/mount/stropts.c	2015-06-17 16:45:22.446536335 +0200
@@ -838,6 +838,7 @@
 		return result;
 
 	switch (errno) {
+	case EIO:
 	case EPROTONOSUPPORT:
 		/* A clear indication that the server or our
 		 * client does not support NFS version 4 and minor */

(In reply to Anders Blomdell from comment #8)
> BTW, my server that gives problems is an old Fedora :-(
hmm... this is getting a bit worrisome.. 
 
> This is my blind and stupid fix to nfs-utils:
> 
> --- utils/mount/stropts.c~	2015-06-17 16:28:16.413539978 +0200
> +++ utils/mount/stropts.c	2015-06-17 16:45:22.446536335 +0200
> @@ -838,6 +838,7 @@
>  		return result;
>  
>  	switch (errno) {
> +	case EIO:
>  	case EPROTONOSUPPORT:
>  		/* A clear indication that the server or our
>  		 * client does not support NFS version 4 and minor */

Lets use this to start the upstream discussion. Would you mind
posting this to linux-nfs.org using the patch
guidelines under 
   https://www.kernel.org/doc/Documentation/SubmittingPatches

I have no idea where or if it will go, but this new "feature"
is definitely exercising untested code in legacy servers.

(In reply to Anders Blomdell from comment #8)
> BTW, my server that gives problems is an old Fedora :-(

Ugh.  What kernel version is that Fedora server running?

(In reply to J. Bruce Fields from comment #10)
> (In reply to Anders Blomdell from comment #8)
> > BTW, my server that gives problems is an old Fedora :-(
> 
> Ugh.  What kernel version is that Fedora server running?
2.6.35.14-106.fc14.i686.PAE

(In reply to Steve Dickson from comment #9)
> Lets use this to start the upstream discussion. Would you mind
> posting this to linux-nfs.org using the patch
> guidelines under 
>    https://www.kernel.org/doc/Documentation/SubmittingPatches
> 
> I have no idea where or if it will go, but this new "feature"
> is definitely exercising untested code in legacy servers.
Hopefully done.

Created attachment 1040329 [details]
Patch submitted to upstream

(In reply to Anders Blomdell from comment #13)
> Created attachment 1040329 [details]
> Patch submitted to upstream

Thank you!

On 2015-06-18 14:53, Trond Myklebust wrote:> On Thu, Jun 18, 2015 at 8:28 AM, Anders Blomdell
> <anders.blomdell.se> wrote:
>> On 2015-06-18 13:49, Trond Myklebust wrote:
>>> On Thu, Jun 18, 2015 at 4:04 AM, Anders Blomdell
>>> <anders.blomdell.se> wrote:
>>>>
>>>> I have a problem with a 4.0.4 client refusing to mount from a 2.6.35 server
>>>> due to NFS4ERR_INVAL returned during nfs4_discover_server_trunking. See
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1228272.
>>>
>>>
>>> Why should we change the clients if the server is in clear and obvious
>>> violation of the spec?
>> In order to make clients work with servers that worked well with previous versions
>> of nfs-utils, the cultprit probably being commit f9802988 that bumped the default
>> autonegotion version to 4.2, what the patch does is only to negotiate a lower version
>> in case of errors, and hence making 1.3.2 working with servers that worked with
>> 1.3.1 (that only tried version 4[.0]).
>>
>> Will probably save some people some time.
> 
> This is what /etc/nfsmount.conf is for. We don't fix clients that are
> working correctly according to the protocol spec.
> 
> Trond

So this is obviously NOTABUG (sadly enough)

(In reply to Anders Blomdell from comment #11)
> (In reply to J. Bruce Fields from comment #10)
> > (In reply to Anders Blomdell from comment #8)
> > > BTW, my server that gives problems is an old Fedora :-(
> > 
> > Ugh.  What kernel version is that Fedora server running?
> 2.6.35.14-106.fc14.i686.PAE

I took a look at the code for that kernel and can't see an obvious explanation for the difference.  Judging from the network trace it's correctly handling the case of minorversions that are totally unsupported by the code (minorversion >=2), but not a minorversion that's supported by the code by currently runtime-disabled (minorversion=1).  There's probably some obvious logic error I'm not seeing there....  In any case the bug's no longer present upstream so I'm not inclined to investigate further for the sake of an old Fedora version, but if someone wants to it's probably something easy to fix.

Anyway, the fact that this affects both knfsd and Solaris servers leans me a bit more towards doing a workaround, but I'm also sympathetic to Trond's point of view that we shouldn't accumulate workarounds.

(In reply to Anders Blomdell from comment #11)
> (In reply to J. Bruce Fields from comment #10)
> > (In reply to Anders Blomdell from comment #8)
> > > BTW, my server that gives problems is an old Fedora :-(
> > 
> > Ugh.  What kernel version is that Fedora server running?
> 2.6.35.14-106.fc14.i686.PAE

I have two (2) old Fedora servers both running   2.6.35.14-106.fc14.i686   and feeding two (2) clients one running Fedora 21 & one running Fedora 22. Fedora 21 mounts NFS files fine, Fedora 22 does not mount NFS files. 

Fedora 22 fails to mount {4.0.5-300.fc22.x86_64}
mount.nfs: timeout set for Thu Jun 25 18:16:58 2015
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.12.102,clientaddr=192.168.12.104'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'vers=4.1,addr=192.168.12.102,clientaddr=192.168.12.104'
mount.nfs: mount(2): Input/output error
mount.nfs: mount system call failed

Fedora 21 mounts fine {4.0.5-200.fc21.x86_64}

Looks very nfsd kernel version specific.. I can probably reproduce that.

Jun 26 06:55:53 olfedorapants kernel: [  220.354879] nfsd_dispatch: vers 4 proc 1
Jun 26 06:55:53 olfedorapants kernel: [  220.354880] nfsv4 compound op #1/1: 42 (OP_EXCHANGE_ID)
Jun 26 06:55:53 olfedorapants kernel: [  220.354882] nfsd4_exchange_id rqstp=ffff880071860000 exid=ffff88007c05a080 clname.len=33 clname.data=ffff88007184b080 ip_addr=10.0.1.82 flags 101, spa_how 0
Jun 26 06:55:53 olfedorapants kernel: [  220.354884] nfsv4 compound op ffff88007c05a078 opcnt 1 #1: 42: status 22
Jun 26 06:55:53 olfedorapants kernel: [  220.354885] nfsv4 compound returned 22

A lack of the return code dprintk for nfsd4_exchange_id makes it seem like we're doing an early return, and there's only one place that can happen - in the exchange_id name/flags check:
fs/nfsd/nfs4state.c
1123 __be32
1124 nfsd4_exchange_id(struct svc_rqst *rqstp,
1125           struct nfsd4_compound_state *cstate,
1126           struct nfsd4_exchange_id *exid)
...
1142     if (!check_name(exid->clname) || (exid->flags & ~EXCHGID4_FLAG_MASK_A))
1143         return nfserr_inval;
1144

and in this knfsd kernel:
#define EXCHGID4_FLAG_MASK_A                   0x40070003

And my flags are (client is 4.0.5-200.fc21 w/ upstream nfs-utils):
flags: 0x00000101
    0... .... .... .... .... .... .... .... = EXCHGID4_FLAG_CONFIRMED_R: Not set
    .0.. .... .... .... .... .... .... .... = EXCHGID4_FLAG_UPD_CONFIRMED_REC_A: Not set
    .... .... .... .0.. .... .... .... .... = EXCHGID4_FLAG_USE_PNFS_DS: Not set
    .... .... .... ..0. .... .... .... .... = EXCHGID4_FLAG_USE_PNFS_MDS: Not set
    .... .... .... ...0 .... .... .... .... = EXCHGID4_FLAG_USE_NON_PNFS: Not set
    .... .... .... .... .... ...1 .... .... = EXCHGID4_FLAG_BIND_PRINC_STATEID: Set
    .... .... .... .... .... .... .... ..0. = EXCHGID4_FLAG_SUPP_MOVED_MIGR: Not set
    .... .... .... .... .... .... .... ...1 = EXCHGID4_FLAG_SUPP_MOVED_REFER: Set

The NFS client started setting EXCHGID4_FLAG_BIND_PRINC_STATEID here:
4f0b429 NFSv4.1: Enable state protection
with the earliest tag in v3.11

The NFS server updated EXCHGID4_FLAG_MASK_A to 0x40070103 here:
357f54d NFS fix the setting of exchange id flag
with the earliest tag in v2.6.38

So clients later than 2.6.38 mounting nfsv4.1 on servers older than 2.6.38 will run into this problem.

(In reply to Benjamin Coddington from comment #18)
> So clients later than 2.6.38 mounting nfsv4.1 on servers older than 2.6.38
> will run into this problem.

Wrong! this should have been:

Clients later than 3.11 mounting nfsv4.1 on servers older than 2.6.38 will run into this problem.

Fixes?

Make the client's stateid/cred binding optional/tunable?
Make mount.nfs downgrade, retry on this particular error?

The workaround should be to disable v4.1 on the old knfsd servers, and the nfs clients will properly, gracefully negotiate down to v4.0 as before.

(In reply to Benjamin Coddington from comment #20)
> Fixes?
> 
> Make the client's stateid/cred binding optional/tunable?
> Make mount.nfs downgrade, retry on this particular error?
The error would be EIO.... Hmm... The change would not go upstream
but it might make Fedora users bit happier.... which would be a good thing.. 

> 
> The workaround should be to disable v4.1 on the old knfsd servers, and the
> nfs clients will properly, gracefully negotiate down to v4.0 as before.
I'm assuming most people don't like to touch old servers because
they just work...

(In reply to Benjamin Coddington from comment #20)
> The workaround should be to disable v4.1 on the old knfsd servers, and the
> nfs clients will properly, gracefully negotiate down to v4.0 as before.

The kernel was defaulting 4.1 to off until 3.11 (and for reasons, I doubt this is the only case where an old 4.1 server could break on client upgrade).

Was something in nfs-utils overriding that, or are the bug reports all from folks who explicitly turned on 4.1 on their server?

Or is the older minorversion-checking logic somehow letting us get through to the exchange_id code even when 4.1 support is off?  (What's the content of /proc/fs/versions?)

(In reply to J. Bruce Fields from comment #22)
> (In reply to Benjamin Coddington from comment #20)
> > The workaround should be to disable v4.1 on the old knfsd servers, and the
> > nfs clients will properly, gracefully negotiate down to v4.0 as before.
> 
> The kernel was defaulting 4.1 to off until 3.11 (and for reasons, I doubt
> this is the only case where an old 4.1 server could break on client upgrade).
> 
> Was something in nfs-utils overriding that, or are the bug reports all from
> folks who explicitly turned on 4.1 on their server?
> 
> Or is the older minorversion-checking logic somehow letting us get through
> to the exchange_id code even when 4.1 support is off?  (What's the content
> of /proc/fs/versions?)

I don't think so.  When I disabled 4.1 the mount would fall down to 4.0 properly.  These must be reports from people who have enabled 4.1.

(In reply to Benjamin Coddington from comment #23)
> I don't think so.  When I disabled 4.1 the mount would fall down to 4.0
> properly.  These must be reports from people who have enabled 4.1.

OK, so it sounds like 1) these servers are running long-unsupported Fedora versions, and 2) their administrators turned on experimental code.

I have some sympathy, but we can't fix the old server now, and I don't think it's worth a workaround; closing as NOTABUG.

Thanks to everyone who helped investigate.

To help document the workarounds:

- if using a Fedora server: it should be enough just to turn off 4.1 on the server (probably just a matter of fixing the RPCNFSDARGS line in /etc/sysconfig/nfs).

- if using another server: mount with -overs=4.0, or use nfsmount.conf to set the default version for that server to 4.0 (see "man nfsmount.conf")

(In reply to J. Bruce Fields from comment #24)
> (In reply to Benjamin Coddington from comment #23)
> > I don't think so.  When I disabled 4.1 the mount would fall down to 4.0
> > properly.  These must be reports from people who have enabled 4.1.
> 
> OK, so it sounds like 1) these servers are running long-unsupported Fedora
> versions, and 2) their administrators turned on experimental code.
Assumption 2 is unfortunately false :-( (unless running Fedora is considered turning on experimental code ;-))

  # uname -a 
  Linux sperry-01 2.6.35.14-106.fc14.i686.PAE #1 SMP Wed Nov 23 13:39:51 UTC 2011 i686 i686 i386 GNU/Linux
  # rpm -q -f /boot/config-$(uname -r) 
  kernel-PAE-2.6.35.14-106.fc14.i686
  # grep  'CONFIG_NFS_V4_1=y' /boot/config-$(uname -r)
  CONFIG_NFS_V4_1=y


> I have some sympathy, but we can't fix the old server now, and I don't think
> it's worth a workaround; closing as NOTABUG.
OK, duly noted...

> 
> Thanks to everyone who helped investigate.
> 
> To help document the workarounds:
> 
> - if using a Fedora server: it should be enough just to turn off 4.1 on the
> server (probably just a matter of fixing the RPCNFSDARGS line in
> /etc/sysconfig/nfs).
RPCNFSDARGS+="-N 4.1"

> 
> - if using another server: mount with -overs=4.0, or use nfsmount.conf to
> set the default version for that server to 4.0 (see "man nfsmount.conf")

(In reply to Anders Blomdell from comment #25)
> 
> > I have some sympathy, but we can't fix the old server now, and I don't think
> > it's worth a workaround; closing as NOTABUG.
> OK, duly noted...
> 
> > 
> > Thanks to everyone who helped investigate.
> > 
> > To help document the workarounds:
> > 
> > - if using a Fedora server: it should be enough just to turn off 4.1 on the
> > server (probably just a matter of fixing the RPCNFSDARGS line in
> > /etc/sysconfig/nfs).
> RPCNFSDARGS+="-N 4.1"
> 
> > 
> > - if using another server: mount with -overs=4.0, or use nfsmount.conf to
> > set the default version for that server to 4.0 (see "man nfsmount.conf")

RPCNFSDARGS+="-N 4.1" WORKS VERY WELL on both FC21 & FC22 machines.
Thank You