Bug 1250632

10082 is NFS4ERR_WRONG_CRED.

Most useful if possible would be a network trace of the failed mount, so start something like:

  tcpdump -wtmp.pcap -iem1 -f'host myserver'

then try the mount, then kill tcpdump and attach tmp.pcap (and/or take a look at it in wireshark; what we'd like to see is which rpc got the WRONG_CRED error and what the arguments of that call looked like).

Hm, I wonder if the change to attempt krb5 first on state establishment went in somewhere between f21 and f22?  In which case maybe some sort of incomplete krb5 setup might be to blame.

Created attachment 1059578 [details]
network trace of failed krb5 mount

Here you go.  Thanks!

Does the problem occur if the client tries to mount with vers=4.1?  I see from the capture that an initial mount attempt w/ 4.2 is made and refused, then the client tries again with 4.1..

May be something about the server recognizing the previous credential when there's a minor version mis-match?

The problem still occurs with vers=4.1 added to the mount string.

When I tried vers=4.0 though, the mount succeeded.  Thanks!  At least now I have a work-around!

I have also run into this "error 10082" on Fedora 23 and the vers=4.0 option allowed successful mount. Thanks @Gregory Lee Bartholomew for reporting this.


NFS Server:-
Ubuntu 14.04
Kernel 3.13.0-68-generic x86_64
nfs-common        1:1.2.8-6ubuntu1.1
nfs-kernel-server 1:1.2.8-6ubuntu1.1

/etc/exports

/nfsexport/backup 192.168.1.0/24(rw,sec=krb5:krb5i:krb5p,no_subtree_check,async,root_squash)


NFS Client:-
Fedora 23 relatively fresh install
Kernel            4.2.6-300.fc23.x86_64
nfs-utils.x86_64  1:1.3.3-1.rc1.fc23

working mount on client was via:-

sudo mount -t nfs4 -o sec=krb5,vers=4.0 <my-nfs-server>:/nfsexport/backup /mnt -vvv


Without the vers=4.0 option in the mount I get:-

sudo mount -t nfs4 -o sec=krb5 <my-nfs-server>:/nfsexport/backup /mnt -vvv
mount.nfs4: timeout set for Tue Nov 24 13:58:10 2015
mount.nfs4: trying text-based options 'sec=krb5,vers=4.2,addr=192.168.1.200,clientaddr=192.168.1.186'
mount.nfs4: mount(2): Protocol not supported
mount.nfs4: trying text-based options 'sec=krb5,vers=4.1,addr=192.168.1.200,clientaddr=192.168.1.186'

with no command prompt return until I Ctrl+C
Directly after Ctrl+C "journalctl -f" on the client shows:-

NFS: nfs4_discover_server_trunking unhandled error -512. Exiting with error EIO

Just adding this info in case it helps. I'm good with vers=4.0 for now as will be upgrading server to Fedora 23 soon anyway.

Thanks all.

Bug 1283341 also reports a failure that reproduces in 4. in 4.0, and that involves a WRONG_CRED failure.  Might be worth trying those patches (but I haven't looked any more closely than that).

Having looked at the trace--yes, the EXCHANGE_ID is succeeding with MACH_CRED protection set, then the CREATE_SESSION is failing with WRONG_CRED, which is the same behavior we saw in bug 1283341.

How did you set up kerberos, and what does your keytab look like on the client?  (output of keytab -k).

(In reply to J. Bruce Fields from comment #6)
> Bug 1283341 also reports a failure that reproduces in 4. in 4.0

(Sorry, that "4." should have been "4.1 or higher"!)

@J. Bruce Fields
My kerberos is hand configured (no FreeIPA just yet) and I have no DNS records indicating the kerberos server but have managed without this to date.

domain = lan
kerberos realm = LAN

I chose to use a machine principal on the client and only use kerberos for NFS:-

sudo klist -e -k /etc/krb5.keytab
[sudo] password for <username>: 
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 NORMAN-VOSTRO-1720$@LAN (aes256-cts-hmac-sha1-96) 
   3 NORMAN-VOSTRO-1720$@LAN (arcfour-hmac) 
   3 NORMAN-VOSTRO-1720$@LAN (des3-cbc-sha1) 
   3 NORMAN-VOSTRO-1720$@LAN (des-cbc-crc) 

Where "norman-vostro-1720" is the hostname of the client machine and this machine principal was created on the KDC server and then installed on the client as follows:-

On the KDC Server in kadmin as <adminuser>/admin
addprinc -randkey NORMAN-VOSTRO-1720$

On the Client machine "norman-vostro-1720" I retrieve and install the machine principal.
sudo kadmin -p <adminuser>/admin -q "ktadd NORMAN-VOSTRO-1720$"

The client user <username> gets their ticket on the client via pam or kinit.
My understanding of the relevant principals on the KDC via listprincs:-

<username>@LAN
NORMAN-VOSTRO-1720$@LAN
nfs/norman-vostro-1720.lan@LAN
nfs/<kdc-and-nfs-servername>.lan@LAN

The KDC/NFS server has the nfs/<kdc-and-nfs-servername>.lan@LAN principal added to it's keytab file via:-

sudo kadmin -p <adminuser>/admin -q "ktadd nfs/<kdc-and-nfs-servername>.lan"

resulting in the following contents in the KDC/NFS server's keytab file ie:-

sudo klist -e -k /etc/krb5.keytab

   2 nfs/kdc-and-nfs-servername.lan@LAN (aes256-cts-hmac-sha1-96) 
   2 nfs/kdc-and-nfs-servername.lan@LAN (arcfour-hmac) 
   2 nfs/kdc-and-nfs-servername.lan@LAN (des3-cbc-sha1) 
   2 nfs/kdc-and-nfs-servername.lan@LAN (des-cbc-crc) 


Hope that helps with narrowing it down a tad.

Thanks.
Please don't hesitate to request more info if required, I will be blowing this nfs/kdc server away soon though to move to F23 server.

Thanks for the additional details, the client names that lack the "/" are what triggered the issue in bug 1283341 as well, so this is almost certainly the same problem.  If you want to roll your own kernel you can apply the patches described there.  This isn't fixed in Fedora 23 yet either, I'm not sure of the process to apply those patches.  They should be in the (not-yet-released) 4.5 kernel.  Meanwhile the workaround is to stick to 4.0, or use names on the clients like "nfs/hostname@REALM".

@J. Bruce Fields That's great thanks. Is there any chance of getting a public link to bug 1283341 as the link supplied is restricted access. Thanks.

(In reply to Philip Guyton from comment #11)
> @J. Bruce Fields That's great thanks. Is there any chance of getting a
> public link to bug 1283341 as the link supplied is restricted access. Thanks.

Whoops, I'll see if I can fix that.  Meanwhile, you can get those patches from the followups to:

  http://lkml.kernel.org/r/1448385497-23737-1-git-send-email-bfields@redhat.com

(Also included in the for-4.5 branch at git://linux-nfs.org/~bfields/linux.git)

@J. Bruce Fields Thanks for all the info and your help with this and for sorting out the public access on the referenced bug. Much appreciated.

Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.