Bug 1250632 - lease expired failed on NFSv4 server ... with error code 10082
lease expired failed on NFSv4 server ... with error code 10082
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: nfs-utils (Show other bugs)
22
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Steve Dickson
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-05 11:35 EDT by Gregory Lee Bartholomew
Modified: 2016-07-19 13:22 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-19 13:22:03 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
network trace of failed krb5 mount (10.30 KB, application/octet-stream)
2015-08-05 13:38 EDT, Gregory Lee Bartholomew
no flags Details

  None (edit)
Description Gregory Lee Bartholomew 2015-08-05 11:35:23 EDT
Description of problem:

Upon upgrade of my workstation to Fedora 22, NFS mounts cease to work with the error:

lease expired failed on NFSv4 server <my-nfs-server> with error code 10082

showing in /var/log/messages.

Other workstations running Fedora versions 19, 20 or 21 continue to work without issue.

The server is running:

kernel-3.14.27-100.fc19.x86_64
nfs-utils-1.2.8-6.3.fc19.x86_64

The non-working workstation is running:

kernel-4.1.3-200.fc22.x86_64
nfs-utils-1.3.2-9.fc22.x86_64

The server-side export is defined as:

/export -fsid=0,sec=sys:krb5,ro,no_root_squash *
/export/home -fsid=1,mp,rw,async,no_root_squash <my-trusted-clients> *(sec=krb5)

The client in question is attempting to mount with the mount-string:

-fstype=nfs4,sec=krb5,noatime,nodiratime <my-nfs-server>:/home/<my-username>

Version-Release number of selected component (if applicable):

nfs-utils-1.3.2-9.fc22.x86_64

How reproducible:

I re-imaged my workstation back to Fedora 21 and the NFS mount resumed working.

I then re-performed the upgrade using "fedup --network 22" and followed the directions on the fedup wiki page (rpm --rebuilddb, dnf distro-sync, and rpmconf -a) and the problem returned.

Steps to Reproduce:

1. fedup --network 22

Actual results:

"... lease expired failed ..."

Expected results:

successful mount

Additional info:
Comment 1 J. Bruce Fields 2015-08-05 12:34:06 EDT
10082 is NFS4ERR_WRONG_CRED.

Most useful if possible would be a network trace of the failed mount, so start something like:

  tcpdump -wtmp.pcap -iem1 -f'host myserver'

then try the mount, then kill tcpdump and attach tmp.pcap (and/or take a look at it in wireshark; what we'd like to see is which rpc got the WRONG_CRED error and what the arguments of that call looked like).

Hm, I wonder if the change to attempt krb5 first on state establishment went in somewhere between f21 and f22?  In which case maybe some sort of incomplete krb5 setup might be to blame.
Comment 2 Gregory Lee Bartholomew 2015-08-05 13:38:00 EDT
Created attachment 1059578 [details]
network trace of failed krb5 mount

Here you go.  Thanks!
Comment 3 Benjamin Coddington 2015-08-05 14:22:44 EDT
Does the problem occur if the client tries to mount with vers=4.1?  I see from the capture that an initial mount attempt w/ 4.2 is made and refused, then the client tries again with 4.1..

May be something about the server recognizing the previous credential when there's a minor version mis-match?
Comment 4 Gregory Lee Bartholomew 2015-08-05 14:42:29 EDT
The problem still occurs with vers=4.1 added to the mount string.

When I tried vers=4.0 though, the mount succeeded.  Thanks!  At least now I have a work-around!
Comment 5 Philip Guyton 2015-11-24 09:21:11 EST
I have also run into this "error 10082" on Fedora 23 and the vers=4.0 option allowed successful mount. Thanks @Gregory Lee Bartholomew for reporting this.


NFS Server:-
Ubuntu 14.04
Kernel 3.13.0-68-generic x86_64
nfs-common        1:1.2.8-6ubuntu1.1
nfs-kernel-server 1:1.2.8-6ubuntu1.1

/etc/exports

/nfsexport/backup 192.168.1.0/24(rw,sec=krb5:krb5i:krb5p,no_subtree_check,async,root_squash)


NFS Client:-
Fedora 23 relatively fresh install
Kernel            4.2.6-300.fc23.x86_64
nfs-utils.x86_64  1:1.3.3-1.rc1.fc23

working mount on client was via:-

sudo mount -t nfs4 -o sec=krb5,vers=4.0 <my-nfs-server>:/nfsexport/backup /mnt -vvv


Without the vers=4.0 option in the mount I get:-

sudo mount -t nfs4 -o sec=krb5 <my-nfs-server>:/nfsexport/backup /mnt -vvv
mount.nfs4: timeout set for Tue Nov 24 13:58:10 2015
mount.nfs4: trying text-based options 'sec=krb5,vers=4.2,addr=192.168.1.200,clientaddr=192.168.1.186'
mount.nfs4: mount(2): Protocol not supported
mount.nfs4: trying text-based options 'sec=krb5,vers=4.1,addr=192.168.1.200,clientaddr=192.168.1.186'

with no command prompt return until I Ctrl+C
Directly after Ctrl+C "journalctl -f" on the client shows:-

NFS: nfs4_discover_server_trunking unhandled error -512. Exiting with error EIO

Just adding this info in case it helps. I'm good with vers=4.0 for now as will be upgrading server to Fedora 23 soon anyway.

Thanks all.
Comment 6 J. Bruce Fields 2015-11-24 13:23:35 EST
Bug 1283341 also reports a failure that reproduces in 4. in 4.0, and that involves a WRONG_CRED failure.  Might be worth trying those patches (but I haven't looked any more closely than that).
Comment 7 J. Bruce Fields 2015-11-24 13:34:58 EST
Having looked at the trace--yes, the EXCHANGE_ID is succeeding with MACH_CRED protection set, then the CREATE_SESSION is failing with WRONG_CRED, which is the same behavior we saw in bug 1283341.

How did you set up kerberos, and what does your keytab look like on the client?  (output of keytab -k).
Comment 8 J. Bruce Fields 2015-11-24 14:37:20 EST
(In reply to J. Bruce Fields from comment #6)
> Bug 1283341 also reports a failure that reproduces in 4. in 4.0

(Sorry, that "4." should have been "4.1 or higher"!)
Comment 9 Philip Guyton 2015-11-25 10:20:34 EST
@J. Bruce Fields
My kerberos is hand configured (no FreeIPA just yet) and I have no DNS records indicating the kerberos server but have managed without this to date.

domain = lan
kerberos realm = LAN

I chose to use a machine principal on the client and only use kerberos for NFS:-

sudo klist -e -k /etc/krb5.keytab
[sudo] password for <username>: 
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 NORMAN-VOSTRO-1720$@LAN (aes256-cts-hmac-sha1-96) 
   3 NORMAN-VOSTRO-1720$@LAN (arcfour-hmac) 
   3 NORMAN-VOSTRO-1720$@LAN (des3-cbc-sha1) 
   3 NORMAN-VOSTRO-1720$@LAN (des-cbc-crc) 

Where "norman-vostro-1720" is the hostname of the client machine and this machine principal was created on the KDC server and then installed on the client as follows:-

On the KDC Server in kadmin as <adminuser>/admin
addprinc -randkey NORMAN-VOSTRO-1720$

On the Client machine "norman-vostro-1720" I retrieve and install the machine principal.
sudo kadmin -p <adminuser>/admin -q "ktadd NORMAN-VOSTRO-1720$"

The client user <username> gets their ticket on the client via pam or kinit.
My understanding of the relevant principals on the KDC via listprincs:-

<username>@LAN
NORMAN-VOSTRO-1720$@LAN
nfs/norman-vostro-1720.lan@LAN
nfs/<kdc-and-nfs-servername>.lan@LAN

The KDC/NFS server has the nfs/<kdc-and-nfs-servername>.lan@LAN principal added to it's keytab file via:-

sudo kadmin -p <adminuser>/admin -q "ktadd nfs/<kdc-and-nfs-servername>.lan"

resulting in the following contents in the KDC/NFS server's keytab file ie:-

sudo klist -e -k /etc/krb5.keytab

   2 nfs/kdc-and-nfs-servername.lan@LAN (aes256-cts-hmac-sha1-96) 
   2 nfs/kdc-and-nfs-servername.lan@LAN (arcfour-hmac) 
   2 nfs/kdc-and-nfs-servername.lan@LAN (des3-cbc-sha1) 
   2 nfs/kdc-and-nfs-servername.lan@LAN (des-cbc-crc) 


Hope that helps with narrowing it down a tad.

Thanks.
Please don't hesitate to request more info if required, I will be blowing this nfs/kdc server away soon though to move to F23 server.
Comment 10 J. Bruce Fields 2015-11-25 12:10:49 EST
Thanks for the additional details, the client names that lack the "/" are what triggered the issue in bug 1283341 as well, so this is almost certainly the same problem.  If you want to roll your own kernel you can apply the patches described there.  This isn't fixed in Fedora 23 yet either, I'm not sure of the process to apply those patches.  They should be in the (not-yet-released) 4.5 kernel.  Meanwhile the workaround is to stick to 4.0, or use names on the clients like "nfs/hostname@REALM".
Comment 11 Philip Guyton 2015-11-28 11:26:49 EST
@J. Bruce Fields That's great thanks. Is there any chance of getting a public link to bug 1283341 as the link supplied is restricted access. Thanks.
Comment 12 J. Bruce Fields 2015-11-28 16:59:26 EST
(In reply to Philip Guyton from comment #11)
> @J. Bruce Fields That's great thanks. Is there any chance of getting a
> public link to bug 1283341 as the link supplied is restricted access. Thanks.

Whoops, I'll see if I can fix that.  Meanwhile, you can get those patches from the followups to:

  http://lkml.kernel.org/r/1448385497-23737-1-git-send-email-bfields@redhat.com

(Also included in the for-4.5 branch at git://linux-nfs.org/~bfields/linux.git)
Comment 13 Philip Guyton 2015-12-01 15:30:52 EST
@J. Bruce Fields Thanks for all the info and your help with this and for sorting out the public access on the referenced bug. Much appreciated.
Comment 14 Fedora End Of Life 2016-07-19 13:22:03 EDT
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.