Bug 754552 - Unable to mount nfs4 krb5p shares exported by a fedora16 server
Unable to mount nfs4 krb5p shares exported by a fedora16 server
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: nfs-utils (Show other bugs)
16
Unspecified Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: Steve Dickson
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-16 14:56 EST by Thomas Sailer
Modified: 2012-03-22 15:32 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-22 15:32:31 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
packet dump (6.94 KB, application/octet-stream)
2011-11-16 15:32 EST, Thomas Sailer
no flags Details
strace of rpc.svcgssd (113.91 KB, application/octet-stream)
2011-11-16 16:51 EST, Thomas Sailer
no flags Details

  None (edit)
Description Thomas Sailer 2011-11-16 14:56:40 EST
Description of problem:
After upgrading FreeIPAv1/FC14 to FreeIPAv2/FC16, I am unable to mount krb5p exported home directories. Clients are FC16.

Version-Release number of selected component (if applicable):
nfs-utils-1.2.5-1.fc16.x86_64
nfs-utils-1.2.5-3.fc16.x86_64

How reproducible:
always

Steps to Reproduce:
1.mount -t nfs4 -o soft,intr,rsize=8192,wsize=8192,rw,sec=krb5p server.xxxxx.com:/yyyyy z

  
Actual results:
mount.nfs4: access denied by server while mounting server.xxxxx.com:/yyyyy


Expected results:
mounting succeeds

Additional info:

On the client, rpc.gssd reports:
Warning: rpcsec_gss library does not support setting debug level
beginning poll
dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00
dir_notify_handler: sig 37 si 0x7fff5f5f1570 data 0x7fff5f5f1440
dir_notify_handler: sig 37 si 0x7fff5f5f0df0 data 0x7fff5f5f0cc0
handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt3a)
handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt3a)
process_krb5_upcall: service is '<null>'
Full hostname for 'server.xxxxx.com' is 'server.xxxxx.com'
Full hostname for 'client.xxxxx.com' is 'client.xxxxx.com'
No key table entry found for CLIENT.XXXXX.COM$@XXXXX.COM while getting keytab entry for 'CLIENT.XXXXX.COM$@XXXXX.COM'
No key table entry found for root/client.xxxxx.com@XXXXX.COM while getting keytab entry for 'root/client.xxxxx.com@XXXXX.COM'
Success getting keytab entry for 'nfs/client.xxxxx.com@XXXXX.COM'
Successfully obtained machine credentials for principal 'nfs/client.xxxxx.com@XXXXX.COM' stored in ccache 'FILE:/tmp/krb5cc_machine_XXXXX.COM'
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXXX.COM' are good until 1321556514
using FILE:/tmp/krb5cc_machine_XXXXX.COM as credentials cache for machine creds
using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_XXXXX.COM
creating context using fsuid 0 (save_uid 0)
creating tcp client for server server.xxxxx.com
DEBUG: port already set to 2049
creating context with server nfs@server.xxxxx.com
WARNING: Failed to create krb5 context for user with uid 0 for server server.xxxxx.com
WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_XXXXX.COM for server server.xxxxx.com
WARNING: Machine cache is prematurely expired or corrupted trying to recreate cache for server server.xxxxx.com
Full hostname for 'server.xxxxx.com' is 'server.xxxxx.com'
Full hostname for 'client.xxxxx.com' is 'client.xxxxx.com'
No key table entry found for CLIENT.XXXXX.COM$@XXXXX.COM while getting keytab entry for 'CLIENT.XXXXX.COM$@XXXXX.COM'
No key table entry found for root/client.xxxxx.com@XXXXX.COM while getting keytab entry for 'root/client.xxxxx.com@XXXXX.COM'
Success getting keytab entry for 'nfs/client.xxxxx.com@XXXXX.COM'
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXXX.COM' are good until 1321556514
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXXX.COM' are good until 1321556514
using FILE:/tmp/krb5cc_machine_XXXXX.COM as credentials cache for machine creds
using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_XXXXX.COM
creating context using fsuid 0 (save_uid 0)
creating tcp client for server server.xxxxx.com
DEBUG: port already set to 2049
creating context with server nfs@server.xxxxx.com
WARNING: Failed to create krb5 context for user with uid 0 for server server.xxxxx.com
WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_XXXXX.COM for server server.xxxxx.com
WARNING: Failed to create machine krb5 context with any credentials cache for server server.xxxxx.com
doing error downcall
dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00
dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00
dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00
dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00
dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00
dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00
dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt3b
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt3a

And on the server, rpc.svcgssd reports:
leaving poll
handling null request
svcgssd_limit_krb5_enctypes: Calling gss_set_allowable_enctypes with 7 enctypes from defaults
sname = nfs/client.xxxxx.com@XXXXX.COM
DEBUG: serialize_krb5_ctx: lucid version!
prepare_krb5_rfc4121_buffer: protocol 1
prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32
doing downcall
mech: krb5, hndl len: 4, ctx len 52, timeout: 1321556514 (86400 from now), clnt: nfs@client.xxxxx.com, uid: -1, gid: -1, num aux grps: 0:
sending null reply
writing message: \x \x6082....\x6081....
entering poll
leaving poll
handling null request
svcgssd_limit_krb5_enctypes: Calling gss_set_allowable_enctypes with 7 enctypes from defaults
sname = nfs/client.xxxxx.com@XXXXX.COM
DEBUG: serialize_krb5_ctx: lucid version!
prepare_krb5_rfc4121_buffer: protocol 1
prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32
doing downcall
mech: krb5, hndl len: 4, ctx len 52, timeout: 1321556514 (86400 from now), clnt: nfs@client.xxxxx.com, uid: -1, gid: -1, num aux grps: 0:
sending null reply
writing message: \x \x6082.... 1321470174 0 0 \x02000000 \x6081....
finished handling null request
entering poll

Enctypes available in the key table are aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, des3-cbc-sha1 and arcfour-hmac.

I also tried enabling weak crypto, and to use only des-cbc-crc keys, without success.
Comment 1 J. Bruce Fields 2011-11-16 15:14:00 EST
I assume the same happens with sec=krb5?

How do you see the init_sec context exchange end on the wire?  (A packet trace--tcpdump -s0 -wtmp.pcap, try the mount, then attach tmp.pcap--might help.)
Comment 2 Thomas Sailer 2011-11-16 15:31:34 EST
Yes, the same happens with sec=krb5.
Comment 3 Thomas Sailer 2011-11-16 15:32:04 EST
Created attachment 534080 [details]
packet dump
Comment 4 Steve Dickson 2011-11-16 15:51:43 EST
> No key table entry found for CLIENT.XXXXX.COM$@XXXXX.COM while getting keytab
I thinking this is the problem... gssd is not find a key tab entry....
Comment 5 J. Bruce Fields 2011-11-16 16:18:03 EST
(In reply to comment #4)
> > No key table entry found for CLIENT.XXXXX.COM$@XXXXX.COM while getting keytab
> I thinking this is the problem... gssd is not find a key tab entry....

Well, but a couple lines down there's:

  keytab entry for 'root/client.xxxxx.com@XXXXX.COM'
  Success getting keytab entry for 'nfs/client.xxxxx.com@XXXXX.COM'

So I think it was just searching for one and it had to search a little farther.

Looking at trace.... server is returning GSS_S_NO_CONTEXT error, along with a perfectly good context.  Maybe a mismatch between what svcgssd thinks is supported and what the kernel can do?

Is svcgssd's downcall to give the context to the kernel failing?  You could find that out by strace'ing svcgssd.
Comment 6 Thomas Sailer 2011-11-16 16:51:02 EST
Created attachment 534103 [details]
strace of rpc.svcgssd

you are correct, the write to /proc/net/rpc/auth.rpcsec.context/channel fails with EOPNOTSUPP
Comment 7 Thomas Sailer 2011-11-16 17:15:56 EST
Seems like rpcsec_gss_krb5 does not get autoloaded. modprobe rpcsec_gss_krb5 makes the mount work.
Comment 8 J. Bruce Fields 2011-11-16 17:58:31 EST
OK, that will be fixed in the kernel by upstream 058c5c99999609e3de7e15b49049665f02d06577 "rpc: allow autoloading of gss mechanisms".

In the mean time, we should be loading rpcsec_gss_krb5 by hand before starting rpc.svcgssd.  It looks like that got lost in the migration to systemd.  Steved?

rpc.svcgssd's handling of errors is also a little odd: in the null init_sec_context response it's returning an error correctly, but also returning a context, which seems fairly pointless.
Comment 9 Thomas Sailer 2011-11-16 18:21:52 EST
Something went wrong with the update (preupgrade seems to consistently fail to upgrade the bootloader), so I'm apparently still using the old kernel. That might explain why the autoload failed, as your patch seems to be present in fc16 kernels.
Comment 10 J. Bruce Fields 2011-11-16 18:28:26 EST
Oh, OK, that explains it.  I think we should probably still do a manual modprobe of rpcsec_gss_krb5 for another fedora cycle or so, just in case--people might want to try old kernels for debugging purposes, etc.
Comment 11 Thomas Sailer 2011-11-16 18:35:43 EST
So this is actually a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=737731
Comment 12 Thomas Sailer 2011-11-16 18:36:53 EST
But yes, modprobing rpcsec_gss_krb5 for another release seems like a good idea to me
Comment 13 Steve Dickson 2011-11-17 07:29:31 EST
(In reply to comment #8)
> OK, that will be fixed in the kernel by upstream
> 058c5c99999609e3de7e15b49049665f02d06577 "rpc: allow autoloading of gss
> mechanisms".
> 
> In the mean time, we should be loading rpcsec_gss_krb5 by hand before starting
> rpc.svcgssd.  It looks like that got lost in the migration to systemd.  Steved?
> 
I'll look into seeing if I can get the systemd script to do the modprob for the short term...
Comment 14 Steve Dickson 2011-11-18 10:22:09 EST
(In reply to comment #8)
> OK, that will be fixed in the kernel by upstream
> 058c5c99999609e3de7e15b49049665f02d06577 "rpc: allow autoloading of gss
> mechanisms".
This fix is in the kernel-3.1.1-1.fc16 kernel which is the latest 
F16 kernel. So at this point all that's needed is a yum upgrade kernel

Note You need to log in before you can comment on or make changes to this bug.