Description of problem: After upgrading FreeIPAv1/FC14 to FreeIPAv2/FC16, I am unable to mount krb5p exported home directories. Clients are FC16. Version-Release number of selected component (if applicable): nfs-utils-1.2.5-1.fc16.x86_64 nfs-utils-1.2.5-3.fc16.x86_64 How reproducible: always Steps to Reproduce: 1.mount -t nfs4 -o soft,intr,rsize=8192,wsize=8192,rw,sec=krb5p server.xxxxx.com:/yyyyy z Actual results: mount.nfs4: access denied by server while mounting server.xxxxx.com:/yyyyy Expected results: mounting succeeds Additional info: On the client, rpc.gssd reports: Warning: rpcsec_gss library does not support setting debug level beginning poll dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00 dir_notify_handler: sig 37 si 0x7fff5f5f1570 data 0x7fff5f5f1440 dir_notify_handler: sig 37 si 0x7fff5f5f0df0 data 0x7fff5f5f0cc0 handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt3a) handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt3a) process_krb5_upcall: service is '<null>' Full hostname for 'server.xxxxx.com' is 'server.xxxxx.com' Full hostname for 'client.xxxxx.com' is 'client.xxxxx.com' No key table entry found for CLIENT.XXXXX.COM$@XXXXX.COM while getting keytab entry for 'CLIENT.XXXXX.COM$@XXXXX.COM' No key table entry found for root/client.xxxxx.com while getting keytab entry for 'root/client.xxxxx.com' Success getting keytab entry for 'nfs/client.xxxxx.com' Successfully obtained machine credentials for principal 'nfs/client.xxxxx.com' stored in ccache 'FILE:/tmp/krb5cc_machine_XXXXX.COM' INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXXX.COM' are good until 1321556514 using FILE:/tmp/krb5cc_machine_XXXXX.COM as credentials cache for machine creds using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_XXXXX.COM creating context using fsuid 0 (save_uid 0) creating tcp client for server server.xxxxx.com DEBUG: port already set to 2049 creating context with server nfs.com WARNING: Failed to create krb5 context for user with uid 0 for server server.xxxxx.com WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_XXXXX.COM for server server.xxxxx.com WARNING: Machine cache is prematurely expired or corrupted trying to recreate cache for server server.xxxxx.com Full hostname for 'server.xxxxx.com' is 'server.xxxxx.com' Full hostname for 'client.xxxxx.com' is 'client.xxxxx.com' No key table entry found for CLIENT.XXXXX.COM$@XXXXX.COM while getting keytab entry for 'CLIENT.XXXXX.COM$@XXXXX.COM' No key table entry found for root/client.xxxxx.com while getting keytab entry for 'root/client.xxxxx.com' Success getting keytab entry for 'nfs/client.xxxxx.com' INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXXX.COM' are good until 1321556514 INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXXX.COM' are good until 1321556514 using FILE:/tmp/krb5cc_machine_XXXXX.COM as credentials cache for machine creds using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_XXXXX.COM creating context using fsuid 0 (save_uid 0) creating tcp client for server server.xxxxx.com DEBUG: port already set to 2049 creating context with server nfs.com WARNING: Failed to create krb5 context for user with uid 0 for server server.xxxxx.com WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_XXXXX.COM for server server.xxxxx.com WARNING: Failed to create machine krb5 context with any credentials cache for server server.xxxxx.com doing error downcall dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00 dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00 dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00 dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00 dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00 dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00 dir_notify_handler: sig 37 si 0x7fff5f5f5d30 data 0x7fff5f5f5c00 destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt3b destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt3a And on the server, rpc.svcgssd reports: leaving poll handling null request svcgssd_limit_krb5_enctypes: Calling gss_set_allowable_enctypes with 7 enctypes from defaults sname = nfs/client.xxxxx.com DEBUG: serialize_krb5_ctx: lucid version! prepare_krb5_rfc4121_buffer: protocol 1 prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32 doing downcall mech: krb5, hndl len: 4, ctx len 52, timeout: 1321556514 (86400 from now), clnt: nfs.com, uid: -1, gid: -1, num aux grps: 0: sending null reply writing message: \x \x6082....\x6081.... entering poll leaving poll handling null request svcgssd_limit_krb5_enctypes: Calling gss_set_allowable_enctypes with 7 enctypes from defaults sname = nfs/client.xxxxx.com DEBUG: serialize_krb5_ctx: lucid version! prepare_krb5_rfc4121_buffer: protocol 1 prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32 doing downcall mech: krb5, hndl len: 4, ctx len 52, timeout: 1321556514 (86400 from now), clnt: nfs.com, uid: -1, gid: -1, num aux grps: 0: sending null reply writing message: \x \x6082.... 1321470174 0 0 \x02000000 \x6081.... finished handling null request entering poll Enctypes available in the key table are aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, des3-cbc-sha1 and arcfour-hmac. I also tried enabling weak crypto, and to use only des-cbc-crc keys, without success.
I assume the same happens with sec=krb5? How do you see the init_sec context exchange end on the wire? (A packet trace--tcpdump -s0 -wtmp.pcap, try the mount, then attach tmp.pcap--might help.)
Yes, the same happens with sec=krb5.
Created attachment 534080 [details] packet dump
> No key table entry found for CLIENT.XXXXX.COM$@XXXXX.COM while getting keytab I thinking this is the problem... gssd is not find a key tab entry....
(In reply to comment #4) > > No key table entry found for CLIENT.XXXXX.COM$@XXXXX.COM while getting keytab > I thinking this is the problem... gssd is not find a key tab entry.... Well, but a couple lines down there's: keytab entry for 'root/client.xxxxx.com' Success getting keytab entry for 'nfs/client.xxxxx.com' So I think it was just searching for one and it had to search a little farther. Looking at trace.... server is returning GSS_S_NO_CONTEXT error, along with a perfectly good context. Maybe a mismatch between what svcgssd thinks is supported and what the kernel can do? Is svcgssd's downcall to give the context to the kernel failing? You could find that out by strace'ing svcgssd.
Created attachment 534103 [details] strace of rpc.svcgssd you are correct, the write to /proc/net/rpc/auth.rpcsec.context/channel fails with EOPNOTSUPP
Seems like rpcsec_gss_krb5 does not get autoloaded. modprobe rpcsec_gss_krb5 makes the mount work.
OK, that will be fixed in the kernel by upstream 058c5c99999609e3de7e15b49049665f02d06577 "rpc: allow autoloading of gss mechanisms". In the mean time, we should be loading rpcsec_gss_krb5 by hand before starting rpc.svcgssd. It looks like that got lost in the migration to systemd. Steved? rpc.svcgssd's handling of errors is also a little odd: in the null init_sec_context response it's returning an error correctly, but also returning a context, which seems fairly pointless.
Something went wrong with the update (preupgrade seems to consistently fail to upgrade the bootloader), so I'm apparently still using the old kernel. That might explain why the autoload failed, as your patch seems to be present in fc16 kernels.
Oh, OK, that explains it. I think we should probably still do a manual modprobe of rpcsec_gss_krb5 for another fedora cycle or so, just in case--people might want to try old kernels for debugging purposes, etc.
So this is actually a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=737731
But yes, modprobing rpcsec_gss_krb5 for another release seems like a good idea to me
(In reply to comment #8) > OK, that will be fixed in the kernel by upstream > 058c5c99999609e3de7e15b49049665f02d06577 "rpc: allow autoloading of gss > mechanisms". > > In the mean time, we should be loading rpcsec_gss_krb5 by hand before starting > rpc.svcgssd. It looks like that got lost in the migration to systemd. Steved? > I'll look into seeing if I can get the systemd script to do the modprob for the short term...
(In reply to comment #8) > OK, that will be fixed in the kernel by upstream > 058c5c99999609e3de7e15b49049665f02d06577 "rpc: allow autoloading of gss > mechanisms". This fix is in the kernel-3.1.1-1.fc16 kernel which is the latest F16 kernel. So at this point all that's needed is a yum upgrade kernel