Description of problem: After upgrading a fully up-to-date and working fedora13 client to fedora14 two days ago, I am no longer able to mount nfs4 krb5p shares exported by a fedora12 freeipa server. Version-Release number of selected component (if applicable): nfs-utils-1.2.3-1.fc14.x86_64 How reproducible: always, on two different clients Steps to Reproduce: mount -t nfs4 -o soft,intr,rsize=8192,wsize=8192,rw,sec=krb5p server.xxxx.xxx:/some/path/exported /tmp/x Actual results: mount fails Expected results: mount succeeds Additional info: rpc.gssd on the client reports the following: beginning poll dir_notify_handler: sig 37 si 0x7fff99e83030 data 0x7fff99e82f00 dir_notify_handler: sig 37 si 0x7fff99e7f930 data 0x7fff99e7f800 dir_notify_handler: sig 37 si 0x7fff99e82ef0 data 0x7fff99e82dc0 handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt38) handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt38) process_krb5_upcall: service is '<null>' Full hostname for 'server.xxxx.xxx' is 'server.xxxx.xxx' Full hostname for 'clnt.xxxx.xxx' is 'clnt.xxxx.xxx' Key table entry not found while getting keytab entry for 'root/clnt.xxxx.xxx' Success getting keytab entry for 'nfs/clnt.xxxx.xxx' Successfully obtained machine credentials for principal 'nfs/clnt.xxxx.xxx' stored in ccache 'FILE:/tmp/krb5cc_machine_XXXX.XXX' INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXX.XXX' are good until 1289651734 using FILE:/tmp/krb5cc_machine_XXXX.XXX as credentials cache for machine creds using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_XXXX.XXX creating context using fsuid 0 (save_uid 0) creating tcp client for server server.xxxx.xxx DEBUG: port already set to 2049 creating context with server nfs.xxx WARNING: Failed to create krb5 context for user with uid 0 for server server.xxxx.xxx WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_XXXX.XXX for server server.xxxx.xxx WARNING: Machine cache is prematurely expired or corrupted trying to recreate cache for server server.xxxx.xxx Full hostname for 'server.xxxx.xxx' is 'server.xxxx.xxx' Full hostname for 'clnt.xxxx.xxx' is 'clnt.xxxx.xxx' Key table entry not found while getting keytab entry for 'root/clnt.xxxx.xxx' Success getting keytab entry for 'nfs/clnt.xxxx.xxx' INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXX.XXX' are good until 1289651734 INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXX.XXX' are good until 1289651734 using FILE:/tmp/krb5cc_machine_XXXX.XXX as credentials cache for machine creds using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_XXXX.XXX creating context using fsuid 0 (save_uid 0) creating tcp client for server server.xxxx.xxx DEBUG: port already set to 2049 creating context with server nfs.xxx WARNING: Failed to create krb5 context for user with uid 0 for server server.xxxx.xxx WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_XXXX.XXX for server server.xxxx.xxx WARNING: Failed to create machine krb5 context with any credentials cache for server server.xxxx.xxx doing error downcall dir_notify_handler: sig 37 si 0x7fff99e83030 data 0x7fff99e82f00 dir_notify_handler: sig 37 si 0x7fff99e83030 data 0x7fff99e82f00 dir_notify_handler: sig 37 si 0x7fff99e82f30 data 0x7fff99e82e00 dir_notify_handler: sig 37 si 0x7fff99e7dfb0 data 0x7fff99e7de80 dir_notify_handler: sig 37 si 0x7fff99e7dfb0 data 0x7fff99e7de80 dir_notify_handler: sig 37 si 0x7fff99e7dfb0 data 0x7fff99e7de80 dir_notify_handler: sig 37 si 0x7fff99e7dfb0 data 0x7fff99e7de80 destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt39 destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt38 I need to downgrade the kernel and krb5* to the Fedora13 version to get nfs4 working again.
*** Bug 652275 has been marked as a duplicate of this bug. ***
I'm suffering from the same issue, although I'm attempting to mount nfs3 from Debian Lenny (linux 2.6.32-bpo.5-amd64, nfs-utils 1.2.2-4~cpo50+1). /etc/krb5.conf contains: [libdefaults] default_realm = INT.COREFILING.COM dns_lookup_kdc = true ticket_lifetime = 1d renew_lifetime = 7d forwardable = true proxiable = true allow_weak_crypto = true The client's keytab contains (domain and realm stripped for line wrapping): 1 1 nfs/fedora14...@REALM (DES cbc mode with RSA-MD5) 2 1 nfs/fedora14...@REALM (DES cbc mode with RSA-MD4) 3 1 nfs/fedora14...@REALM (DES cbc mode with CRC-32) 4 1 nfs/fedora14...@REALM (AES-256 CTS mode with 96-bit SHA-1 HMAC) 5 1 nfs/fedora14...@REALM (Triple DES cbc mode with HMAC/sha1) 6 1 nfs/fedora14...@REALM (ArcFour with HMAC/md5) When attempting a mount, gssd on the client gives the same output as comment 0. syslog on the server contains: mountd[1590]: authenticated mount request from fedora14.int.corefiling.com:838 for /home/archive (/home) rpc.svcgssd[1588]: leaving poll rpc.svcgssd[1588]: handling null request rpc.svcgssd[1588]: sname = nfs/fedora14.int.corefiling.com.COM rpc.svcgssd[1588]: libnfsidmap: using domain: int.corefiling.com rpc.svcgssd[1588]: libnfsidmap: using translation method: nsswitch rpc.svcgssd[1588]: DEBUG: serialize_krb5_ctx: lucid version! rpc.svcgssd[1588]: ERROR: prepare_krb5_rfc_cfx_buffer: not implemented rpc.svcgssd[1588]: serialize_krb5_ctx: prepare_krb5_*_buffer failed (retcode = -1) rpc.svcgssd[1588]: ERROR: failed serializing krb5 context for kernel rpc.svcgssd[1588]: WARNING: handle_nullreq: serialize_context_for_kernel failed rpc.svcgssd[1588]: sending null reply rpc.svcgssd[1588]: writing message: \x \x608202bf06092a864886f71201020201006e8202ae308202aaa003020105a10302010ea20703050020000000a38201a06182019c30820198a003020105a1141b12494e542e434f524546494c494e472e434f4da22d302ba003020101a12430221b036e66731b1b6576656e6c6f64652e696e742e636f726566696c696e672e636f6da382014a30820146a003020112a103020101a282013804820134dab959ad1bbad212fe3da41881b8415a5f40e1e6ef2f38e1ae5bb0e7720572e7d80887cba41acea4f604165d3c558705c14b737f06a9474812211d05a8185f2cd9585eeabeb43ddce62ea9621b2d0c5f5fe8fde25a11cc68475c4b5f1155417b12e4c1cc8bef1baddd4ddfe89de6bb4f874a0200d0d0dac94b05e293e0bc953d5ea764ed163cc5a51c1d5fce2d87f9f9f99183e4b1e8562342c1f38cb051ed1269f8aa0348935edc99202a2ca5e8f725cc7eba50306519894e3fb38b67cb6a32dabddfbc4a898bbed592488eea69781b2f0d81d0e95c1ad11da02443cfc71cd7d5dbcd24adba901e68a7b99931d8e422d10ae1449285e8dd3bf4bbd2f799ff9d8ebdcae94900d60cbab28982a64721394dd0df2623a58907c752618e3e01448540dde97ead624003896508049d65e0ca6a76df99a481f03081eda003020112a281e50481e211eb17afd0a8820217fb2c75ef09452730627ad86f7412bddb280e1cf15d8af8039ab0fc975d8a3b2d7218a38934ea80aaaef45e4be47566eea3d6bcbcc2814e99b66330cea0d9f43e2c911f1cdacce9dfe78a3e43fd2768433b9d970ba8c4a98d9f667cf00842d22180bc6f62749bcb6d7469f97df2f60dccbf4c758675b1e0b072cc61b3ce2a90ebf30886bd28d4b34ece3507590fae3ac528416c483597b92ec48bbe3cf94f9eff8ac8e0486a3e82b35f460022cc31c2f727d956316beddacb4b13716f7a7fcb31403bb09001fc50709921da5708c45824793c7afbc9537dc735 1290769634 851968 0 \x \x rpc.svcgssd[1588]: finished handling null request rpc.svcgssd[1588]: entering poll rpc.svcgssd[1588]: leaving poll rpc.svcgssd[1588]: handling null request rpc.svcgssd[1588]: sname = nfs/fedora14.int.corefiling.com.COM The "DEBUG: serialize_krb5_ctx: lucid version!" to sname lines are then repeated 5 times (one for each keytab entry?) Removing the AES-256, 3DES and ArcFour entries from the client's keytab in an attempt to force it to only use DES keys cause gssd to not even attempt communication. It fails to find valid keys: rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872570 data 0x7ffff2872440 rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872570 data 0x7ffff2872440 rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872570 data 0x7ffff2872440 rpc.gssd[2307]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7) rpc.gssd[2307]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' rpc.gssd[2307]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7) rpc.gssd[2307]: process_krb5_upcall: service is '<null>' rpc.gssd[2307]: Full hostname for 'nfs1.int.corefiling.com' is 'nfs1.int.corefiling.com' rpc.gssd[2307]: Full hostname for 'fedora14.int.corefiling.com' is 'fedora14.int.corefiling.com' rpc.gssd[2307]: Key table entry not found while getting keytab entry for 'root/fedora14.int.corefiling.com@' rpc.gssd[2307]: Success getting keytab entry for 'nfs/fedora14.int.corefiling.com@' rpc.gssd[2307]: WARNING: Key table entry not found while getting initial ticket for principal 'nfs/fedora14.int.corefiling.com.COM' using keytab 'WRFILE:/etc/krb5.keytab' rpc.gssd[2307]: ERROR: No credentials found for connection to server nfs1.int.corefiling.com rpc.gssd[2307]: doing error downcall rpc.gssd[2307]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7) rpc.gssd[2307]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' rpc.gssd[2307]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7) rpc.gssd[2307]: process_krb5_upcall: service is '<null>' rpc.gssd[2307]: Full hostname for 'nfs1.int.corefiling.com' is 'nfs1.int.corefiling.com' rpc.gssd[2307]: Full hostname for 'fedora14.int.corefiling.com' is 'fedora14.int.corefiling.com' rpc.gssd[2307]: Key table entry not found while getting keytab entry for 'root/fedora14.int.corefiling.com@' rpc.gssd[2307]: Success getting keytab entry for 'nfs/fedora14.int.corefiling.com@' rpc.gssd[2307]: WARNING: Key table entry not found while getting initial ticket for principal 'nfs/fedora14.int.corefiling.com.COM' using keytab 'WRFILE:/etc/krb5.keytab' rpc.gssd[2307]: ERROR: No credentials found for connection to server nfs1.int.corefiling.com rpc.gssd[2307]: doing error downcall rpc.gssd[2307]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7) rpc.gssd[2307]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' rpc.gssd[2307]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7) rpc.gssd[2307]: process_krb5_upcall: service is '<null>' rpc.gssd[2307]: Full hostname for 'nfs1.int.corefiling.com' is 'nfs1.int.corefiling.com' rpc.gssd[2307]: Full hostname for 'fedora14.int.corefiling.com' is 'fedora14.int.corefiling.com' rpc.gssd[2307]: Key table entry not found while getting keytab entry for 'root/fedora14.int.corefiling.com@' rpc.gssd[2307]: Success getting keytab entry for 'nfs/fedora14.int.corefiling.com@' rpc.gssd[2307]: WARNING: Key table entry not found while getting initial ticket for principal 'nfs/fedora14.int.corefiling.com.COM' using keytab 'WRFILE:/etc/krb5.keytab' rpc.gssd[2307]: ERROR: No credentials found for connection to server nfs1.int.corefiling.com rpc.gssd[2307]: doing error downcall rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40 rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40 rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40 rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40 rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40 rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40 rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40 rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40 rpc.gssd[2307]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt8 rpc.gssd[2307]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt7 rpc.gssd[2307]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt6 I created a user with only DES keys. They could authenticate without problem so allow_weak_crypto was being honored by other parts of the system. Adding "default_tkt_enctypes = des-cbc-md5 des-cbc-md4 des-cbc-crc" to [libdefaults] section of krb5.conf allows the mount to succeed, even if using the keytab that contains the original 6 enctypes: rpc.gssd[2569]: dir_notify_handler: sig 37 si 0x7fff945037f0 data 0x7fff945036c0 rpc.gssd[2569]: dir_notify_handler: sig 37 si 0x7fff945037f0 data 0x7fff945036c0 rpc.gssd[2569]: dir_notify_handler: sig 37 si 0x7fff945036b0 data 0x7fff94503580 rpc.gssd[2569]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt10) rpc.gssd[2569]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' rpc.gssd[2569]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt10) rpc.gssd[2569]: process_krb5_upcall: service is '<null>' rpc.gssd[2569]: Full hostname for 'nfs1.int.corefiling.com' is 'nfs1.int.corefiling.com' rpc.gssd[2569]: Full hostname for 'fedora14.int.corefiling.com' is 'fedora14.int.corefiling.com' rpc.gssd[2569]: Key table entry not found while getting keytab entry for 'root/fedora14.int.corefiling.com@' rpc.gssd[2569]: Success getting keytab entry for 'nfs/fedora14.int.corefiling.com@' rpc.gssd[2569]: Successfully obtained machine credentials for principal 'nfs/fedora14.int.corefiling.com.COM' stored in ccache 'FILE:/tmp/krb5cc_machine_INT.COREFILING.COM' rpc.gssd[2569]: INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_INT.COREFILING.COM' are good until 1290861615 rpc.gssd[2569]: using FILE:/tmp/krb5cc_machine_INT.COREFILING.COM as credentials cache for machine creds rpc.gssd[2569]: using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_INT.COREFILING.COM rpc.gssd[2569]: creating context using fsuid 0 (save_uid 0) rpc.gssd[2569]: creating tcp client for server nfs1.int.corefiling.com rpc.gssd[2569]: DEBUG: port already set to 2049 rpc.gssd[2569]: creating context with server nfs.corefiling.com rpc.gssd[2569]: DEBUG: serialize_krb5_ctx: lucid version! rpc.gssd[2569]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8 rpc.gssd[2569]: doing downcall Restricting default_tkt_enctypes may be a temporary solution until the server gets upgraded.
Does taking the machine's FQDN out of the localhost line in /etc/hosts help?
Unfortunately not. It wasn't there to begin with. But I've taken the machine's FQDN out of the localhost6 line, and that didn't help either. Original /etc/hosts: # Do not remove the following line, or various programs # that require network functionality will fail. 192.168.1.244 client.xxxx.xxx client # Added by NetworkManager 127.0.0.1 localhost.localdomain localhost ::1 client.xxxx.xxx client localhost6.localdomain6 localhost6 I've now tried like this: 192.168.1.244 client # Added by NetworkManager 127.0.0.1 localhost.localdomain localhost ::1 client localhost6.localdomain6 localhost6 And like this: 192.168.1.244 client # Added by NetworkManager 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6 All versions didn't work. Then I reverted the /etc/hosts change, downgraded krb5, restarted rpcgssd and autofs, and it worked again.
I can confirm this between an F14 server and client. On the server: rpc.svcgssd -fvvvvv entering poll leaving poll handling null request sname = nfs/cipix@CIPISRVNETWORK DEBUG: serialize_krb5_ctx: lucid version! prepare_krb5_rfc4121_buffer: protocol 1 prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32 doing downcall mech: krb5, hndl len: 4, ctx len 52, timeout: 1291645057 (86281 from now), clnt: nfs@cipix, uid: -1, gid: -1, num aux grps: 0: : qword_eol: fflush failed: errno 38 (Function not implemented) The client ALWAYS serializes key with "enctype 18 and size 32", whereas it should say "serializing keys with enctype 4 and length 8" (signifying des) Adding default_tkt_enctypes = des-cbc-crc:normal des-cbc-md4 des-cbc-md5 changes nothing
Note, I solved this problem by upgrading to the rawhide kernel 2.6.36.1, which apparently can handle the higher crypto with enctype 18 and size 32
I'm reassigning this to ipa, as it was solved for me by backporting an ipa patch to use openldap instead of mozldap. The package that works for me is here: http://sailer.fedorapeople.org/ipa-1.2.2-5.fc14.jnx.src.rpm
Everything works using the latest rawhdide kernel and nfs packages
This should be fixed by the patch in BZ 658832.
I seem to suffer from the same problem with a CentOS 5.6 server and a Fedora 14 client when trying to mount a directory via NFSv4 and sec=krb5 with Kerberos enabled. Kerberos itself seems to work (kadmin stuff is working), NFSv4 itself works (exporting and mounting the old way just restricting by host/subnet). As soon as I enable the gss export entries and add sec=krb5 to the client it fails with "access denied", and I'm seeing "ERROR: prepare_krb5_rfc_cfx_buffer: not implemented" in the server's log for rpc.svcgssd. Currently I have a minimal setup: LDAP with a single user "tim", Kerberos with principals for KDC, nfs/server, host/server and nfs/client and host/client. I have exported the respective nfs/ and host/ keys on both, client and server. I tried the methods from comment #2, but it didn't change anything. I can provide logs if that would help. Any idea how to get this working?
I have fixed this for now. If you run into this it is crucial to export the des-cbc-crc:normal key and only this key type on both, client and server and allow weak cryptos on both. I had done the former only on the client, which results in errors. The technical note of bug #573968 (upper right corner) explicitly states that better crypto algorithms are currently unsupported with NFS. Does anybody know if this is still the case with RHEL 6 and Fedora 14 (if you have an equally new server system)?
I have Fedora 14 and the aes256 cipher works perfectly, I think since 2.6.38. More specifically in my /var/kerberos/krb5kdc/kadm5.acl, I have supported_enctypes = aes256-cts:normal And nfs works great
Closing as per Rob's comment.