Bug 652273 - Unable to mount nfs4 krb5p shares exported by a fedora12 server
Unable to mount nfs4 krb5p shares exported by a fedora12 server
Status: CLOSED UPSTREAM
Product: Fedora
Classification: Fedora
Component: ipa (Show other bugs)
14
x86_64 Linux
low Severity medium
: ---
: ---
Assigned To: Rob Crittenden
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-11-11 09:15 EST by Thomas Sailer
Modified: 2012-03-27 02:51 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-27 02:51:19 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Thomas Sailer 2010-11-11 09:15:06 EST
Description of problem:
After upgrading a fully up-to-date and working fedora13 client to fedora14 two days ago, I am no longer able to mount nfs4 krb5p shares exported by a fedora12 freeipa server.

Version-Release number of selected component (if applicable):
nfs-utils-1.2.3-1.fc14.x86_64

How reproducible:
always, on two different clients

Steps to Reproduce:
mount -t nfs4 -o soft,intr,rsize=8192,wsize=8192,rw,sec=krb5p server.xxxx.xxx:/some/path/exported /tmp/x
  
Actual results:
mount fails

Expected results:
mount succeeds

Additional info:

rpc.gssd on the client reports the following:

beginning poll
dir_notify_handler: sig 37 si 0x7fff99e83030 data 0x7fff99e82f00
dir_notify_handler: sig 37 si 0x7fff99e7f930 data 0x7fff99e7f800
dir_notify_handler: sig 37 si 0x7fff99e82ef0 data 0x7fff99e82dc0
handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt38)
handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt38)
process_krb5_upcall: service is '<null>'
Full hostname for 'server.xxxx.xxx' is 'server.xxxx.xxx'
Full hostname for 'clnt.xxxx.xxx' is 'clnt.xxxx.xxx'
Key table entry not found while getting keytab entry for 'root/clnt.xxxx.xxx@XXXX.XXX'
Success getting keytab entry for 'nfs/clnt.xxxx.xxx@XXXX.XXX'
Successfully obtained machine credentials for principal 'nfs/clnt.xxxx.xxx@XXXX.XXX' stored in ccache 'FILE:/tmp/krb5cc_machine_XXXX.XXX'
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXX.XXX' are good until 1289651734
using FILE:/tmp/krb5cc_machine_XXXX.XXX as credentials cache for machine creds
using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_XXXX.XXX
creating context using fsuid 0 (save_uid 0)
creating tcp client for server server.xxxx.xxx
DEBUG: port already set to 2049
creating context with server nfs@server.xxxx.xxx
WARNING: Failed to create krb5 context for user with uid 0 for server server.xxxx.xxx
WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_XXXX.XXX for server server.xxxx.xxx
WARNING: Machine cache is prematurely expired or corrupted trying to recreate cache for server server.xxxx.xxx
Full hostname for 'server.xxxx.xxx' is 'server.xxxx.xxx'
Full hostname for 'clnt.xxxx.xxx' is 'clnt.xxxx.xxx'
Key table entry not found while getting keytab entry for 'root/clnt.xxxx.xxx@XXXX.XXX'
Success getting keytab entry for 'nfs/clnt.xxxx.xxx@XXXX.XXX'
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXX.XXX' are good until 1289651734
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_XXXX.XXX' are good until 1289651734
using FILE:/tmp/krb5cc_machine_XXXX.XXX as credentials cache for machine creds
using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_XXXX.XXX
creating context using fsuid 0 (save_uid 0)
creating tcp client for server server.xxxx.xxx
DEBUG: port already set to 2049
creating context with server nfs@server.xxxx.xxx
WARNING: Failed to create krb5 context for user with uid 0 for server server.xxxx.xxx
WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_XXXX.XXX for server server.xxxx.xxx
WARNING: Failed to create machine krb5 context with any credentials cache for server server.xxxx.xxx
doing error downcall
dir_notify_handler: sig 37 si 0x7fff99e83030 data 0x7fff99e82f00
dir_notify_handler: sig 37 si 0x7fff99e83030 data 0x7fff99e82f00
dir_notify_handler: sig 37 si 0x7fff99e82f30 data 0x7fff99e82e00
dir_notify_handler: sig 37 si 0x7fff99e7dfb0 data 0x7fff99e7de80
dir_notify_handler: sig 37 si 0x7fff99e7dfb0 data 0x7fff99e7de80
dir_notify_handler: sig 37 si 0x7fff99e7dfb0 data 0x7fff99e7de80
dir_notify_handler: sig 37 si 0x7fff99e7dfb0 data 0x7fff99e7de80
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt39
destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt38

I need to downgrade the kernel and krb5* to the Fedora13 version to get
nfs4 working again.
Comment 1 Thomas Sailer 2010-11-11 09:19:22 EST
*** Bug 652275 has been marked as a duplicate of this bug. ***
Comment 2 Daniel Piddock 2010-11-26 06:48:52 EST
I'm suffering from the same issue, although I'm attempting to mount nfs3 from Debian Lenny (linux 2.6.32-bpo.5-amd64, nfs-utils 1.2.2-4~cpo50+1).

/etc/krb5.conf contains:
[libdefaults]
  default_realm = INT.COREFILING.COM
  dns_lookup_kdc = true
  ticket_lifetime = 1d
  renew_lifetime = 7d
  forwardable = true
  proxiable = true
  allow_weak_crypto = true

The client's keytab contains (domain and realm stripped for line wrapping):
1    1 nfs/fedora14...@REALM (DES cbc mode with RSA-MD5)
2    1 nfs/fedora14...@REALM (DES cbc mode with RSA-MD4)
3    1 nfs/fedora14...@REALM (DES cbc mode with CRC-32)
4    1 nfs/fedora14...@REALM (AES-256 CTS mode with 96-bit SHA-1 HMAC)
5    1 nfs/fedora14...@REALM (Triple DES cbc mode with HMAC/sha1)
6    1 nfs/fedora14...@REALM (ArcFour with HMAC/md5)

When attempting a mount, gssd on the client gives the same output as comment 0. syslog on the server contains:
mountd[1590]: authenticated mount request from fedora14.int.corefiling.com:838 for /home/archive (/home)
rpc.svcgssd[1588]: leaving poll
rpc.svcgssd[1588]: handling null request
rpc.svcgssd[1588]: sname = nfs/fedora14.int.corefiling.com@INT.COREFILING.COM
rpc.svcgssd[1588]: libnfsidmap: using domain: int.corefiling.com
rpc.svcgssd[1588]: libnfsidmap: using translation method: nsswitch
rpc.svcgssd[1588]: DEBUG: serialize_krb5_ctx: lucid version!
rpc.svcgssd[1588]: ERROR: prepare_krb5_rfc_cfx_buffer: not implemented
rpc.svcgssd[1588]: serialize_krb5_ctx: prepare_krb5_*_buffer failed (retcode = -1)
rpc.svcgssd[1588]: ERROR: failed serializing krb5 context for kernel
rpc.svcgssd[1588]: WARNING: handle_nullreq: serialize_context_for_kernel failed
rpc.svcgssd[1588]: sending null reply
rpc.svcgssd[1588]: writing message: \x \x608202bf06092a864886f71201020201006e8202ae308202aaa003020105a10302010ea20703050020000000a38201a06182019c30820198a003020105a1141b12494e542e434f524546494c494e472e434f4da22d302ba003020101a12430221b036e66731b1b6576656e6c6f64652e696e742e636f726566696c696e672e636f6da382014a30820146a003020112a103020101a282013804820134dab959ad1bbad212fe3da41881b8415a5f40e1e6ef2f38e1ae5bb0e7720572e7d80887cba41acea4f604165d3c558705c14b737f06a9474812211d05a8185f2cd9585eeabeb43ddce62ea9621b2d0c5f5fe8fde25a11cc68475c4b5f1155417b12e4c1cc8bef1baddd4ddfe89de6bb4f874a0200d0d0dac94b05e293e0bc953d5ea764ed163cc5a51c1d5fce2d87f9f9f99183e4b1e8562342c1f38cb051ed1269f8aa0348935edc99202a2ca5e8f725cc7eba50306519894e3fb38b67cb6a32dabddfbc4a898bbed592488eea69781b2f0d81d0e95c1ad11da02443cfc71cd7d5dbcd24adba901e68a7b99931d8e422d10ae1449285e8dd3bf4bbd2f799ff9d8ebdcae94900d60cbab28982a64721394dd0df2623a58907c752618e3e01448540dde97ead624003896508049d65e0ca6a76df99a481f03081eda003020112a281e50481e211eb17afd0a8820217fb2c75ef09452730627ad86f7412bddb280e1cf15d8af8039ab0fc975d8a3b2d7218a38934ea80aaaef45e4be47566eea3d6bcbcc2814e99b66330cea0d9f43e2c911f1cdacce9dfe78a3e43fd2768433b9d970ba8c4a98d9f667cf00842d22180bc6f62749bcb6d7469f97df2f60dccbf4c758675b1e0b072cc61b3ce2a90ebf30886bd28d4b34ece3507590fae3ac528416c483597b92ec48bbe3cf94f9eff8ac8e0486a3e82b35f460022cc31c2f727d956316beddacb4b13716f7a7fcb31403bb09001fc50709921da5708c45824793c7afbc9537dc735 1290769634 851968 0 \x \x
rpc.svcgssd[1588]: finished handling null request
rpc.svcgssd[1588]: entering poll
rpc.svcgssd[1588]: leaving poll
rpc.svcgssd[1588]: handling null request
rpc.svcgssd[1588]: sname = nfs/fedora14.int.corefiling.com@INT.COREFILING.COM

The "DEBUG: serialize_krb5_ctx: lucid version!" to sname lines are then repeated 5 times (one for each keytab entry?)

Removing the AES-256, 3DES and ArcFour entries from the client's keytab in an attempt to force it to only use DES keys cause gssd to not even attempt communication. It fails to find valid keys:
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872570 data 0x7ffff2872440
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872570 data 0x7ffff2872440
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872570 data 0x7ffff2872440
rpc.gssd[2307]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7)
rpc.gssd[2307]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
rpc.gssd[2307]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7)
rpc.gssd[2307]: process_krb5_upcall: service is '<null>'
rpc.gssd[2307]: Full hostname for 'nfs1.int.corefiling.com' is 'nfs1.int.corefiling.com'
rpc.gssd[2307]: Full hostname for 'fedora14.int.corefiling.com' is 'fedora14.int.corefiling.com'
rpc.gssd[2307]: Key table entry not found while getting keytab entry for 'root/fedora14.int.corefiling.com@'
rpc.gssd[2307]: Success getting keytab entry for 'nfs/fedora14.int.corefiling.com@'
rpc.gssd[2307]: WARNING: Key table entry not found while getting initial ticket for principal 'nfs/fedora14.int.corefiling.com@INT.COREFILING.COM' using keytab 'WRFILE:/etc/krb5.keytab'
rpc.gssd[2307]: ERROR: No credentials found for connection to server nfs1.int.corefiling.com
rpc.gssd[2307]: doing error downcall
rpc.gssd[2307]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7)
rpc.gssd[2307]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
rpc.gssd[2307]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7)
rpc.gssd[2307]: process_krb5_upcall: service is '<null>'
rpc.gssd[2307]: Full hostname for 'nfs1.int.corefiling.com' is 'nfs1.int.corefiling.com'
rpc.gssd[2307]: Full hostname for 'fedora14.int.corefiling.com' is 'fedora14.int.corefiling.com'
rpc.gssd[2307]: Key table entry not found while getting keytab entry for 'root/fedora14.int.corefiling.com@'
rpc.gssd[2307]: Success getting keytab entry for 'nfs/fedora14.int.corefiling.com@'
rpc.gssd[2307]: WARNING: Key table entry not found while getting initial ticket for principal 'nfs/fedora14.int.corefiling.com@INT.COREFILING.COM' using keytab 'WRFILE:/etc/krb5.keytab'
rpc.gssd[2307]: ERROR: No credentials found for connection to server nfs1.int.corefiling.com
rpc.gssd[2307]: doing error downcall
rpc.gssd[2307]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7)
rpc.gssd[2307]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
rpc.gssd[2307]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt7)
rpc.gssd[2307]: process_krb5_upcall: service is '<null>'
rpc.gssd[2307]: Full hostname for 'nfs1.int.corefiling.com' is 'nfs1.int.corefiling.com'
rpc.gssd[2307]: Full hostname for 'fedora14.int.corefiling.com' is 'fedora14.int.corefiling.com'
rpc.gssd[2307]: Key table entry not found while getting keytab entry for 'root/fedora14.int.corefiling.com@'
rpc.gssd[2307]: Success getting keytab entry for 'nfs/fedora14.int.corefiling.com@'
rpc.gssd[2307]: WARNING: Key table entry not found while getting initial ticket for principal 'nfs/fedora14.int.corefiling.com@INT.COREFILING.COM' using keytab 'WRFILE:/etc/krb5.keytab'
rpc.gssd[2307]: ERROR: No credentials found for connection to server nfs1.int.corefiling.com
rpc.gssd[2307]: doing error downcall
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40
rpc.gssd[2307]: dir_notify_handler: sig 37 si 0x7ffff2872070 data 0x7ffff2871f40
rpc.gssd[2307]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt8
rpc.gssd[2307]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt7
rpc.gssd[2307]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt6

I created a user with only DES keys. They could authenticate without problem so allow_weak_crypto was being honored by other parts of the system.

Adding "default_tkt_enctypes = des-cbc-md5 des-cbc-md4 des-cbc-crc" to [libdefaults] section of krb5.conf allows the mount to succeed, even if using the keytab that contains the original 6 enctypes:
rpc.gssd[2569]: dir_notify_handler: sig 37 si 0x7fff945037f0 data 0x7fff945036c0
rpc.gssd[2569]: dir_notify_handler: sig 37 si 0x7fff945037f0 data 0x7fff945036c0
rpc.gssd[2569]: dir_notify_handler: sig 37 si 0x7fff945036b0 data 0x7fff94503580
rpc.gssd[2569]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt10)
rpc.gssd[2569]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
rpc.gssd[2569]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt10)
rpc.gssd[2569]: process_krb5_upcall: service is '<null>'
rpc.gssd[2569]: Full hostname for 'nfs1.int.corefiling.com' is 'nfs1.int.corefiling.com'
rpc.gssd[2569]: Full hostname for 'fedora14.int.corefiling.com' is 'fedora14.int.corefiling.com'
rpc.gssd[2569]: Key table entry not found while getting keytab entry for 'root/fedora14.int.corefiling.com@'
rpc.gssd[2569]: Success getting keytab entry for 'nfs/fedora14.int.corefiling.com@'
rpc.gssd[2569]: Successfully obtained machine credentials for principal 'nfs/fedora14.int.corefiling.com@INT.COREFILING.COM' stored in ccache 'FILE:/tmp/krb5cc_machine_INT.COREFILING.COM'
rpc.gssd[2569]: INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_INT.COREFILING.COM' are good until 1290861615
rpc.gssd[2569]: using FILE:/tmp/krb5cc_machine_INT.COREFILING.COM as credentials cache for machine creds
rpc.gssd[2569]: using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_INT.COREFILING.COM
rpc.gssd[2569]: creating context using fsuid 0 (save_uid 0)
rpc.gssd[2569]: creating tcp client for server nfs1.int.corefiling.com
rpc.gssd[2569]: DEBUG: port already set to 2049
rpc.gssd[2569]: creating context with server nfs@nfs1.int.corefiling.com
rpc.gssd[2569]: DEBUG: serialize_krb5_ctx: lucid version!
rpc.gssd[2569]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
rpc.gssd[2569]: doing downcall

Restricting default_tkt_enctypes may be a temporary solution until the server gets upgraded.
Comment 3 Steve Dickson 2010-12-01 13:31:45 EST
Does taking the machine's FQDN out of the localhost line in /etc/hosts help?
Comment 4 Thomas Sailer 2010-12-02 13:17:37 EST
Unfortunately not. It wasn't there to begin with. But I've taken the machine's FQDN out of the localhost6 line, and that didn't help either.

Original /etc/hosts:
# Do not remove the following line, or various programs
# that require network functionality will fail.
192.168.1.244   client.xxxx.xxx        client     # Added by NetworkManager
127.0.0.1       localhost.localdomain   localhost
::1     client.xxxx.xxx        client     localhost6.localdomain6 localhost6

I've now tried like this:
192.168.1.244   client     # Added by NetworkManager
127.0.0.1       localhost.localdomain   localhost
::1     client     localhost6.localdomain6 localhost6

And like this:
192.168.1.244   client     # Added by NetworkManager
127.0.0.1       localhost.localdomain   localhost
::1     localhost6.localdomain6 localhost6

All versions didn't work.

Then I reverted the /etc/hosts change, downgraded krb5, restarted rpcgssd and autofs, and it worked again.
Comment 5 Dirk Cummings 2010-12-05 09:28:40 EST
I can confirm this between an F14 server and client.

On the server:

 rpc.svcgssd -fvvvvv
entering poll
leaving poll
handling null request
sname = nfs/cipix@CIPISRVNETWORK
DEBUG: serialize_krb5_ctx: lucid version!
prepare_krb5_rfc4121_buffer: protocol 1
prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32
doing downcall
mech: krb5, hndl len: 4, ctx len 52, timeout: 1291645057 (86281 from now), clnt: nfs@cipix, uid: -1, gid: -1, num aux grps: 0:
: qword_eol: fflush failed: errno 38 (Function not implemented)


The client ALWAYS serializes key with "enctype 18 and size 32", whereas it should say "serializing keys with enctype 4 and length 8" (signifying des)

Adding  default_tkt_enctypes = des-cbc-crc:normal des-cbc-md4 des-cbc-md5
changes nothing
Comment 6 Dirk Cummings 2010-12-07 04:57:06 EST
Note, I solved this problem by upgrading to the rawhide kernel 2.6.36.1, which apparently can handle the higher crypto with enctype 18 and size 32
Comment 7 Thomas Sailer 2010-12-08 10:38:51 EST
I'm reassigning this to ipa, as it was solved for me by backporting an ipa patch to use openldap instead of mozldap. The package that works for me is here:
http://sailer.fedorapeople.org/ipa-1.2.2-5.fc14.jnx.src.rpm
Comment 8 Dirk Cummings 2010-12-26 05:05:32 EST
Everything works using the latest rawhdide kernel and nfs packages
Comment 9 Rob Crittenden 2011-03-10 18:19:10 EST
This should be fixed by the patch in BZ 658832.
Comment 10 Tim Niemueller 2011-04-12 07:46:53 EDT
I seem to suffer from the same problem with a CentOS 5.6 server and a Fedora 14 client when trying to mount a directory via NFSv4 and sec=krb5 with Kerberos enabled. Kerberos itself seems to work (kadmin stuff is working), NFSv4 itself works (exporting and mounting the old way just restricting by host/subnet). As soon as I enable the gss export entries and add sec=krb5 to the client it fails with "access denied", and I'm seeing "ERROR: prepare_krb5_rfc_cfx_buffer: not implemented" in the server's log for rpc.svcgssd.

Currently I have a minimal setup: LDAP with a single user "tim", Kerberos with principals for KDC, nfs/server, host/server and nfs/client and host/client. I have exported the respective nfs/ and host/ keys on both, client and server.

I tried the methods from comment #2, but it didn't change anything. I can provide logs if that would help. Any idea how to get this working?
Comment 11 Tim Niemueller 2011-04-13 08:57:32 EDT
I have fixed this for now. If you run into this it is crucial to export the des-cbc-crc:normal key and only this key type on both, client and server and allow weak cryptos on both. I had done the former only on the client, which results in errors. The technical note of bug #573968 (upper right corner) explicitly states that better crypto algorithms are currently unsupported with NFS. Does anybody know if this is still the case with RHEL 6 and Fedora 14 (if you have an equally new server system)?
Comment 12 Dirk Cummings 2011-05-18 19:49:29 EDT
I have Fedora 14 and the aes256 cipher works perfectly, I think since 2.6.38.

More specifically in my /var/kerberos/krb5kdc/kadm5.acl, I have
 supported_enctypes = aes256-cts:normal

And nfs works great
Comment 13 Martin Kosek 2012-03-27 02:51:19 EDT
Closing as per Rob's comment.

Note You need to log in before you can comment on or make changes to this bug.