Bug 802469

Summary: RHEL6 fails to mount from RHEL5/Debian 6 nfs4 server with sec=krb5
Product: Red Hat Enterprise Linux 6 Reporter: Michael Weiser <m.weiser>
Component: nfs-utilsAssignee: Steve Dickson <steved>
Status: CLOSED ERRATA QA Contact: yanfu,wang <yanwang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2CC: nalin, yanwang
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: nfs-utils-1.2.3-30.el6 Doc Type: Rebase: Bug Fixes Only
Doc Text:
Cause: nfs-utils-1.2.3 which allows to use encryption types stronger than Single DES: Consequence: mount using the -o sec=krb5 option fails to legacy servers Fix: Add a -l flag to allow only Single DES to be used Result: Secure mounts work with legacy servers
 Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 10:55:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
add -l option to gssd of nfs-utils-1.2.3 to enable legacy behaviour and (untested) subkey negotiation patches for RHEL5/6
none
add -l option to gssd of nfs-utils-1.2.3 to enable legacy behaviour and (untested) subkey negotiation patches for RHEL5/6 none

Description Michael Weiser 2012-03-12 16:09:02 UTC 
Created attachment 569447 [details]
add -l option to gssd of nfs-utils-1.2.3 to enable legacy behaviour and (untested) subkey negotiation patches for RHEL5/6

Description of problem:
When trying to mount an export from a RHEL5 server via NFSv4 with security method krb5, a RHEL6 client reports:

mount.nfs: access denied by server while mounting nfs:/

The RHEL5 server logs the following error in the kernel log:
gss_kerberos_mech: unsupported algorithm 23

Version-Release number of selected component (if applicable):
RHEL6 client: nfs-utils-1.2.3-15.el6.x86_64
RHEL5 server: nfs-utils-1.0.9-60.el5.i386

How reproducible:

Steps to Reproduce:
1. mount -o vers=4,sec=krb5 nfs:/ /mnt
  
Actual results:
mount.nfs: access denied by server while mounting nfs:/

Expected results:
no output, /mnt mounted from server nfs

Additional info:
The RHEL6 client is 6.2 x86_64 with current updates.
The RHEL5 server is 5.8 i386 with current updates.
Both are VMware VMs.

The problem seems to be caused by new functionality in nfs-utils-1.2.3 which allows to use encryption types stronger than Single DES: Previously, the NFSv4 client (gssd in particular) would proactively limit Kerberos encryption types for session keys to des-cbc-crc (and perhaps -md5 and -md4). So while the actual ticket might be encrypted using something way stronger, the included session key would only be Single DES.

Due to this self-limitation of the client, the server would never get anything else than Single DES session keys and therefore had no mechanism of working around the problems caused by stronger session keys with a kernel that does not support them.

Beginning with version 1.2.3 of nfs-utils, gssd does no longer limits itself to Single DES. Instead, it detects the encryption types supported by the *client's* kernel and limits the session key's encryption type to those reported as supported. This, however, ignores the fact, that the *server's* kernel might not support them. Since the user space of the server (svcgssd) does not check what its kernel supports and has no mechanism of renegotiating with the client, it happily accepts stronger encryption types and tries to push them into the kernel. This causes the above error message in the kernel log.

A direct workaround is to set the following options in /etc/krb5.conf of client and server:

[libdefaults]
default_tkt_enctypes = des-cbc-md5
permitted_enctypes = des-cbc-md5

, add des-cbc-md5 keys to the keytabs of both machines and allow Single DES for both machines' principals on the KDC (MS AD 2008r2 in particular wants it enabled explicitly). This however not only limits the encryption types of session keys but all tickets as well and applies to the whole machine not just the NFSv4 service. This has a needlessly high security impact on both machines.

I quickly implemented a more constrained fix that adds an option -l to rpc.gssd that effectively reverts it to legacy behaviour: It ignores its local kernel's capabilities and uses just Single DES. This option could be set in /etc/sysconfig/nfs:GSSDARGS on all RHEL6 clients that need to work with RHEL5 servers. It could possibly be enhanced to be configurable for individual on a per-machine basis. Inclusion of this or a similar patch in an updated package would be greatly appreciated.

Also, according to http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=622146, the problem is addressed by recent patches to nfs-utils and mit-krb5, allowing subkey negotiation to be used to negotiate an encryption type supported by the kernels of both client and server. The debian bug report gives a good overview of what has to be done on client and server. It also references and includes patches for nfs-utils and mit-krb5.

However, this method does not seem to work at the moment: I've tried it with a current squeeze installation without success. Also, it could not work with RHEL5 since its mit-krb5 is version 1.6 while subkey negotiation was introduced with 1.7. I tried backporting the patches anyway but, as was to be expected, had no luck with it.

A so-patched RHEL6 should however be able to mount exports of a Debian squeeze server that has the mentioned updated packages installed. Since it does not currently work on the Debian side, I've had no luck trying if it actually does work. Since such a solution to my actual RHEL5/6 mounting problem (by possibly backporting subkey negotiation to mit-krb5 1.6) would be great, I include my changes to the RHEL packages anyway.

Thanks in advance,
Michael
Comment 2 Michael Weiser 2012-03-12 16:19:05 UTC 
Created attachment 569449 [details]
add -l option to gssd of nfs-utils-1.2.3 to enable legacy behaviour and (untested) subkey negotiation patches for RHEL5/6

It's actually .bz2 not .gz. Sorry.

The principal patch is patches/rhel6/nfs-utils-1.2.3-legacy-enctypes.patch.

The rest is compiling but untested subkey negotiation stuff. For RHEL5 it needs libgssglue > 0.3 for the -n option to gssd to continue working. I grabbed that one from Fedora Rawhide.
Comment 3 Steve Dickson 2012-03-12 19:05:46 UTC 
Would it be possible to post these patch to the upstream NFS mailing list (linux-nfs.org) in the proper format... tia...
Comment 4 Michael Weiser 2012-03-12 19:30:46 UTC 
(In reply to comment #3)
> Would it be possible to post these patch to the upstream NFS mailing list
> (linux-nfs.org) in the proper format... tia...

I will do that. I have also contacted the Debian package maintainers from the bug I mentioned to determine if and possibly why their fix is not currently working.

How likely is it that a fix from upstream will be backported to RHEL6?

I have also asked our customer who is seeing this problem to open a case via RedHat Customer Center referencing this bug.

Thanks,
Micha
Comment 5 Steve Dickson 2012-03-14 03:40:33 UTC 
(In reply to comment #4)
> (In reply to comment #3)
> > Would it be possible to post these patch to the upstream NFS mailing list
> > (linux-nfs.org) in the proper format... tia...
> 
> I will do that. I have also contacted the Debian package maintainers from the
> bug I mentioned to determine if and possibly why their fix is not currently
> working.
thank you... 
> 
> How likely is it that a fix from upstream will be backported to RHEL6?
If it gets accepted I see no reason not to backport the patch...

> 
> I have also asked our customer who is seeing this problem to open a case via
> RedHat Customer Center referencing this bug.
Perfect! thanks!
Comment 6 Michael Weiser 2012-04-16 13:44:23 UTC 
A fix has been committed upstream: http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commit;h=16f151834e63d8df9f852b7e265f17c689553c35

Can it be backported to RHEL6?

Thanks,
Micha
Comment 7 RHEL Program Management 2012-05-03 05:14:57 UTC 
Since RHEL 6.3 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.
Comment 10 yanfu,wang 2012-12-11 03:51:50 UTC 
hi Michael，
Seems I got failed test result, could you help to check my set up?

My /etc/krb5.conf in KDC, nfs server and client shown as below:
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = RHTS.ENG.NAY.REDHAT.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 RHTS.ENG.NAY.REDHAT.COM = {
  kdc = ibm-x3550m3-05.rhts.eng.nay.redhat.com
  admin_server = ibm-x3550m3-05.rhts.eng.nay.redhat.com
 }

[domain_realm]
 .rhts.eng.nay.redhat.com = RHTS.ENG.NAY.REDHAT.COM
 rhts.eng.nay.redhat.com = RHTS.ENG.NAY.REDHAT.COM


nfs server(RHEL5.8):
[root@ibm-hs21-01 ~]# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/ibm-hs21-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (AES-256 CTS mode with 96-bit SHA-1 HMAC) 
   2 nfs/ibm-hs21-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (AES-128 CTS mode with 96-bit SHA-1 HMAC) 
   2 nfs/ibm-hs21-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (Triple DES cbc mode with HMAC/sha1) 
   2 nfs/ibm-hs21-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (ArcFour with HMAC/md5) 
   3 nfs/ibm-hs21-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (DES cbc mode with CRC-32) 


nfs client(RHEL6.2 kernel and nfs-utils):
[root@ibm-x3650-03 ~]# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (aes256-cts-hmac-sha1-96) 
   2 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (aes128-cts-hmac-sha1-96) 
   2 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (des3-cbc-sha1) 
   2 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (arcfour-hmac) 
   3 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (des-cbc-crc) 


When I do mount from RHEL6 client with above setup, I could got 'access denied', but no 'gss_kerberos_mech: unsupported algorithm 23' in server log:
[root@ibm-x3650-03 ~]# mount -t nfs -o sec=krb5 ibm-hs21-01.rhts.eng.nay.redhat.com:/ /mnt
mount.nfs: access denied by server while mounting ibm-hs21-01.rhts.eng.nay.redhat.com:/

When I update to new 6.4 nfs-utils package(-35 version), set '-l' option in /etc/sysconfig/nfs:GSSDARGS and restart rpc.gssd, still fails to mount.

So I doubt my configuration is different from you. Could you help to check it? Thanks.
Comment 11 Nalin Dahyabhai 2012-12-11 16:13:35 UTC 
In case it helps, my expectation is that the server is only capable of using DES, and the client, by default, will refuse to use DES.

You'll need to set "allow_weak_crypto = true" in the [libdefaults] section of the client's /etc/krb5.conf file to allow it to use DES.  You may also need to set up the server's keytab again so that it only contains DES keys for the "nfs" service (or rather, so that the KDC will only have DES keys for it).
Comment 12 yanfu,wang 2012-12-12 10:26:48 UTC 
(In reply to comment #11)
> In case it helps, my expectation is that the server is only capable of using
> DES, and the client, by default, will refuse to use DES.
> 
> You'll need to set "allow_weak_crypto = true" in the [libdefaults] section
> of the client's /etc/krb5.conf file to allow it to use DES.  
yes, it made mount work, but I want to reproduce mount access denied like comment #0.

> You may also 
> need to set up the server's keytab again so that it only contains DES keys
> for the "nfs" service (or rather, so that the KDC will only have DES keys
> for it).
Now, my server only contains DES key for nfs:
[root@ibm-x3550m4-01 ~]# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/ibm-x3550m4-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (DES cbc mode with CRC-32) 

Client still is:
[root@ibm-x3650-03 ~]# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   8 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (des-cbc-crc) 
   9 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (aes256-cts-hmac-sha1-96) 
   9 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (aes128-cts-hmac-sha1-96) 
   9 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (des3-cbc-sha1) 
   9 nfs/ibm-x3650-03.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (arcfour-hmac) 

I'm not set "allow_weak_crypto = true" in order to trigger the initial issue, but run mount got unexpected work result:
[root@ibm-x3650-03 ~]# mount -t nfs ibm-x3550m4-01.rhts.eng.nay.redhat.com: /mnt
[root@ibm-x3650-03 ~]# echo $?
0

You said client will refuse to use DES by default, but seems not from the testing.
Do you have any hints? thanks in advance.
Comment 13 Michael Weiser 2012-12-12 11:15:40 UTC 
(In reply to comment #12)

> > In case it helps, my expectation is that the server is only capable of using
> > DES, and the client, by default, will refuse to use DES.
> > 
> > You'll need to set "allow_weak_crypto = true" in the [libdefaults] section
> > of the client's /etc/krb5.conf file to allow it to use DES.  
> yes, it made mount work, but I want to reproduce mount access denied like
> comment #0.

> Do you have any hints? thanks in advance.

No, the whole point of this bug is to make a RHEL6 nfs server work (again) with a keytab that contains stronger keys than Single DES when talking to an older server. Previous versions (RHEL5) would request (and get) tickets encrypted with a strong key but only a Single DES session key within. This would not require setting allow_weak_crypto = true and would still satisfy the kernel's NFSv4 implementation's limitation to Single DES.

Discussion on the nfs mailing list revealed that it never really was intended to work that way. It is also most likely specific to the Linux implementation. But since it does work and makes administrators' lives a lot simpler, I pushed for the patch to be included still. Here's the whole discussion on linux-nfs.org: http://markmail.org/message/4aezdk2ustyuqh6n.

With the updated package, RHEL6's rpc.gssd should now have a command line option -l. When started with this option, it will ignore its kernel's enhanced crypto capabilities and *not* request a ticket with strong encryption *and* a strong session key within. Instead it will revert to the *l*egacy behaviour of requesting a ticket with a Single DES session key inside.

So check that you have started rpc.gssd on the RHEL6 client with -l and watch the ticket cache /tmp/krb5cc_machine_REALM. It should look like this:

[root@rhel5s1 ~]# klist -e /tmp/krb5cc_machine_MYDOMAIN.EXAMPLE
Ticket cache: FILE:/tmp/krb5cc_machine_MYDOMAIN.EXAMPLE
Default principal: nfs/rhel5s1.mydomain.example

Valid starting Expires Service principal
03/14/12 13:53:32 03/14/12 23:53:32 krbtgt/MYDOMAIN.EXAMPLE
 renew until 03/15/12 13:53:32, Etype (skey, tkt): AES-256 CTS mode with 96-bit SHA-1 HMAC, AES-256 CTS mode
with 96-bit SHA-1 HMAC
03/14/12 13:53:32 03/14/12 23:53:32 nfs/rhel5s1.mydomain.example@
 renew until 03/15/12 13:53:32, Etype (skey, tkt): DES cbc mode with CRC-32, ArcFour with HMAC/md5 

and not like this:

[root@rhel6 ~]# klist -e /tmp/krb5cc_machine_MYDOMAIN.EXAMPLE Ticket cache: FILE:/tmp/krb5cc_machine_MYDOMAIN.EXAMPLE Default principal: nfs/rhel6.mydomain.example

Valid starting Expires Service principal
03/14/12 14:00:19 03/15/12 00:00:20 krbtgt/MYDOMAIN.EXAMPLE 
 renew until 03/21/12 14:00:19, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
03/14/12 14:00:20 03/15/12 00:00:20
nfs/rhel5s1.mydomain.example
 renew until 03/21/12 14:00:19, Etype (skey, tkt): arcfour-hmac, arcfour-hmac 

Note the DES cbc session key in the first case.

I have not yet gotten around to testing it myself but hope to do so within the next few days.

Thank you,
Michael Weiser
Comment 14 yanfu,wang 2012-12-19 09:54:29 UTC 
(In reply to comment #13)
> (In reply to comment #12)
> 
> > > In case it helps, my expectation is that the server is only capable of using
> > > DES, and the client, by default, will refuse to use DES.
> > > 
> > > You'll need to set "allow_weak_crypto = true" in the [libdefaults] section
> > > of the client's /etc/krb5.conf file to allow it to use DES.  
> > yes, it made mount work, but I want to reproduce mount access denied like
> > comment #0.
> 
> > Do you have any hints? thanks in advance.
> 
> No, the whole point of this bug is to make a RHEL6 nfs server work (again)
> with a keytab that contains stronger keys than Single DES when talking to an
> older server. Previous versions (RHEL5) would request (and get) tickets
> encrypted with a strong key but only a Single DES session key within. This
> would not require setting allow_weak_crypto = true and would still satisfy
> the kernel's NFSv4 implementation's limitation to Single DES.
> 
> Discussion on the nfs mailing list revealed that it never really was
> intended to work that way. It is also most likely specific to the Linux
> implementation. But since it does work and makes administrators' lives a lot
> simpler, I pushed for the patch to be included still. Here's the whole
> discussion on linux-nfs.org:
> http://markmail.org/message/4aezdk2ustyuqh6n.
> 
> With the updated package, RHEL6's rpc.gssd should now have a command line
> option -l. When started with this option, it will ignore its kernel's
> enhanced crypto capabilities and *not* request a ticket with strong
> encryption *and* a strong session key within. Instead it will revert to the
> *l*egacy behaviour of requesting a ticket with a Single DES session key
> inside.
> 
> So check that you have started rpc.gssd on the RHEL6 client with -l and
> watch the ticket cache /tmp/krb5cc_machine_REALM. It should look like this:
> 
> [root@rhel5s1 ~]# klist -e /tmp/krb5cc_machine_MYDOMAIN.EXAMPLE
> Ticket cache: FILE:/tmp/krb5cc_machine_MYDOMAIN.EXAMPLE
> Default principal: nfs/rhel5s1.mydomain.example
> 
> Valid starting Expires Service principal
> 03/14/12 13:53:32 03/14/12 23:53:32 krbtgt/MYDOMAIN.EXAMPLE
>  renew until 03/15/12 13:53:32, Etype (skey, tkt): AES-256 CTS mode with
> 96-bit SHA-1 HMAC, AES-256 CTS mode
> with 96-bit SHA-1 HMAC
> 03/14/12 13:53:32 03/14/12 23:53:32 nfs/rhel5s1.mydomain.example@
>  renew until 03/15/12 13:53:32, Etype (skey, tkt): DES cbc mode with CRC-32,
> ArcFour with HMAC/md5 
> 
> and not like this:
> 
> [root@rhel6 ~]# klist -e /tmp/krb5cc_machine_MYDOMAIN.EXAMPLE Ticket cache:
> FILE:/tmp/krb5cc_machine_MYDOMAIN.EXAMPLE Default principal:
> nfs/rhel6.mydomain.example
> 
> Valid starting Expires Service principal
> 03/14/12 14:00:19 03/15/12 00:00:20 krbtgt/MYDOMAIN.EXAMPLE 
>  renew until 03/21/12 14:00:19, Etype (skey, tkt): aes256-cts-hmac-sha1-96,
> aes256-cts-hmac-sha1-96
> 03/14/12 14:00:20 03/15/12 00:00:20
> nfs/rhel5s1.mydomain.example
>  renew until 03/21/12 14:00:19, Etype (skey, tkt): arcfour-hmac,
> arcfour-hmac 
> 
> Note the DES cbc session key in the first case.
> 

Seems I got 'WARNING: KDC has no support for encryption type... ' when added -l option to rpc.gssd and mount still failed.
nfs client:
[root@ibm-dx360m4-01 ~]# rpc.gssd -fvvv -l
beginning poll
dir_notify_handler: sig 37 si 0x7fff938d7070 data 0x7fff938d6f40
dir_notify_handler: sig 37 si 0x7fff938d7070 data 0x7fff938d6f40
dir_notify_handler: sig 37 si 0x7fff938d6fb0 data 0x7fff938d6e80
handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt4d)
handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt4d)
process_krb5_upcall: service is '<null>'
Full hostname for 'ibm-x3550m4-01.rhts.eng.nay.redhat.com' is 'ibm-x3550m4-01.rhts.eng.nay.redhat.com'
Full hostname for 'ibm-dx360m4-01.rhts.eng.bos.redhat.com' is 'ibm-dx360m4-01.rhts.eng.bos.redhat.com'
No key table entry found for IBM-DX360M4-01.RHTS.ENG.BOS.REDHAT.COM$@RHTS.ENG.NAY.REDHAT.COM while getting keytab entry for 'IBM-DX360M4-01.RHTS.ENG.BOS.REDHAT.COM$@RHTS.ENG.NAY.REDHAT.COM'
No key table entry found for root/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM while getting keytab entry for 'root/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM'
Success getting keytab entry for 'nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM'
WARNING: KDC has no support for encryption type while getting initial ticket for principal 'nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM' using keytab 'WRFILE:/etc/krb5.keytab'
ERROR: No credentials found for connection to server ibm-x3550m4-01.rhts.eng.nay.redhat.com

[root@ibm-dx360m4-01 ~]# klist -ke
Keytab name: WRFILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM (aes256-cts-hmac-sha1-96) 
   2 nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM (aes128-cts-hmac-sha1-96) 
   2 nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM (des3-cbc-sha1) 
   2 nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM (arcfour-hmac) 
   3 nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM (des-cbc-crc) 


nfs server:
[root@ibm-x3550m4-01 ~]# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/ibm-x3550m4-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (DES cbc mode with CRC-32) 


The /etc/krb5.conf is same as comment #10.

Is there any configure I should do?
Comment 15 Michael Weiser 2012-12-19 10:38:37 UTC 
> Seems I got 'WARNING: KDC has no support for encryption type... ' when added
> -l option to rpc.gssd and mount still failed.

> WARNING: KDC has no support for encryption type while getting initial ticket
> for principal
> 'nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM' using

> Is there any configure I should do?

Well, I'm not really in a position to debug your test setup, but I'd guess, you need to enable Single DES on your KDC. I do need to do so when using Windows AD (2003, 2003R2, 2008R2) but you might if you're using an MIT or Heimdal KDC or have special settings on a Windows KDC.

Where do I find the packages you're testing? I'd have had time today to give them a whirl in my setup but can't seem to find them.
Comment 16 yanfu,wang 2012-12-19 10:46:50 UTC 
I set up KDC in RHEL6.4. 
# rpm -qa|grep krb5
pam_krb5-2.3.11-9.el6.x86_64
krb5-workstation-1.10.3-6.el6.x86_64
krb5-server-1.10.3-6.el6.x86_64
krb5-libs-1.10.3-6.el6.x86_64
krb5-devel-1.10.3-6.el6.x86_64

There's RHEL6.4 beta available now, but maybe you need account:
http://www.redhat.com/about/news/archive/2012/12/latest-beta-available-of-red-hat-enterprise-linux-6-now-available

I found if I set “allow_weak_crypto = yes”  in /etc/krb5.conf on KDC and nfs client, mount could work. But from my understanding, mount should work when add '-l' option to rpc.gssd, even though without the workaround, is it right?
Comment 17 Michael Weiser 2012-12-19 11:02:58 UTC 
(In reply to comment #16)
> I set up KDC in RHEL6.4. 
> # rpm -qa|grep krb5
> pam_krb5-2.3.11-9.el6.x86_64
> krb5-workstation-1.10.3-6.el6.x86_64
> krb5-server-1.10.3-6.el6.x86_64
> krb5-libs-1.10.3-6.el6.x86_64
> krb5-devel-1.10.3-6.el6.x86_64
> 
> There's RHEL6.4 beta available now, but maybe you need account:
> http://www.redhat.com/about/news/archive/2012/12/latest-beta-available-of-
> red-hat-enterprise-linux-6-now-available

Okay, got it. I will most likely not be able to finish testing today and will be on vacation until 2013-01-07. I'll get back to you then.

> I found if I set “allow_weak_crypto = yes”  in /etc/krb5.conf on KDC and nfs
> client, mount could work. But from my understanding, mount should work when
> add '-l' option to rpc.gssd, even though without the workaround, is it right?

Correct, on the client side. You seem however need to activate Single DES on your KDC. This might require setting allow_weak_crypto in your server's krb5.conf, I'm not sure. It may also require extending the list of supported_enctypes in your kdc.conf.
Comment 18 Michael Weiser 2012-12-19 11:36:29 UTC 
> Okay, got it. I will most likely not be able to finish testing today and
> will be on vacation until 2013-01-07. I'll get back to you then.

It does seem to work. When rpc.gssd is started with (undocumented) -l on the RHEL6 client, it can mount from a RHEL5 server with strong keys in it's keytab. 

I do however need allow_weak_crypto = true on the RHEL6 client. I seem to have lost track of this condition over the months but my test VM still had it in it's krb5.conf.

[root@rhel5s1 ~]# cat /etc/krb5.conf 
[libdefaults]
 default_realm = MYDOMAIN.EXAMPLE
[root@rhel5s1 ~]# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/nfs.mydomain.example (ArcFour with HMAC/md5) 
   2 nfs/nfs.mydomain.example (AES-256 CTS mode with 96-bit SHA-1 HMAC) 

[root@rhel6 ~]# cat /etc/krb5.conf
[libdefaults]
 default_realm = MYDOMAIN.EXAMPLE
 allow_weak_crypto = true
[root@rhel6 ~]# klist -ke
Keytab name: WRFILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/rhel6.mydomain.example (aes256-cts-hmac-sha1-96) 
   2 nfs/rhel6.mydomain.example (arcfour-hmac) 
[root@rhel6 ~]# klist -e /tmp/krb5cc_machine_MYDOMAIN.EXAMPLE 
Ticket cache: FILE:/tmp/krb5cc_machine_MYDOMAIN.EXAMPLE
Default principal: nfs/rhel6.mydomain.example

Valid starting     Expires            Service principal
12/19/12 12:28:47  12/19/12 22:28:47  krbtgt/MYDOMAIN.EXAMPLE
        renew until 12/20/12 12:28:47, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 
12/19/12 12:28:47  12/19/12 22:28:47  nfs/nfs.mydomain.example
        renew until 12/20/12 12:28:47, Etype (skey, tkt): des-cbc-crc, arcfour-hmac 
[root@rhel6 ~]# mount -o vers=4,sec=krb5 nfs:/ /mnt
[root@rhel6 ~]#
Comment 19 yanfu,wang 2012-12-20 03:43:49 UTC 
I got the reason, that I need to let keytab on nfs server and client contain stronger keys first, then set allow_weak_crypto = true on the RHEL6 client krb5.conf.
nfs server:
[root@ibm-x3550m4-01 ~]# klist -ke
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 nfs/ibm-x3550m4-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (AES-256 CTS mode with 96-bit SHA-1 HMAC) 
   3 nfs/ibm-x3550m4-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (AES-128 CTS mode with 96-bit SHA-1 HMAC) 
   3 nfs/ibm-x3550m4-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (Triple DES cbc mode with HMAC/sha1) 
   3 nfs/ibm-x3550m4-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM (ArcFour with HMAC/md5) 

nfs client:
[root@ibm-dx360m4-01 ~]# klist -ke
Keytab name: WRFILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM (aes256-cts-hmac-sha1-96) 
   2 nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM (aes128-cts-hmac-sha1-96) 
   2 nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM (des3-cbc-sha1) 
   2 nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM (arcfour-hmac) 

So on RHEL6.2 old nfs-utils-1.2.3-15.el6:
[root@ibm-dx360m4-01 ~]# mount -t nfs ibm-x3550m4-01.rhts.eng.nay.redhat.com:/ /mnt -o sec=krb5 
mount.nfs: access denied by server while mounting ibm-x3550m4-01.rhts.eng.nay.redhat.com:/

klist shown the ticket with strong encryption *and* a strong session key within:
[root@ibm-dx360m4-01 ~]# klist -e /tmp/krb5cc_machine_RHTS.ENG.NAY.REDHAT.COM 
Ticket cache: FILE:/tmp/krb5cc_machine_RHTS.ENG.NAY.REDHAT.COM
Default principal: nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM

Valid starting     Expires            Service principal
12/19/12 22:03:55  12/20/12 22:03:54  krbtgt/RHTS.ENG.NAY.REDHAT.COM.NAY.REDHAT.COM
	renew until 12/19/12 22:03:55, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 
12/19/12 22:03:56  12/20/12 22:03:54  nfs/ibm-x3550m4-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM
	renew until 12/19/12 22:03:55, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 


Updated and verified on new nfs-utils-1.2.3-35.el6:
Start rpc.gssd with -l option, do mount and work now:
[root@ibm-dx360m4-01 ~]# rpc.gssd -fvvv -l

[root@ibm-dx360m4-01 ~]# mount -t nfs ibm-x3550m4-01.rhts.eng.nay.redhat.com:/ /mnt -o sec=krb5 
[root@ibm-dx360m4-01 ~]# echo $?
0

[root@ibm-dx360m4-01 ~]# klist -e /tmp/krb5cc_machine_RHTS.ENG.NAY.REDHAT.COM 
Ticket cache: FILE:/tmp/krb5cc_machine_RHTS.ENG.NAY.REDHAT.COM
Default principal: nfs/ibm-dx360m4-01.rhts.eng.bos.redhat.com.NAY.REDHAT.COM

Valid starting     Expires            Service principal
12/19/12 22:07:38  12/20/12 22:07:38  krbtgt/RHTS.ENG.NAY.REDHAT.COM.NAY.REDHAT.COM
	renew until 12/19/12 22:07:38, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 
12/19/12 22:07:39  12/20/12 22:07:38  nfs/ibm-x3550m4-01.rhts.eng.nay.redhat.com.NAY.REDHAT.COM
	renew until 12/19/12 22:07:38, Etype (skey, tkt): des-cbc-crc, aes256-cts-hmac-sha1-96 
note:
klist shown the *l*egacy behaviour of requesting a ticket with a Single DES session key inside now.

So the function is verified, but I found rpc.gssd man page lack '-l' option explain, so filed Bug 889022 to let Steve fix it.
Comment 21 errata-xmlrpc 2013-02-21 10:55:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0468.html