Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 537193

Summary: Expired krb5 credentials on a v4 share for one user will cause reads from all other users to hang
Product: Red Hat Enterprise Linux 5 Reporter: Sachin Prabhu <sprabhu>
Component: kernelAssignee: Steve Dickson <steved>
Status: CLOSED WONTFIX QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: high Docs Contact:
Priority: medium    
Version: 5.4CC: aleksey, bfields, jackyc, jlayton, klaus.steinberger, rdassen, rwheeler, shshaikh, steved, tao
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-10-03 15:04:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
The backport of the first patch (commit 7c67db3a) in Comment 4
none
The backport of the second patch (commit a2b2bb88) in Comment 4
none
he backport of the first patch (commit 7c67db3a) in Comment 4 (kABI correct)
none
The backport of the first patch (commit 7c67db3a) in Comment 4 (kABI corrected) none

Description Sachin Prabhu 2009-11-12 18:41:09 UTC
Setup:
nfsv4 mount using sec=krb5i
nfsuser has file /mnt/nfsuser/a openwhen the users credentials expire.
nfsuser1 is another user attempting to read another file on the same share. The process hangs when attempting to read the file.

The process cat run by nfsuser1 first attempts to reclaim all expired leases on this server. To do this, it starts the reclaimer thread for this server.

The reclaimer thread which goes through the list of all open states on the server and attempts to renew them. In this case, it encounters the file /mnt/nfsuser/a which is owned by nfsuser(whose credentials have expired).  The reclaimer then attempts to refresh the credentials and makes an upcall to rpc.gssd. rpc.gssd encounters some error which results in a EACCESS error returned to the reclaimer. The reclaimer exits.

The process cat receives NFS4ERR_EXPIRED and the do..while() loop in nfs4_do_open() never exists  and results in the reclaimer being called continuously which in turn continously fails with the following message

Nov 10 19:24:07 dhcp211-98 rpc.gssd[2132]: ERROR: GSS-API: error in gss_acquire_cred(): The referenced credential has expired - No error
Nov 10 19:24:07 dhcp211-98 rpc.gssd[2132]: WARNING: Failed to create krb5 context for user with uid 5100 for server server.example.com


The following kernel patch to the upstream kernel works around the problem

commit a2b2bb8822c78806930bbb4d4c5bb3ae69648fd0
Author: Trond Myklebust <Trond.Myklebust>
Date:   Tue Apr 8 16:02:17 2008 -0400

  NFSv4: Attempt to use machine credentials in SETCLIENTID calls
 
  Signed-off-by: Trond Myklebust <Trond.Myklebust>

Filing this under nfs-utils as the problem appears to be caused due to rpc.gssd.

Comment 1 Sachin Prabhu 2009-11-12 18:44:18 UTC
Reproducer:

- Configure NFSv4 server with kerberos authentication. Here we have three systems. NFSv4 server, client and kerberos server. All these three systems are running RHEL-5.4. 2.6.18-164.el5

- On the nfs server and client create two users as below and set them a password.

# useradd -u 5100 nfsuser
# useradd -u 5101 nfsuser1

- On the nfs share create a two directories named "nfsuser" and "nfsuser1" which are owned by nfsuser and nfsuser1 respectively. Create two files in those directories.

Eg, if the nfs share is /nfs, then we will have /nfs/nfsuser/a and /nfs/nfsuser1/a.

- Mount the share on the client.

[root@nfsclient ~]# grep nfs4 /proc/mounts

nfsserver.example.com:/ /mnt nfs4 rw,vers=4,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=3,sec=krb5i,addr=dhcp210-136.gsslab.pnq.redhat.com 0 0

- On one console do "su - nfsuser" and on another root console, do "su - nfsuser1".

- Get a ticket valid for 1 minute for nfsuer.

[nfsuser@nfsclient ~]$ kinit -l 1m
Password for nfsuser:

- Read from nfs share.

[nfsuser@nfsclient ~]$ sleep 10000 < /mnt/nfsuser/a &
[1] 19341

- Immediately, on the other console for nfsuser1, get a ticket which is valid for one day.

[nfsuser1@nfsclient ~]$ kinit nfsuser1
Password for nfsuser1:

- Make sure that you can read the below file without any problems.

[nfsuser1@nfsclient ~]$ cat /mnt/nfsuser1/a

- Wait for five minutes or still the nfsuser's credentials expire.

[nfsuser1@nfsclient ~]$ sleep 300

- Then try to read the file again.

[nfsuser1@nfsclient ~]$ cat /mnt/nfsuser1/a

[it hangs indefinitely]

Comment 2 Sachin Prabhu 2009-11-16 13:03:52 UTC
Re-assigning this to kernel instead since the problem appears to be with the nfs client code.

Comment 4 Sachin Prabhu 2010-08-18 16:47:59 UTC
The upstream patch which fixes the issue was identified following a git bisect.

commit a2b2bb8822c78806930bbb4d4c5bb3ae69648fd0
Author: Trond Myklebust <Trond.Myklebust>
Date:   Tue Apr 8 16:02:17 2008 -0400

    NFSv4: Attempt to use machine credentials in SETCLIENTID calls

This need the following patch which adds cl_machine_cred to struct nfs_client and on first glance, any changes to this struct appears to break KABI

commit 7c67db3a8a98045744f06fcd6d8f476d9df0ba5c
Author: Trond Myklebust <Trond.Myklebust>
Date:   Mon Apr 7 20:50:11 2008 -0400

    NFSv4: Reintroduce machine creds
    
    We need to try to ensure that we always use the same credentials whenever
    we re-establish the clientid on the server. If not, the server won't
    recognise that we're the same client, and so may not allow us to recover
    state.
    
    Signed-off-by: Trond Myklebust <Trond.Myklebust>

Comment 6 RHEL Program Management 2011-06-20 21:28:08 UTC
This request was evaluated by Red Hat Product Management for inclusion in Red Hat Enterprise Linux 5.7 and Red Hat does not plan to fix this issue the currently developed update.

Contact your manager or support representative in case you need to escalate this bug.

Comment 8 Steve Dickson 2011-09-29 20:17:49 UTC
Created attachment 525636 [details]
The backport of the first patch (commit 7c67db3a) in Comment 4

Comment 9 Steve Dickson 2011-09-29 20:18:57 UTC
Created attachment 525638 [details]
The backport of the second patch (commit a2b2bb88) in Comment 4

Comment 14 Steve Dickson 2011-09-30 15:24:00 UTC
Created attachment 525791 [details]
he backport of the first patch (commit 7c67db3a) in Comment 4 (kABI correct)

Comment 15 Steve Dickson 2011-10-01 15:01:09 UTC
Created attachment 525871 [details]
The backport of the first patch (commit 7c67db3a) in Comment 4 (kABI corrected)

Comment 18 Steve Dickson 2011-10-03 15:04:24 UTC
(In reply to comment #4)
> The upstream patch which fixes the issue was identified following a git bisect.
> 
> commit a2b2bb8822c78806930bbb4d4c5bb3ae69648fd0
> Author: Trond Myklebust <Trond.Myklebust>
> Date:   Tue Apr 8 16:02:17 2008 -0400
> 
>     NFSv4: Attempt to use machine credentials in SETCLIENTID calls
> 
> This need the following patch which adds cl_machine_cred to struct nfs_client
> and on first glance, any changes to this struct appears to break KABI
> 
> commit 7c67db3a8a98045744f06fcd6d8f476d9df0ba5c
> Author: Trond Myklebust <Trond.Myklebust>
> Date:   Mon Apr 7 20:50:11 2008 -0400
> 
>     NFSv4: Reintroduce machine creds
> 
>     We need to try to ensure that we always use the same credentials whenever
>     we re-establish the clientid on the server. If not, the server won't
>     recognise that we're the same client, and so may not allow us to recover
>     state.
> 
>     Signed-off-by: Trond Myklebust <Trond.Myklebust>

Unfortunately pulling in these patches are not that straightforward. The
are depended on a fairly complicated infrastructure that does not 
exist in RHEL5.     

So I'm thinking we are going to have to live with what exists today...

Comment 19 Jacky 2012-01-17 08:16:08 UTC
(In reply to comment #6)
> This request was evaluated by Red Hat Product Management for inclusion in Red
> Hat Enterprise Linux 5.7 and Red Hat does not plan to fix this issue the
> currently developed update.
> 
> Contact your manager or support representative in case you need to escalate
> this bug.

Hi there, I can re-produce the above error in Red Hat Enterprise Linux 5.7 ...

Comment 20 Jacky 2012-01-17 08:24:02 UTC
(In reply to comment #6)
> This request was evaluated by Red Hat Product Management for inclusion in Red
> Hat Enterprise Linux 5.7 and Red Hat does not plan to fix this issue the
> currently developed update.
> 
> Contact your manager or support representative in case you need to escalate
> this bug.

Hi there, I can re-produce the above error in Red Hat Enterprise Linux 5.7 ...