Bug 1732706

Summary: [RGW-NFS]: nfs-ganesha aborts due to "Cannot acquire credentials for principal nfs"
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Tejas <tchandra>
Component: RGWAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: CLOSED ERRATA QA Contact: Rachana Patel <racpatel>
Severity: urgent Docs Contact:
Priority: high    
Version: 3.3CC: assingh, cbodley, ceph-eng-bugs, dang, ivancich, kbader, mbenjamin, racpatel, rgowdege, sweil, tserlin, vashastr
Target Milestone: rcKeywords: Regression
Target Release: 3.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: nfs-ganesha-2.7.4-10.el7cp, ceph-12.2.12-45.el7cp Ubuntu: nfs-ganesha_2.7.4-10redhat1xenial, ceph_12.2.12-42redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-21 15:11:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ganesha.log from last crash (I think it has an extra startup at the beginning)
none
re-run with CACHE_INODE = FULL_DEBUG; none

Description Tejas 2019-07-24 07:49:19 UTC 
Description of problem:
   After startup in 3.3 , nfs ganesha daemon goes down with any command.
ceph version 12.2.12-40.el7cp

Steps to Reproduce:
1. Bring up ganesha on 3.3
2. Mount with v3 or v4, try a ls command on mount point
3. The nfs-ganesha daemon goes down

Additional info:

a]# mkdir test1
mkdir: cannot create directory ‘test1’: Input/output error


Ganesha logs:
24/07/2019 04:29:26 : epoch 5d37de9f : magna009 : ganesha.nfsd-19144[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
24/07/2019 04:29:26 : epoch 5d37de9f : magna009 : ganesha.nfsd-19144[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
24/07/2019 04:29:26 : epoch 5d37de9f : magna009 : ganesha.nfsd-19144[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
24/07/2019 04:29:26 : epoch 5d37de9f : magna009 : ganesha.nfsd-19144[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (2:2)
Comment 8 Matt Benjamin (redhat) 2019-08-01 14:04:47 UTC 
Created attachment 1597470 [details]
ganesha.log from last crash (I think it has an extra startup at the beginning)
Comment 9 Matt Benjamin (redhat) 2019-08-01 14:10:35 UTC 
Created attachment 1597472 [details]
re-run with CACHE_INODE = FULL_DEBUG;
Comment 10 Daniel Gryniewicz 2019-08-01 14:26:37 UTC 
This is https://github.com/nfs-ganesha/nfs-ganesha/commit/0beeae7c76dbf43ae388728ddd6e8065fad4c207

I'll backport it.
Comment 11 Daniel Gryniewicz 2019-08-01 14:59:02 UTC 
This has been backported.
Comment 13 Giridhar Ramaraju 2019-08-05 13:10:01 UTC 
Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate. 

Regards,
Giri
Comment 14 Giridhar Ramaraju 2019-08-05 13:11:10 UTC 
Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate. 

Regards,
Giri
Comment 45 errata-xmlrpc 2019-08-21 15:11:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2538
Comment 46 Tejas 2019-09-04 08:10:15 UTC 
*** Bug 1732788 has been marked as a duplicate of this bug. ***