Bug 1466109 - [GSS] idmapper is squashing uid to some random uid in nfs-ganesha
[GSS] idmapper is squashing uid to some random uid in nfs-ganesha
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: nfs-ganesha (Show other bugs)
All All
medium Severity medium
: ---
: ---
Assigned To: Kaleb KEITHLEY
Manisha Saini
Depends On:
  Show dependency treegraph
Reported: 2017-06-29 01:07 EDT by Riyas Abdulrasak
Modified: 2018-02-09 04:52 EST (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2018-02-09 04:52:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Riyas Abdulrasak 2017-06-29 01:07:29 EDT
Description of problem:

idmapper is not properly mapping uids when nfsv4(nfs-ganesh) is used in RHGS 3.2. 

The "Domain" mentioned in idmap.conf is same on both the client and the server. 

Version-Release number of selected component (if applicable):

RHGS 3.2

How reproducible:

Reproducible at customer environment

Steps to Reproduce:
1. Setup a nfs-ganesha cluster
2. Kerberize nfs-ganesha 
3. Mount the share on client and create files using a domain user

Actual results:

The uids getting squashed to an unknown uid

Expected results:

uids should be mapped correctly. 

Additional info:

- We have collected NFS ganesha debug logs, configuration files and the tcpdump. I will attach these to bz. 

- A snip from the logs 

29/06/2017 11:30:04 : epoch 54100000 : server.example.com : ganesha.nfsd-86527[work-46] nfs_req_creds :ID MAPPER :WARN :Could not map principal user1@example.com to uid
29/06/2017 11:30:04 : epoch 54100000 : server.example.com : ganesha.nfsd-86527[work-46] xdr_encode_nfs4_princ :ID MAPPER :INFO :nfs4_uid_to_name failed with code -2.
29/06/2017 11:30:04 : epoch 54100000 : server.example.com : ganesha.nfsd-86527[work-46] xdr_encode_nfs4_princ :ID MAPPER :INFO :Lookup for -2 failed, using numeric owner
29/06/2017 11:30:04 : epoch 54100000 : server.example.com : ganesha.nfsd-86527[work-46] xdr_encode_nfs4_princ :ID MAPPER :INFO :nfs4_gid_to_name failed with code -2

- getent passwd user is properly giving uid from the client and the server. 

- "rpc.idmapd -f -vv" gives the same  domain name on both the nodes.
Comment 11 Frank Filz 2017-06-30 09:21:06 EDT
What are the respective idmapd.conf files? These issues are almost always something not right in idmapd.conf.

Is idmapd running on both client and server?

Any error logs from idmapd?

If absolutely necessary we can turn on Ganesha logging for idmapping, and also collect tcpdump of some simple operations, but most likely we can solve the problem by having the idmapd.conf files.
Comment 16 Frank Filz 2017-07-05 13:10:33 EDT
Hmm, I can't tell what credentials are in the kerberos ticket. That would be another thing to check since the ownership of a created file depends on the credentials the request comes in with (so that's actually before idmapping).
Comment 26 Daniel Gryniewicz 2017-07-24 09:55:21 EDT
use_getpwnam is false, so principal2uid() will always fail.  Try setting UseGetpwnam to true?
Comment 27 Daniel Gryniewicz 2017-07-24 09:57:26 EDT
Hmmm... I think that check should only be used if NFSIDMAP isn't in use, but it appears to be in use all the time.
Comment 37 Daniel Gryniewicz 2017-08-02 10:02:23 EDT
Did the customer try all settings being the same case?
Comment 38 Kaleb KEITHLEY 2017-08-16 08:50:24 EDT
need reproducer, unable to reproduce locally.
Comment 39 Prerna Sony 2017-09-08 06:18:00 EDT
@Kaleb, I am trying to reproduce this issue , but facing problems in creating this environment only. 

Do we need any further info/logs from customer?
Since it is getting very delayed, can we go on remote session with customer for live debugging if customer agrees , sometimes early next week?
Comment 40 Kaleb KEITHLEY 2017-09-08 07:58:27 EDT
I can join a call if necessary. Not sure I can add anything atm though.

Mike Flannery and John Call have been exchanging emails with Frank Filz about idmapping.

You need to install and run idmapd on all the systems and use the same config on the server and all the clients.

Maybe Mike can shed some light on the subject after his recent experiments?
Comment 41 John Call 2017-09-13 14:11:10 EDT
Hi Kaleb,

not sure if this will help in this case, but it helped in my recent customer engagement...  My main problem was identifying what names needed to be mapped, and what format the names were coming in on...  I used journalctl to observe usernames that require mapping via # journalctl -l | grep "idmap" OR # nfsidmap -l

Then I configured /etc/idmapd.conf similarly on both client and server

Use # nfsidmap -c to clear the client mapping cache after making changes to /etc/idmapd.conf, or reboot.

Use # systemctl restart nfs-ganesha to update mappings on the server

Static mappings and ACLs can work around unknown UIDs -- See appendix

[root@client ~]# hostname

[root@client ~]# id mflanner
uid=1001(mflanner) gid=1001(mflanner) groups=1001(mflanner),190(systemd-journal)

[root@client ~]# egrep -v '^$|^#' /etc/idmapd.conf
Domain = cluster.net
Local-Realms = msft.cluster.net
Nobody-User = myuser
Nobody-Group = mygroup
Method = nsswitch,static
MSFT\gclark@cluster.net = mflanner
Comment 46 Kaleb KEITHLEY 2018-02-06 05:38:54 EST
No, I don't have any update. I was not able to reproduce it, and neither were Jiffin or Frank.

Reading through all the comments Frank seems fairly certain it must be some misconfiguration of K5 or uidmapper.

Frank, did we get everything we need to be able to diagnose this?

Note You need to log in before you can comment on or make changes to this bug.