Bug 730160

Summary:	rpc.idmapd: name does not map into domain
Product:	Red Hat Enterprise Linux 6	Reporter:	Dimitri Maziuk <dmaziuk>
Component:	nfs-utils	Assignee:	Steve Dickson <steved>
Status:	CLOSED CURRENTRELEASE	QA Contact:	yanfu,wang <yanwang>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	6.3	CC:	bfields, sadekok, tom
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-03-08 17:05:38 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dimitri Maziuk 2011-08-12 00:04:38 UTC

Description of problem:

I guess this is a related to bug 715430, however, in this case the ownership is remapped to nobody:nobody. As this is a CVS repo, nobody can actually use it. The log has

... rpc.idmapd[1301]: nss_getpwnam: name 'root' does not map into domain 'bmrb.wisc.edu'
... rpc.idmapd[1301]: nss_getpwnam: name 'nobody' does not map into domain 'bmrb.wisc.edu'

(mounted off a NetApp with no root squash.)

Version-Release number of selected component (if applicable):

nfs-utils-1.2.2-7

How reproducible:

always
  
Actual results:

ls -n /mnt/koi/cvs/software/  
total 976                                          
drwxrwxr-x  8 99 99   4096 Nov  4  2002 adit-nmr   
drwxrwxr-x 28 99 99   4096 Dec 22  2005 adit-nmr-prog
...


Expected results:

After disabling nfs v.4 server-side:

ls -l /cvsmain
total 976                         
drwxrwxr-x  8 340 102 4096 Nov  4  2002 adit-nmr
drwxrwxr-x 28 340 102 4096 Dec 22  2005 adit-nmr-prog
...

Additional info:

accounts are in OpenLDAP, I used "ls -n" above to eliminate ldap lookups. 99:99 is the nobody:nobody in my /etc/group & /etc/passwd.

Comment 2 Steve Dickson 2011-08-16 11:42:55 UTC

Question, what is the Method variable set to in the [Translation] 
section of /etc/idmapd.conf. The default is nsswitch which means
uids and gids will be looked up in /etc/password and /etc/groups. 

To have rpc.idmapd do LDAP lookups set to Method=umich_ldap

Comment 3 Dimitri Maziuk 2011-08-16 21:37:11 UTC

It is the default. LDAP is the reason I used "ls -n" instead of "ls -l", it has nothing to do with idmapd.conf.

Comment 4 Steve Dickson 2011-08-17 13:08:10 UTC

(In reply to comment #3)
> It is the default. LDAP is the reason I used "ls -n" instead of "ls -l", it has
> nothing to do with idmapd.conf.

Ok.. Could you please set Verbosity = 3 in the [General] section
of the /etc/idmapd.conf file and then restart the rpc.idmapd daemon
with 'service rpcidmapd restart" command.

Then using the dmesg command, please post or attach the debug
logs from rpc.idmapd.

Comment 6 Dimitri Maziuk 2011-08-17 18:46:42 UTC

root@squid log]# service rpcidmapd restart
Stopping RPC idmapd:                                       [  OK  ]
Starting RPC idmapd: rpc.idmapd: libnfsidmap: using (default) domain: bmrb.wisc.edu

rpc.idmapd: libnfsidmap: loaded plugin /usr/lib64/libnfsidmap/nsswitch.so for method nsswitch

                                                           [  OK  ]
[root@squid log]# mount koi-internal:/vol/cvs/software /mnt
[root@squid log]# ls -n /mnt/
total 976                                 
drwxrwxr-x  8 99 99   4096 Nov  4  2002 adit-nmr
drwxrwxr-x 28 99 99   4096 Dec 22  2005 adit-nmr-prog
...

-----------------------
/var/log/messages:
------------------------

Aug 17 13:35:12 squid rpc.idmapd[6768]: libnfsidmap: using (default) domain: bmrb.wisc.edu
Aug 17 13:35:12 squid rpc.idmapd[6768]: libnfsidmap: loaded plugin /usr/lib64/libnfsidmap/nsswitch.so for method nsswitch
Aug 17 13:35:12 squid rpc.idmapd[6769]: Expiration time is 600 seconds.
Aug 17 13:35:12 squid rpc.idmapd[6769]: Opened /proc/net/rpc/nfs4.nametoid/channel
Aug 17 13:35:12 squid rpc.idmapd[6769]: Opened /proc/net/rpc/nfs4.idtoname/channel
Aug 17 13:35:24 squid rpc.idmapd[6769]: New client: d
Aug 17 13:35:24 squid rpc.idmapd[6769]: Opened /var/lib/nfs/rpc_pipefs//nfs/clntd/idmap
Aug 17 13:35:24 squid rpc.idmapd[6769]: New client: e
Aug 17 13:35:24 squid rpc.idmapd[6769]: nss_getpwnam: name 'root' does not map into domain 'bmrb.wisc.edu'
Aug 17 13:35:24 squid rpc.idmapd[6769]: Client d: (user) name "root" -> id "99"
Aug 17 13:35:24 squid rpc.idmapd[6769]: Client d: (group) name "nobody" -> id "99"
Aug 17 13:35:24 squid rpc.idmapd[6769]: New client: f
Aug 17 13:35:24 squid rpc.idmapd[6769]: nss_getpwnam: name 'nobody' does not map into domain 'bmrb.wisc.edu'
Aug 17 13:35:24 squid rpc.idmapd[6769]: Client d: (user) name "nobody" -> id "99"
Aug 17 13:35:24 squid rpc.idmapd[6769]: Stale client: e
Aug 17 13:35:24 squid rpc.idmapd[6769]: #011-> closed /var/lib/nfs/rpc_pipefs//nfs/clnte/idmap

-----------------------
Furthermore, I took a closer look at netapp config and added
 options nfs.v4.id.domain bmrb.wisc.edu

That got rid of the "root" lookup, here's the log:
-----------------------

Aug 17 13:42:31 squid rpc.idmapd[6769]: New client: 19
Aug 17 13:42:31 squid rpc.idmapd[6769]: Client 18: (user) name "root.edu" -> id "0"
Aug 17 13:42:31 squid rpc.idmapd[6769]: Client 18: (group) name "nobody" -> id "99"
Aug 17 13:42:31 squid rpc.idmapd[6769]: New client: 1a
Aug 17 13:42:31 squid rpc.idmapd[6769]: nss_getpwnam: name 'nobody' does not map into domain 'bmrb.wisc.edu'
Aug 17 13:42:31 squid rpc.idmapd[6769]: Client 18: (user) name "nobody" -> id "99"
Aug 17 13:42:31 squid rpc.idmapd[6769]: Stale client: 19
Aug 17 13:42:31 squid rpc.idmapd[6769]: #011-> closed /var/lib/nfs/rpc_pipefs//nfs/clnt19/idmap

Comment 8 Steve Dickson 2012-02-29 16:06:28 UTC

(In reply to comment #6)
> -----------------------
> /var/log/messages:
> ------------------------
> 
> Aug 17 13:35:12 squid rpc.idmapd[6768]: libnfsidmap: using (default) domain:
> bmrb.wisc.edu
>
> Aug 17 13:35:24 squid rpc.idmapd[6769]: nss_getpwnam: name 'root'
> does not map into domain 'bmrb.wisc.edu'
>
> Aug 17 13:35:24 squid rpc.idmapd[6769]: nss_getpwnam: name 'nobody' does not
> map into domain 'bmrb.wisc.edu'
The problem is the client and server are in different DNS domians.
Added a "Domain = netapp.com" in the /etc/idmapd.conf on the client.
This should fix the problem.


>
> 
> -----------------------
> Furthermore, I took a closer look at netapp config and added
>  options nfs.v4.id.domain bmrb.wisc.edu
> 
> That got rid of the "root" lookup, here's the log:
> -----------------------
> 
> Aug 17 13:42:31 squid rpc.idmapd[6769]: nss_getpwnam: name 'nobody' does not
> map into domain 'bmrb.wisc.edu'
> Aug 17 13:42:31 squid rpc.idmapd[6769]: Client 18: (user) name "nobody" -> id
> "99"
>
Actually you probably want to back this change out so you do get 
the @netapp.com domain part.

Comment 9 Dimitri Maziuk 2012-02-29 18:10:18 UTC

(In reply to comment #8)

> The problem is the client and server are in different DNS domians.
> Added a "Domain = netapp.com" in the /etc/idmapd.conf on the client.
> This should fix the problem.

I think you misunderstand. NetApp is one of the top storage hardware companies, I don't own it nor their DNS domain. We at bmrb.wisc.edu *use* one of their filers. (I should think if it was in a different dns domain from our entire university, maybe we'd have noticed an nfs problem long ago already.)

I think it's pretty clear from #6 that NetApp's ONTAP 7.3.6 software is incorrectly sending "netapp.com" by default instead of its DNS domain for nfsv4 id.

The problem with idmapd, though, is that it quietly mounts the filesystem with unusable ownership and permissions. It should either fail to mount or fall back on v3 which has worked properly ever since we got that NetApp. Whether that can be fixed, or has to be left alone and called a "feature", is another question.

Comment 10 Steve Dickson 2012-02-29 18:40:47 UTC

(In reply to comment #9)
> (In reply to comment #8)
> 
> > The problem is the client and server are in different DNS domians.
> > Added a "Domain = netapp.com" in the /etc/idmapd.conf on the client.
> > This should fix the problem.
> 
> I think you misunderstand. NetApp is one of the top storage hardware companies,
> I don't own it nor their DNS domain. We at bmrb.wisc.edu *use* one of their
> filers. (I should think if it was in a different dns domain from our entire
> university, maybe we'd have noticed an nfs problem long ago already.)
> 
> I think it's pretty clear from #6 that NetApp's ONTAP 7.3.6 software is
> incorrectly sending "netapp.com" by default instead of its DNS domain for nfsv4
> id.
Understood... but if added the "Domain = netapp.com" string to the /etc/idmapd.conf file, are the ids mapped properly?

> 
> The problem with idmapd, though, is that it quietly mounts the filesystem with
> unusable ownership and permissions. It should either fail to mount or fall back
> on v3 which has worked properly ever since we got that NetApp. Whether that can
> be fixed, or has to be left alone and called a "feature", is another question.

Newer filers (ONTAP 8.3 I believe) and RHEL6.3 clients will support a notion of unmapped ids. Where the client will actually send the ASCII numeric string of the id (aka "3606") instead of the "user@domain" string. We are hoping this will help with the pain you are experiencing....

Comment 11 Dimitri Maziuk 2012-02-29 19:11:57 UTC

(In reply to comment #10)

> Understood... but if added the "Domain = netapp.com" string to the
> /etc/idmapd.conf file, are the ids mapped properly?

I didn't try because once the verbose logs pointed me in the right direction, I found "options nfs.v4.id.domain" in netapp config and fixed it that way.

> Newer filers (ONTAP 8.3 I believe) and RHEL6.3 clients will support a notion of
> unmapped ids. Where the client will actually send the ASCII numeric string of
> the id (aka "3606") instead of the "user@domain" string. We are hoping this
> will help with the pain you are experiencing....

The real pain is that 8.x series s/w is not available for low-end filers and we have one of those. Personally I'm hoping ceph and btrfs soon let me forget everything I ever knew about netapp and nfs v4 both. ;)