Bug 950056

Summary: nfs.rpc-auth-allow does not work as expected with FQDN
Product: [Community] GlusterFS Reporter: Jeff Darcy <jdarcy>
Component: transportAssignee: Vivek Agarwal <vagarwal>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 3.4.0-betaCC: aavati, amarts, fhirtz, gluster-bugs, jdarcy, kkeithle, rgowdapp, sankarshan, saujain, sdharane, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.4.0 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 906119 Environment:
Last Closed: 2013-07-24 17:50:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 906119, 952693, 962431, 985736    

Description Jeff Darcy 2013-04-09 14:31:55 UTC 
+++ This bug was initially created as a clone of Bug #906119 +++

Though nfs.addr-namelookup is explicitly turned on, mount fails on host which is allowed as per rpc-auth-allow.

Volume Name: test_vol_quota
Type: Distributed-Replicate
Volume ID: f926f09c-67cb-44e0-a952-215f87a41c21
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: 148.86.121.206:/gluster/brick-12
Brick2: 148.86.121.207:/gluster/brick-12
Brick3: 148.86.121.208:/gluster/brick-12
Brick4: 148.86.121.209:/gluster/brick-12
Brick5: 148.86.121.210:/gluster/brick-12
Brick6: 148.86.121.211:/gluster/brick-12
Options Reconfigured:
nfs.addr-namelookup: on
nfs.rpc-auth-allow: etc001495a.etc.test.gs.com,ref532.etc.test.gs.com
features.limit-usage: /:20GB
features.quota: on

Gluster server - 

Volume Name: vivek
Type: Distributed-Replicate
Volume ID: 7299d6c8-6073-4a99-b3d4-3be48bcbc8dc
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: 148.86.121.206:/gluster/gluster1/vivek
Brick2: 148.86.121.207:/gluster/gluster1/vivek
Brick3: 148.86.121.208:/gluster/gluster1/vivek
Brick4: 148.86.121.209:/gluster/gluster1/vivek
Brick5: 148.86.121.210:/gluster/gluster1/vivek
Brick6: 148.86.121.211:/gluster/gluster1/vivek
Options Reconfigured:
nfs.rpc-auth-allow: spidey.etc.test.gs.com
nfs.addr-namelookup: on
features.limit-usage: /:5GB
features.quota: on

Client failed output - 

mkdir -p /mnt/vivek; mount -t nfs -o vers=3 gluster-unrep.dsp.services.gs.com:vivek /mnt/vivek -v
mount.nfs: timeout set for Fri Jan 25 16:50:36 2013                                                                               
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.124'                                                                 
mount.nfs: prog 100003, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.124 prog 100003 vers 3 prot TCP port 38467                                                           
mount.nfs: prog 100005, trying vers=3, prot=17                                                                                    
mount.nfs: portmap query retrying: RPC: Program not registered                                                                    
mount.nfs: prog 100005, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.124 prog 100005 vers 3 prot TCP port 38465                                                           
mount.nfs: mount(2): Permission denied                                                                                            
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.133'                                                                 
mount.nfs: prog 100003, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.133 prog 100003 vers 3 prot TCP port 38467                                                           
mount.nfs: prog 100005, trying vers=3, prot=17                                                                                    
mount.nfs: portmap query retrying: RPC: Program not registered                                                                    
mount.nfs: prog 100005, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.133 prog 100005 vers 3 prot TCP port 38465                                                           
mount.nfs: mount(2): Permission denied                                                                                            
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.125'                                                                 
mount.nfs: prog 100003, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.125 prog 100003 vers 3 prot TCP port 38467                                                           
mount.nfs: prog 100005, trying vers=3, prot=17                                                                                    
mount.nfs: portmap query retrying: RPC: Program not registered                                                                    
mount.nfs: prog 100005, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.125 prog 100005 vers 3 prot TCP port 38465                                                           
mount.nfs: mount(2): Permission denied                                                                                            
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.132'                                                                 
mount.nfs: prog 100003, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.132 prog 100003 vers 3 prot TCP port 38467                                                           
mount.nfs: prog 100005, trying vers=3, prot=17                                                                                    
mount.nfs: portmap query retrying: RPC: Program not registered                                                                    
mount.nfs: prog 100005, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.132 prog 100005 vers 3 prot TCP port 38465                                                           
mount.nfs: mount(2): Permission denied                                                                                            
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.126'                                                                 
mount.nfs: prog 100003, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.126 prog 100003 vers 3 prot TCP port 38467                                                           
mount.nfs: prog 100005, trying vers=3, prot=17                                                                                    
mount.nfs: portmap query retrying: RPC: Program not registered                                                                    
mount.nfs: prog 100005, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.126 prog 100005 vers 3 prot TCP port 38465                                                           
mount.nfs: mount(2): Permission denied                                                                                            
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.129'                                                                 
mount.nfs: prog 100003, trying vers=3, prot=6                                                                                     
mount.nfs: trying 148.86.181.129 prog 100003 vers 3 prot TCP port 38467                                                           
mount.nfs: prog 100005, trying vers=3, prot=17                                                                                    
mount.nfs: portmap query retrying: RPC: Program not registered                                                                    
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 148.86.181.129 prog 100005 vers 3 prot TCP port 38465
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.123'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 148.86.181.123 prog 100003 vers 3 prot TCP port 38467
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 148.86.181.123 prog 100005 vers 3 prot TCP port 38465
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.128'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 148.86.181.128 prog 100003 vers 3 prot TCP port 38467
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 148.86.181.128 prog 100005 vers 3 prot TCP port 38465
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.127'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 148.86.181.127 prog 100003 vers 3 prot TCP port 38467
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 148.86.181.127 prog 100005 vers 3 prot TCP port 38465
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.134'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 148.86.181.134 prog 100003 vers 3 prot TCP port 38467
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 148.86.181.134 prog 100005 vers 3 prot TCP port 38465
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.130'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 148.86.181.130 prog 100003 vers 3 prot TCP port 38467
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 148.86.181.130 prog 100005 vers 3 prot TCP port 38465
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.131'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 148.86.181.131 prog 100003 vers 3 prot TCP port 38467
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 148.86.181.131 prog 100005 vers 3 prot TCP port 38465
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting gluster-unrep.dsp.services.gs.com:vivek


Now, changing it to IP address - 

Volume Name: vivek
Type: Distributed-Replicate
Volume ID: 7299d6c8-6073-4a99-b3d4-3be48bcbc8dc
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: 148.86.121.206:/gluster/gluster1/vivek
Brick2: 148.86.121.207:/gluster/gluster1/vivek
Brick3: 148.86.121.208:/gluster/gluster1/vivek
Brick4: 148.86.121.209:/gluster/gluster1/vivek
Brick5: 148.86.121.210:/gluster/gluster1/vivek
Brick6: 148.86.121.211:/gluster/gluster1/vivek
Options Reconfigured:
nfs.rpc-auth-allow: 10.235.42.103
nfs.addr-namelookup: on
features.limit-usage: /:5GB
features.quota: on


Client success output - 

mkdir -p /mnt/vivek; mount -t nfs -o vers=3 gluster-unrep.dsp.services.gs.com:vivek /mnt/vivek -v
mount.nfs: timeout set for Fri Jan 25 16:51:46 2013
mount.nfs: trying text-based options 'vers=3,addr=148.86.181.129'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 148.86.181.129 prog 100003 vers 3 prot TCP port 38467
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 148.86.181.129 prog 100005 vers 3 prot TCP port 38465
gluster-unrep.dsp.services.gs.com:vivek on /mnt/vivek type nfs (rw,vers=3)

Client side verbose output but nothing much to talk about

#mount -t nfs -o vers=3 10.235.46.109:/vol_rep_dist /local/t/ -v
mount.nfs: timeout set for Mon Jan 28 11:06:56 2013
mount.nfs: trying text-based options 'vers=3,addr=10.235.46.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.235.46.109 prog 100003 vers 3 prot TCP port 38467
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 10.235.46.109 prog 100005 vers 3 prot TCP port 38465
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 10.235.46.109:/vol_rep_dist

Here is trace ouput from gluster nfs log  and volume configuration

[2013-01-28 09:50:25.628452] T [rpcsvc-auth.c:305:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_UNIX
[2013-01-28 09:50:25.628464] T [rpcsvc.c:382:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x2f58cbe3, Ver: 2, Program: 100005, ProgVers: 3, Proc: 1) from rpc-transport (socket.nfs-server)
[2013-01-28 09:50:25.628482] T [auth-unix.c:58:auth_unix_authenticate] 0-rpc-service: Auth Info: machine name: 691684-vm1.etc.test.gs.com, uid: 0, gid: 0
[2013-01-28 09:50:25.628494] T [rpcsvc.c:211:rpcsvc_program_actor] 0-rpc-service: Actor found: MOUNT3 - MNT
[2013-01-28 09:50:25.628521] D [mount3.c:912:mnt3svc_mnt] 0-nfs-mount: dirpath: /vol_rep_dist
[2013-01-28 09:50:25.628540] D [mount3.c:855:mnt3_find_export] 0-nfs-mount: dirpath: /vol_rep_dist
[2013-01-28 09:50:25.628552] D [mount3.c:749:mnt3_mntpath_to_export] 0-nfs-mount: Found export volume: vol_rep_dist
[2013-01-28 09:50:25.628585] I [mount3.c:787:mnt3_check_client_net] 0-nfs-mount: Peer 10.135.50.4:832  not allowed
[2013-01-28 09:50:25.628603] D [mount3.c:934:mnt3svc_mnt] 0-nfs-mount: Client mount not allowed
[2013-01-28 09:50:25.629033] T [rpcsvc.c:1050:rpcsvc_submit_generic] 0-rpc-service: Tx message: 4
[2013-01-28 09:50:25.629063] T [rpcsvc.c:676:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 28, payload: 4, rpc hdr: 24
[2013-01-28 09:50:25.629094] T [rpcsvc.c:1087:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x794348515x, Program: MOUNT3, ProgVers: 3, Proc: 1) to rpc-transport (socket.nfs-server)
[2013-01-28 09:50:25.629284] D [socket.c:184:__socket_rwv] 0-socket.nfs-server: EOF from peer 10.135.50.4:832


# gluster v i vol_rep_dist

Volume Name: vol_rep_dist
Type: Distributed-Replicate
Volume ID: d24d90d1-6531-49b7-b048-28aa2f8c90f6
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.135.94.68:/gluster_mount/disk1/share1
Brick2: 10.135.94.106:/gluster_mount/disk1/share1
Brick3: 10.135.94.76:/gluster_mount/disk1/share1
Brick4: 10.135.94.69:/gluster_mount/disk1/share1
Options Reconfigured:
diagnostics.client-sys-log-level: WARNING
diagnostics.brick-log-level: TRACE
diagnostics.client-log-level: TRACE
nfs.rpc-auth-allow: 691684-vm1.etc.test.gs.com
nfs.addr-namelookup: on
cluster.server-quorum-type: server

--- Additional comment from Frank Hirtz on 2013-01-30 16:49:33 EST ---

Created attachment 690621 [details]
sosreport

--- Additional comment from Rajesh on 2013-01-31 01:50:19 EST ---

*** Bug 903553 has been marked as a duplicate of this bug. ***

--- Additional comment from Vijay Bellur on 2013-02-18 12:34:33 EST ---

CHANGE: http://review.gluster.org/4536 (rpc: change dict key for fqdn) merged in master by Anand Avati (avati)
Comment 1 Anand Avati 2013-05-08 12:19:27 UTC 
REVIEW: http://review.gluster.org/4965 (rpc: change dict key for fqdn) posted (#4) for review on release-3.4 by Rajesh Amaravathi (rajesh)
Comment 2 Amar Tumballi 2013-05-17 05:26:36 UTC 
already part of master
Comment 3 Anand Avati 2013-05-23 16:55:03 UTC 
COMMIT: http://review.gluster.org/4965 committed in release-3.4 by Anand Avati (avati) 
------
commit ec540a0b5df87e526f77c625af7ce5f21365fb4c
Author: Rajesh Amaravathi <rajesh>
Date:   Mon Feb 18 15:51:35 2013 +0530

    rpc: change dict key for fqdn
    
    changed the key from "client.fqdn", which could be wrongly
    construed as belonging to protocol/client, to "fqdn".
    
    This is a backport of 8403f9a2d976c33e01fbd9e4a4b04e8f1e936806.
    
    Change-Id: Ib5f4a875a00b99cd903a29da19bafeb70baaab4e
    BUG: 950056
    Signed-off-by: Rajesh Amaravathi <rajesh>
    Reviewed-on: http://review.gluster.org/4536
    Reviewed-by: Niels de Vos <ndevos>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    Tested-by: Gluster Build System <jenkins.com>
    Signed-off-by: Rajesh Amaravathi <rajesh>
    Reviewed-on: http://review.gluster.org/4965
    Reviewed-by: Jeff Darcy <jdarcy>