Bug 765028 (GLUSTER-3296)

Summary:	nfs mount fails on latest git sync
Product:	[Community] GlusterFS	Reporter:	Saurabh <saurabh>
Component:	nfs	Assignee:	Shehjar Tikoo <shehjart>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	mainline	CC:	gluster-bugs, krishna, sgowda
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Saurabh 2011-08-01 08:03:58 UTC

finding the mountd and nfs are on same port,


[root@centos-qa-client-3 ~]# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    659  status
    100024    1   tcp    662  status
    100021    1   udp  48386  nlockmgr
    100021    3   udp  48386  nlockmgr
    100021    4   udp  48386  nlockmgr
    100021    1   tcp  52811  nlockmgr
    100021    3   tcp  52811  nlockmgr
    100021    4   tcp  52811  nlockmgr
    100005    3   tcp  38467  mountd
    100005    1   tcp  38467  mountd
    100003    3   tcp  38467  nfs
[root@centos-qa-client-3 ~]#

Comment 1 Saurabh 2011-08-01 09:08:52 UTC

git sync'd to this point 


[root@centos-qa-client-3 glusterfs.git]# git log
commit 8da4623f2274faa9e9d88f7d30babb9ea80fb141
Author: Sachidananda Urs <sac>
Date:   Fri Jul 29 16:26:03 2011 +0530

    Remove redundant function `cli_path_strip_trailing_slashes'.
    
    Use cli_canonicalize_path instead, to remove trailing/duplicate slashes.
    
    Change-Id: Ie76267a4fa9ee9986cb36039afad4dfff830d5f8



fuse mount happens properly but nfs mount fails,

logs,

[root@centos-qa-client-2 ~]# mount -t nfs -o vers=3,nolock 10.1.12.135:/dist /mnt/nfs-test2
mount: mount to NFS server '10.1.12.135' failed: RPC Error: Unable to receive.


nfs.log says this,


uest
[2011-08-01 01:08:02.141127] E [rpcsvc.c:474:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
[2011-08-01 01:16:15.775840] E [rpcsvc.c:474:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
(END) 


Tried similar stuff from different machines and found same failures.

Comment 2 shishir gowda 2011-08-02 03:51:42 UTC

Looks like a issue cropped up after RPC unification.
This is happening as gluster rpc auth checks for privileged ports (<1024).
NFS client is binding to port >1024, due to which auth fails on server.

Comment 3 Krishna Srinivas 2011-08-02 06:56:01 UTC

(In reply to comment #1)
> finding the mountd and nfs are on same port,
> 
> 

Filed separate bug 765040 for this.

Comment 4 Saurabh 2011-08-03 03:05:06 UTC

after putting these changes in the nfs-server.vol files,

volume nfs-server
    type nfs/server
    option nfs.dynamic-volumes on
    option rpc-auth.addr.dist.allow *
    option nfs3.dist.volume-id 5e7c5df9-37f0-4164-b439-08c3695ff7fc
    option rpc-auth.ports.insecure on
    option rpc-auth-allow-insecure on
    subvolumes dist
end-volume

the nfs server crashes,

(gdb) bt
#0  0x00000033fbc30265 in raise () from /lib64/libc.so.6
#1  0x00000033fbc31d10 in abort () from /lib64/libc.so.6
#2  0x00000033fbc296e6 in __assert_fail () from /lib64/libc.so.6
#3  0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94
#4  0x00002ac944279683 in __gf_calloc (nmemb=1, size=56, type=103) at mem-pool.c:148
#5  0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at nfs3-helpers.c:1985
#6  0x00002aaaab9c9d6f in nfs3_get_inode_queue (cs=0x2aaaad531044) at nfs3-helpers.c:2008
#7  0x00002aaaab9c9ed5 in nfs3_queue_call_state (cs=0x2aaaad531044) at nfs3-helpers.c:2059
#8  0x00002aaaab9c9f85 in __nfs3_file_open_and_resume (cs=0x2aaaad531044) at nfs3-helpers.c:2081
#9  0x00002aaaab9ca300 in nfs3_file_open_and_resume (cs=0x2aaaad531044, resume=0x2aaaab9b98b2 <nfs3_write_resume>)
    at nfs3-helpers.c:2143
#10 0x00002aaaab9b9a0d in nfs3_write_open_resume (carg=0x2aaaad531044) at nfs3.c:2105
#11 0x00002aaaab9cb436 in nfs3_fh_resolve_inode_done (cs=0x2aaaad531044, inode=0x2aaaabcaa0e0) at nfs3-helpers.c:2423
#12 0x00002aaaab9cd0f7 in nfs3_fh_resolve_inode (cs=0x2aaaad531044) at nfs3-helpers.c:2992
#13 0x00002aaaab9cd194 in nfs3_fh_resolve_resume (cs=0x2aaaad531044) at nfs3-helpers.c:3024
#14 0x00002aaaab9cd38e in nfs3_fh_resolve_root (cs=0x2aaaad531044) at nfs3-helpers.c:3078
#15 0x00002aaaab9cd539 in nfs3_fh_resolve_and_resume (cs=0x2aaaad531044, fh=0x7fff7b34ea00, entry=0x0, 
    resum_fn=0x2aaaab9b999f <nfs3_write_open_resume>) at nfs3-helpers.c:3120
#16 0x00002aaaab9b9eaf in nfs3_write (req=0x2aaaabbdf044, fh=0x7fff7b34ea00, offset=0, count=65536, stable=UNSTABLE, payload=..., 
    iobref=0x157d8990) at nfs3.c:2152
#17 0x00002aaaab9ba17a in nfs3svc_write (req=0x2aaaabbdf044) at nfs3.c:2225
#18 0x00002aaaab9ba227 in nfs3svc_write_vec (req=0x2aaaabbdf044, payload=0x2aaaabbdf0e4, payload_count=1, iobref=0x157d8990)
    at nfs3.c:2241
#19 0x00002ac9444be119 in rpcsvc_handle_rpc_call (svc=0x157b6490, trans=0x157d5460, msg=0x157d69a0) at rpcsvc.c:496
#20 0x00002ac9444be530 in rpcsvc_notify (trans=0x157d5460, mydata=0x157b6490, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0)
    at rpcsvc.c:603
#21 0x00002ac9444c515c in rpc_transport_notify (this=0x157d5460, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0)
    at rpc-transport.c:931
#22 0x00002aaaad325ea7 in socket_event_poll_in (this=0x157d5460) at socket.c:1676
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) fr 3
#3  0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94
94	                GF_ASSERT (0);
(gdb) list
89	        if (!(xl->mem_acct.rec)) {
90	                GF_ASSERT (0);
91	        }
92	
93	        if (type > xl->mem_acct.num_types) {
94	                GF_ASSERT (0);
95	        }
96	
97	        LOCK(&xl->mem_acct.rec[type].lock);
98	        {
(gdb) p xl->mem_acct.num_types
$1 = 90
(gdb) p type
$2 = 103
(gdb) fr 5
#5  0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at nfs3-helpers.c:1985
1985	        inode_q = GF_CALLOC (1, sizeof (*inode_q), gf_nfs_mt_inode_q);
(gdb)

Comment 5 Krishna Srinivas 2011-08-03 03:17:08 UTC

Thanks Saurabh. As this is an unrelated bug, I have created a new bug 765046

(In reply to comment #4)
> after putting these changes in the nfs-server.vol files,
> 
> volume nfs-server
>     type nfs/server
>     option nfs.dynamic-volumes on
>     option rpc-auth.addr.dist.allow *
>     option nfs3.dist.volume-id 5e7c5df9-37f0-4164-b439-08c3695ff7fc
>     option rpc-auth.ports.insecure on
>     option rpc-auth-allow-insecure on
>     subvolumes dist
> end-volume
> 
> the nfs server crashes,
> 
> (gdb) bt
> #0  0x00000033fbc30265 in raise () from /lib64/libc.so.6
> #1  0x00000033fbc31d10 in abort () from /lib64/libc.so.6
> #2  0x00000033fbc296e6 in __assert_fail () from /lib64/libc.so.6
> #3  0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080,
> alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94
> #4  0x00002ac944279683 in __gf_calloc (nmemb=1, size=56, type=103) at
> mem-pool.c:148
> #5  0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at
> nfs3-helpers.c:1985
> #6  0x00002aaaab9c9d6f in nfs3_get_inode_queue (cs=0x2aaaad531044) at
> nfs3-helpers.c:2008
> #7  0x00002aaaab9c9ed5 in nfs3_queue_call_state (cs=0x2aaaad531044) at
> nfs3-helpers.c:2059
> #8  0x00002aaaab9c9f85 in __nfs3_file_open_and_resume (cs=0x2aaaad531044) at
> nfs3-helpers.c:2081
> #9  0x00002aaaab9ca300 in nfs3_file_open_and_resume (cs=0x2aaaad531044,
> resume=0x2aaaab9b98b2 <nfs3_write_resume>)
>     at nfs3-helpers.c:2143
> #10 0x00002aaaab9b9a0d in nfs3_write_open_resume (carg=0x2aaaad531044) at
> nfs3.c:2105
> #11 0x00002aaaab9cb436 in nfs3_fh_resolve_inode_done (cs=0x2aaaad531044,
> inode=0x2aaaabcaa0e0) at nfs3-helpers.c:2423
> #12 0x00002aaaab9cd0f7 in nfs3_fh_resolve_inode (cs=0x2aaaad531044) at
> nfs3-helpers.c:2992
> #13 0x00002aaaab9cd194 in nfs3_fh_resolve_resume (cs=0x2aaaad531044) at
> nfs3-helpers.c:3024
> #14 0x00002aaaab9cd38e in nfs3_fh_resolve_root (cs=0x2aaaad531044) at
> nfs3-helpers.c:3078
> #15 0x00002aaaab9cd539 in nfs3_fh_resolve_and_resume (cs=0x2aaaad531044,
> fh=0x7fff7b34ea00, entry=0x0, 
>     resum_fn=0x2aaaab9b999f <nfs3_write_open_resume>) at nfs3-helpers.c:3120
> #16 0x00002aaaab9b9eaf in nfs3_write (req=0x2aaaabbdf044, fh=0x7fff7b34ea00,
> offset=0, count=65536, stable=UNSTABLE, payload=..., 
>     iobref=0x157d8990) at nfs3.c:2152
> #17 0x00002aaaab9ba17a in nfs3svc_write (req=0x2aaaabbdf044) at nfs3.c:2225
> #18 0x00002aaaab9ba227 in nfs3svc_write_vec (req=0x2aaaabbdf044,
> payload=0x2aaaabbdf0e4, payload_count=1, iobref=0x157d8990)
>     at nfs3.c:2241
> #19 0x00002ac9444be119 in rpcsvc_handle_rpc_call (svc=0x157b6490,
> trans=0x157d5460, msg=0x157d69a0) at rpcsvc.c:496
> #20 0x00002ac9444be530 in rpcsvc_notify (trans=0x157d5460, mydata=0x157b6490,
> event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0)
>     at rpcsvc.c:603
> #21 0x00002ac9444c515c in rpc_transport_notify (this=0x157d5460,
> event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0)
>     at rpc-transport.c:931
> #22 0x00002aaaad325ea7 in socket_event_poll_in (this=0x157d5460) at
> socket.c:1676
> ---Type <return> to continue, or q <return> to quit---q
> Quit
> (gdb) fr 3
> #3  0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080,
> alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94
> 94                    GF_ASSERT (0);
> (gdb) list
> 89            if (!(xl->mem_acct.rec)) {
> 90                    GF_ASSERT (0);
> 91            }
> 92    
> 93            if (type > xl->mem_acct.num_types) {
> 94                    GF_ASSERT (0);
> 95            }
> 96    
> 97            LOCK(&xl->mem_acct.rec[type].lock);
> 98            {
> (gdb) p xl->mem_acct.num_types
> $1 = 90
> (gdb) p type
> $2 = 103
> (gdb) fr 5
> #5  0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at
> nfs3-helpers.c:1985
> 1985            inode_q = GF_CALLOC (1, sizeof (*inode_q), gf_nfs_mt_inode_q);
> (gdb)

Comment 6 Anand Avati 2011-08-08 03:31:15 UTC

CHANGE: http://review.gluster.com/178 (Change-Id: I4e3fbfe37d6d3e8443d5b7b79faf6e364fdb87be) merged in master by Anand Avati (avati)

Comment 7 Anand Avati 2011-08-18 09:12:51 UTC

CHANGE: http://review.gluster.com/247 (Change-Id: Iae755d1460f2c7296fc7980b5257afcf778e7767) merged in master by Vijay Bellur (vijay)