finding the mountd and nfs are on same port, [root@centos-qa-client-3 ~]# rpcinfo -p program vers proto port 100000 2 tcp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 659 status 100024 1 tcp 662 status 100021 1 udp 48386 nlockmgr 100021 3 udp 48386 nlockmgr 100021 4 udp 48386 nlockmgr 100021 1 tcp 52811 nlockmgr 100021 3 tcp 52811 nlockmgr 100021 4 tcp 52811 nlockmgr 100005 3 tcp 38467 mountd 100005 1 tcp 38467 mountd 100003 3 tcp 38467 nfs [root@centos-qa-client-3 ~]#
git sync'd to this point [root@centos-qa-client-3 glusterfs.git]# git log commit 8da4623f2274faa9e9d88f7d30babb9ea80fb141 Author: Sachidananda Urs <sac> Date: Fri Jul 29 16:26:03 2011 +0530 Remove redundant function `cli_path_strip_trailing_slashes'. Use cli_canonicalize_path instead, to remove trailing/duplicate slashes. Change-Id: Ie76267a4fa9ee9986cb36039afad4dfff830d5f8 fuse mount happens properly but nfs mount fails, logs, [root@centos-qa-client-2 ~]# mount -t nfs -o vers=3,nolock 10.1.12.135:/dist /mnt/nfs-test2 mount: mount to NFS server '10.1.12.135' failed: RPC Error: Unable to receive. nfs.log says this, uest [2011-08-01 01:08:02.141127] E [rpcsvc.c:474:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2011-08-01 01:16:15.775840] E [rpcsvc.c:474:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request (END) Tried similar stuff from different machines and found same failures.
Looks like a issue cropped up after RPC unification. This is happening as gluster rpc auth checks for privileged ports (<1024). NFS client is binding to port >1024, due to which auth fails on server.
(In reply to comment #1) > finding the mountd and nfs are on same port, > > Filed separate bug 765040 for this.
after putting these changes in the nfs-server.vol files, volume nfs-server type nfs/server option nfs.dynamic-volumes on option rpc-auth.addr.dist.allow * option nfs3.dist.volume-id 5e7c5df9-37f0-4164-b439-08c3695ff7fc option rpc-auth.ports.insecure on option rpc-auth-allow-insecure on subvolumes dist end-volume the nfs server crashes, (gdb) bt #0 0x00000033fbc30265 in raise () from /lib64/libc.so.6 #1 0x00000033fbc31d10 in abort () from /lib64/libc.so.6 #2 0x00000033fbc296e6 in __assert_fail () from /lib64/libc.so.6 #3 0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94 #4 0x00002ac944279683 in __gf_calloc (nmemb=1, size=56, type=103) at mem-pool.c:148 #5 0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at nfs3-helpers.c:1985 #6 0x00002aaaab9c9d6f in nfs3_get_inode_queue (cs=0x2aaaad531044) at nfs3-helpers.c:2008 #7 0x00002aaaab9c9ed5 in nfs3_queue_call_state (cs=0x2aaaad531044) at nfs3-helpers.c:2059 #8 0x00002aaaab9c9f85 in __nfs3_file_open_and_resume (cs=0x2aaaad531044) at nfs3-helpers.c:2081 #9 0x00002aaaab9ca300 in nfs3_file_open_and_resume (cs=0x2aaaad531044, resume=0x2aaaab9b98b2 <nfs3_write_resume>) at nfs3-helpers.c:2143 #10 0x00002aaaab9b9a0d in nfs3_write_open_resume (carg=0x2aaaad531044) at nfs3.c:2105 #11 0x00002aaaab9cb436 in nfs3_fh_resolve_inode_done (cs=0x2aaaad531044, inode=0x2aaaabcaa0e0) at nfs3-helpers.c:2423 #12 0x00002aaaab9cd0f7 in nfs3_fh_resolve_inode (cs=0x2aaaad531044) at nfs3-helpers.c:2992 #13 0x00002aaaab9cd194 in nfs3_fh_resolve_resume (cs=0x2aaaad531044) at nfs3-helpers.c:3024 #14 0x00002aaaab9cd38e in nfs3_fh_resolve_root (cs=0x2aaaad531044) at nfs3-helpers.c:3078 #15 0x00002aaaab9cd539 in nfs3_fh_resolve_and_resume (cs=0x2aaaad531044, fh=0x7fff7b34ea00, entry=0x0, resum_fn=0x2aaaab9b999f <nfs3_write_open_resume>) at nfs3-helpers.c:3120 #16 0x00002aaaab9b9eaf in nfs3_write (req=0x2aaaabbdf044, fh=0x7fff7b34ea00, offset=0, count=65536, stable=UNSTABLE, payload=..., iobref=0x157d8990) at nfs3.c:2152 #17 0x00002aaaab9ba17a in nfs3svc_write (req=0x2aaaabbdf044) at nfs3.c:2225 #18 0x00002aaaab9ba227 in nfs3svc_write_vec (req=0x2aaaabbdf044, payload=0x2aaaabbdf0e4, payload_count=1, iobref=0x157d8990) at nfs3.c:2241 #19 0x00002ac9444be119 in rpcsvc_handle_rpc_call (svc=0x157b6490, trans=0x157d5460, msg=0x157d69a0) at rpcsvc.c:496 #20 0x00002ac9444be530 in rpcsvc_notify (trans=0x157d5460, mydata=0x157b6490, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0) at rpcsvc.c:603 #21 0x00002ac9444c515c in rpc_transport_notify (this=0x157d5460, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0) at rpc-transport.c:931 #22 0x00002aaaad325ea7 in socket_event_poll_in (this=0x157d5460) at socket.c:1676 ---Type <return> to continue, or q <return> to quit---q Quit (gdb) fr 3 #3 0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94 94 GF_ASSERT (0); (gdb) list 89 if (!(xl->mem_acct.rec)) { 90 GF_ASSERT (0); 91 } 92 93 if (type > xl->mem_acct.num_types) { 94 GF_ASSERT (0); 95 } 96 97 LOCK(&xl->mem_acct.rec[type].lock); 98 { (gdb) p xl->mem_acct.num_types $1 = 90 (gdb) p type $2 = 103 (gdb) fr 5 #5 0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at nfs3-helpers.c:1985 1985 inode_q = GF_CALLOC (1, sizeof (*inode_q), gf_nfs_mt_inode_q); (gdb)
Thanks Saurabh. As this is an unrelated bug, I have created a new bug 765046 (In reply to comment #4) > after putting these changes in the nfs-server.vol files, > > volume nfs-server > type nfs/server > option nfs.dynamic-volumes on > option rpc-auth.addr.dist.allow * > option nfs3.dist.volume-id 5e7c5df9-37f0-4164-b439-08c3695ff7fc > option rpc-auth.ports.insecure on > option rpc-auth-allow-insecure on > subvolumes dist > end-volume > > the nfs server crashes, > > (gdb) bt > #0 0x00000033fbc30265 in raise () from /lib64/libc.so.6 > #1 0x00000033fbc31d10 in abort () from /lib64/libc.so.6 > #2 0x00000033fbc296e6 in __assert_fail () from /lib64/libc.so.6 > #3 0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, > alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94 > #4 0x00002ac944279683 in __gf_calloc (nmemb=1, size=56, type=103) at > mem-pool.c:148 > #5 0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at > nfs3-helpers.c:1985 > #6 0x00002aaaab9c9d6f in nfs3_get_inode_queue (cs=0x2aaaad531044) at > nfs3-helpers.c:2008 > #7 0x00002aaaab9c9ed5 in nfs3_queue_call_state (cs=0x2aaaad531044) at > nfs3-helpers.c:2059 > #8 0x00002aaaab9c9f85 in __nfs3_file_open_and_resume (cs=0x2aaaad531044) at > nfs3-helpers.c:2081 > #9 0x00002aaaab9ca300 in nfs3_file_open_and_resume (cs=0x2aaaad531044, > resume=0x2aaaab9b98b2 <nfs3_write_resume>) > at nfs3-helpers.c:2143 > #10 0x00002aaaab9b9a0d in nfs3_write_open_resume (carg=0x2aaaad531044) at > nfs3.c:2105 > #11 0x00002aaaab9cb436 in nfs3_fh_resolve_inode_done (cs=0x2aaaad531044, > inode=0x2aaaabcaa0e0) at nfs3-helpers.c:2423 > #12 0x00002aaaab9cd0f7 in nfs3_fh_resolve_inode (cs=0x2aaaad531044) at > nfs3-helpers.c:2992 > #13 0x00002aaaab9cd194 in nfs3_fh_resolve_resume (cs=0x2aaaad531044) at > nfs3-helpers.c:3024 > #14 0x00002aaaab9cd38e in nfs3_fh_resolve_root (cs=0x2aaaad531044) at > nfs3-helpers.c:3078 > #15 0x00002aaaab9cd539 in nfs3_fh_resolve_and_resume (cs=0x2aaaad531044, > fh=0x7fff7b34ea00, entry=0x0, > resum_fn=0x2aaaab9b999f <nfs3_write_open_resume>) at nfs3-helpers.c:3120 > #16 0x00002aaaab9b9eaf in nfs3_write (req=0x2aaaabbdf044, fh=0x7fff7b34ea00, > offset=0, count=65536, stable=UNSTABLE, payload=..., > iobref=0x157d8990) at nfs3.c:2152 > #17 0x00002aaaab9ba17a in nfs3svc_write (req=0x2aaaabbdf044) at nfs3.c:2225 > #18 0x00002aaaab9ba227 in nfs3svc_write_vec (req=0x2aaaabbdf044, > payload=0x2aaaabbdf0e4, payload_count=1, iobref=0x157d8990) > at nfs3.c:2241 > #19 0x00002ac9444be119 in rpcsvc_handle_rpc_call (svc=0x157b6490, > trans=0x157d5460, msg=0x157d69a0) at rpcsvc.c:496 > #20 0x00002ac9444be530 in rpcsvc_notify (trans=0x157d5460, mydata=0x157b6490, > event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0) > at rpcsvc.c:603 > #21 0x00002ac9444c515c in rpc_transport_notify (this=0x157d5460, > event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0) > at rpc-transport.c:931 > #22 0x00002aaaad325ea7 in socket_event_poll_in (this=0x157d5460) at > socket.c:1676 > ---Type <return> to continue, or q <return> to quit---q > Quit > (gdb) fr 3 > #3 0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, > alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94 > 94 GF_ASSERT (0); > (gdb) list > 89 if (!(xl->mem_acct.rec)) { > 90 GF_ASSERT (0); > 91 } > 92 > 93 if (type > xl->mem_acct.num_types) { > 94 GF_ASSERT (0); > 95 } > 96 > 97 LOCK(&xl->mem_acct.rec[type].lock); > 98 { > (gdb) p xl->mem_acct.num_types > $1 = 90 > (gdb) p type > $2 = 103 > (gdb) fr 5 > #5 0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at > nfs3-helpers.c:1985 > 1985 inode_q = GF_CALLOC (1, sizeof (*inode_q), gf_nfs_mt_inode_q); > (gdb)
CHANGE: http://review.gluster.com/178 (Change-Id: I4e3fbfe37d6d3e8443d5b7b79faf6e364fdb87be) merged in master by Anand Avati (avati)
CHANGE: http://review.gluster.com/247 (Change-Id: Iae755d1460f2c7296fc7980b5257afcf778e7767) merged in master by Vijay Bellur (vijay)