Bug 765028 (GLUSTER-3296)
Summary: | nfs mount fails on latest git sync | ||
---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Saurabh <saurabh> |
Component: | nfs | Assignee: | Shehjar Tikoo <shehjart> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | |
Severity: | medium | Docs Contact: | |
Priority: | medium | ||
Version: | mainline | CC: | gluster-bugs, krishna, sgowda |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | Type: | --- | |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Saurabh
2011-08-01 08:03:58 UTC
git sync'd to this point [root@centos-qa-client-3 glusterfs.git]# git log commit 8da4623f2274faa9e9d88f7d30babb9ea80fb141 Author: Sachidananda Urs <sac> Date: Fri Jul 29 16:26:03 2011 +0530 Remove redundant function `cli_path_strip_trailing_slashes'. Use cli_canonicalize_path instead, to remove trailing/duplicate slashes. Change-Id: Ie76267a4fa9ee9986cb36039afad4dfff830d5f8 fuse mount happens properly but nfs mount fails, logs, [root@centos-qa-client-2 ~]# mount -t nfs -o vers=3,nolock 10.1.12.135:/dist /mnt/nfs-test2 mount: mount to NFS server '10.1.12.135' failed: RPC Error: Unable to receive. nfs.log says this, uest [2011-08-01 01:08:02.141127] E [rpcsvc.c:474:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request [2011-08-01 01:16:15.775840] E [rpcsvc.c:474:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request (END) Tried similar stuff from different machines and found same failures. Looks like a issue cropped up after RPC unification. This is happening as gluster rpc auth checks for privileged ports (<1024). NFS client is binding to port >1024, due to which auth fails on server. (In reply to comment #1) > finding the mountd and nfs are on same port, > > Filed separate bug 765040 for this. after putting these changes in the nfs-server.vol files, volume nfs-server type nfs/server option nfs.dynamic-volumes on option rpc-auth.addr.dist.allow * option nfs3.dist.volume-id 5e7c5df9-37f0-4164-b439-08c3695ff7fc option rpc-auth.ports.insecure on option rpc-auth-allow-insecure on subvolumes dist end-volume the nfs server crashes, (gdb) bt #0 0x00000033fbc30265 in raise () from /lib64/libc.so.6 #1 0x00000033fbc31d10 in abort () from /lib64/libc.so.6 #2 0x00000033fbc296e6 in __assert_fail () from /lib64/libc.so.6 #3 0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94 #4 0x00002ac944279683 in __gf_calloc (nmemb=1, size=56, type=103) at mem-pool.c:148 #5 0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at nfs3-helpers.c:1985 #6 0x00002aaaab9c9d6f in nfs3_get_inode_queue (cs=0x2aaaad531044) at nfs3-helpers.c:2008 #7 0x00002aaaab9c9ed5 in nfs3_queue_call_state (cs=0x2aaaad531044) at nfs3-helpers.c:2059 #8 0x00002aaaab9c9f85 in __nfs3_file_open_and_resume (cs=0x2aaaad531044) at nfs3-helpers.c:2081 #9 0x00002aaaab9ca300 in nfs3_file_open_and_resume (cs=0x2aaaad531044, resume=0x2aaaab9b98b2 <nfs3_write_resume>) at nfs3-helpers.c:2143 #10 0x00002aaaab9b9a0d in nfs3_write_open_resume (carg=0x2aaaad531044) at nfs3.c:2105 #11 0x00002aaaab9cb436 in nfs3_fh_resolve_inode_done (cs=0x2aaaad531044, inode=0x2aaaabcaa0e0) at nfs3-helpers.c:2423 #12 0x00002aaaab9cd0f7 in nfs3_fh_resolve_inode (cs=0x2aaaad531044) at nfs3-helpers.c:2992 #13 0x00002aaaab9cd194 in nfs3_fh_resolve_resume (cs=0x2aaaad531044) at nfs3-helpers.c:3024 #14 0x00002aaaab9cd38e in nfs3_fh_resolve_root (cs=0x2aaaad531044) at nfs3-helpers.c:3078 #15 0x00002aaaab9cd539 in nfs3_fh_resolve_and_resume (cs=0x2aaaad531044, fh=0x7fff7b34ea00, entry=0x0, resum_fn=0x2aaaab9b999f <nfs3_write_open_resume>) at nfs3-helpers.c:3120 #16 0x00002aaaab9b9eaf in nfs3_write (req=0x2aaaabbdf044, fh=0x7fff7b34ea00, offset=0, count=65536, stable=UNSTABLE, payload=..., iobref=0x157d8990) at nfs3.c:2152 #17 0x00002aaaab9ba17a in nfs3svc_write (req=0x2aaaabbdf044) at nfs3.c:2225 #18 0x00002aaaab9ba227 in nfs3svc_write_vec (req=0x2aaaabbdf044, payload=0x2aaaabbdf0e4, payload_count=1, iobref=0x157d8990) at nfs3.c:2241 #19 0x00002ac9444be119 in rpcsvc_handle_rpc_call (svc=0x157b6490, trans=0x157d5460, msg=0x157d69a0) at rpcsvc.c:496 #20 0x00002ac9444be530 in rpcsvc_notify (trans=0x157d5460, mydata=0x157b6490, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0) at rpcsvc.c:603 #21 0x00002ac9444c515c in rpc_transport_notify (this=0x157d5460, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0) at rpc-transport.c:931 #22 0x00002aaaad325ea7 in socket_event_poll_in (this=0x157d5460) at socket.c:1676 ---Type <return> to continue, or q <return> to quit---q Quit (gdb) fr 3 #3 0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94 94 GF_ASSERT (0); (gdb) list 89 if (!(xl->mem_acct.rec)) { 90 GF_ASSERT (0); 91 } 92 93 if (type > xl->mem_acct.num_types) { 94 GF_ASSERT (0); 95 } 96 97 LOCK(&xl->mem_acct.rec[type].lock); 98 { (gdb) p xl->mem_acct.num_types $1 = 90 (gdb) p type $2 = 103 (gdb) fr 5 #5 0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at nfs3-helpers.c:1985 1985 inode_q = GF_CALLOC (1, sizeof (*inode_q), gf_nfs_mt_inode_q); (gdb) Thanks Saurabh. As this is an unrelated bug, I have created a new bug 765046 (In reply to comment #4) > after putting these changes in the nfs-server.vol files, > > volume nfs-server > type nfs/server > option nfs.dynamic-volumes on > option rpc-auth.addr.dist.allow * > option nfs3.dist.volume-id 5e7c5df9-37f0-4164-b439-08c3695ff7fc > option rpc-auth.ports.insecure on > option rpc-auth-allow-insecure on > subvolumes dist > end-volume > > the nfs server crashes, > > (gdb) bt > #0 0x00000033fbc30265 in raise () from /lib64/libc.so.6 > #1 0x00000033fbc31d10 in abort () from /lib64/libc.so.6 > #2 0x00000033fbc296e6 in __assert_fail () from /lib64/libc.so.6 > #3 0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, > alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94 > #4 0x00002ac944279683 in __gf_calloc (nmemb=1, size=56, type=103) at > mem-pool.c:148 > #5 0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at > nfs3-helpers.c:1985 > #6 0x00002aaaab9c9d6f in nfs3_get_inode_queue (cs=0x2aaaad531044) at > nfs3-helpers.c:2008 > #7 0x00002aaaab9c9ed5 in nfs3_queue_call_state (cs=0x2aaaad531044) at > nfs3-helpers.c:2059 > #8 0x00002aaaab9c9f85 in __nfs3_file_open_and_resume (cs=0x2aaaad531044) at > nfs3-helpers.c:2081 > #9 0x00002aaaab9ca300 in nfs3_file_open_and_resume (cs=0x2aaaad531044, > resume=0x2aaaab9b98b2 <nfs3_write_resume>) > at nfs3-helpers.c:2143 > #10 0x00002aaaab9b9a0d in nfs3_write_open_resume (carg=0x2aaaad531044) at > nfs3.c:2105 > #11 0x00002aaaab9cb436 in nfs3_fh_resolve_inode_done (cs=0x2aaaad531044, > inode=0x2aaaabcaa0e0) at nfs3-helpers.c:2423 > #12 0x00002aaaab9cd0f7 in nfs3_fh_resolve_inode (cs=0x2aaaad531044) at > nfs3-helpers.c:2992 > #13 0x00002aaaab9cd194 in nfs3_fh_resolve_resume (cs=0x2aaaad531044) at > nfs3-helpers.c:3024 > #14 0x00002aaaab9cd38e in nfs3_fh_resolve_root (cs=0x2aaaad531044) at > nfs3-helpers.c:3078 > #15 0x00002aaaab9cd539 in nfs3_fh_resolve_and_resume (cs=0x2aaaad531044, > fh=0x7fff7b34ea00, entry=0x0, > resum_fn=0x2aaaab9b999f <nfs3_write_open_resume>) at nfs3-helpers.c:3120 > #16 0x00002aaaab9b9eaf in nfs3_write (req=0x2aaaabbdf044, fh=0x7fff7b34ea00, > offset=0, count=65536, stable=UNSTABLE, payload=..., > iobref=0x157d8990) at nfs3.c:2152 > #17 0x00002aaaab9ba17a in nfs3svc_write (req=0x2aaaabbdf044) at nfs3.c:2225 > #18 0x00002aaaab9ba227 in nfs3svc_write_vec (req=0x2aaaabbdf044, > payload=0x2aaaabbdf0e4, payload_count=1, iobref=0x157d8990) > at nfs3.c:2241 > #19 0x00002ac9444be119 in rpcsvc_handle_rpc_call (svc=0x157b6490, > trans=0x157d5460, msg=0x157d69a0) at rpcsvc.c:496 > #20 0x00002ac9444be530 in rpcsvc_notify (trans=0x157d5460, mydata=0x157b6490, > event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0) > at rpcsvc.c:603 > #21 0x00002ac9444c515c in rpc_transport_notify (this=0x157d5460, > event=RPC_TRANSPORT_MSG_RECEIVED, data=0x157d69a0) > at rpc-transport.c:931 > #22 0x00002aaaad325ea7 in socket_event_poll_in (this=0x157d5460) at > socket.c:1676 > ---Type <return> to continue, or q <return> to quit---q > Quit > (gdb) fr 3 > #3 0x00002ac9442793e6 in gf_mem_set_acct_info (xl=0x2ac9444b5080, > alloc_ptr=0x7fff7b34e610, size=56, type=103) at mem-pool.c:94 > 94 GF_ASSERT (0); > (gdb) list > 89 if (!(xl->mem_acct.rec)) { > 90 GF_ASSERT (0); > 91 } > 92 > 93 if (type > xl->mem_acct.num_types) { > 94 GF_ASSERT (0); > 95 } > 96 > 97 LOCK(&xl->mem_acct.rec[type].lock); > 98 { > (gdb) p xl->mem_acct.num_types > $1 = 90 > (gdb) p type > $2 = 103 > (gdb) fr 5 > #5 0x00002aaaab9c9c3e in __nfs3_get_inode_queue (cs=0x2aaaad531044) at > nfs3-helpers.c:1985 > 1985 inode_q = GF_CALLOC (1, sizeof (*inode_q), gf_nfs_mt_inode_q); > (gdb) CHANGE: http://review.gluster.com/178 (Change-Id: I4e3fbfe37d6d3e8443d5b7b79faf6e364fdb87be) merged in master by Anand Avati (avati) CHANGE: http://review.gluster.com/247 (Change-Id: Iae755d1460f2c7296fc7980b5257afcf778e7767) merged in master by Vijay Bellur (vijay) |