Bug 1257509 - Disperse volume: df -h on a nfs mount throws Invalid argument error
Disperse volume: df -h on a nfs mount throws Invalid argument error
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: disperse (Show other bugs)
3.1
Unspecified Unspecified
unspecified Severity unspecified
: ---
: RHGS 3.1.1
Assigned To: Ashish Pandey
Bhaskarakiran
: Regression, ZStream
Depends On:
Blocks: 1251815 1260848 1261732 1273711
  Show dependency treegraph
 
Reported: 2015-08-27 04:45 EDT by Bhaskarakiran
Modified: 2016-11-23 18:11 EST (History)
11 users (show)

See Also:
Fixed In Version: glusterfs-3.7.1-15
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1260848 1273711 (view as bug list)
Environment:
Last Closed: 2015-10-05 03:24:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Bhaskarakiran 2015-08-27 04:45:21 EDT
Description of problem:
=======================

df -h on a nfs mount throws invalid argument error. This is on a disperse volume. For a distribute replicate it works properly. This is seen both on RHEL6 and RHEL7

[root@rhs-client32 ~]# mount -t nfs -o vers=3 transformers:/vol1 /mnt/nfs/
[root@rhs-client32 ~]# ll /mnt/nfs/
total 40
drwxr-xr-x.  169 root root  8192 Aug 27 10:31 dirs
drwxr-xr-x. 1002 root root 24576 Aug 26 11:50 files
drwxr-xr-x.    4 root root    38 Aug 25 11:56 linux
drwxr-xr-x.  205 root root  8192 Aug 25 16:34 renames
drwxr-xr-x.    4 root root    38 Aug 25 11:56 tarball
[root@rhs-client32 ~]# 
[root@rhs-client32 ~]# df -h /mnt/nfs/
df: ‘/mnt/nfs/’: Invalid argument

Version-Release number of selected component (if applicable):
=============================================================
3.7.1-12.el7rhgs

[root@transformers ~]# gluster --version
glusterfs 3.7.1 built on Aug 19 2015 05:46:08
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@transformers ~]#

[root@rhs-client32 ~]# rpm -qa |grep gluster
glusterfs-api-3.7.1-12.el7rhgs.x86_64
glusterfs-3.7.1-12.el7rhgs.x86_64
glusterfs-client-xlators-3.7.1-12.el7rhgs.x86_64
glusterfs-fuse-3.7.1-12.el7rhgs.x86_64
glusterfs-libs-3.7.1-12.el7rhgs.x86_64
[root@rhs-client32 ~]# 

How reproducible:
=================
100%

Steps to Reproduce:

1. Create a 8+4 disperse volume
2. NFS mount on the client
3. Do a 'df -h .' on the mount point

Actual results:
===============
Invalid argument error

Expected results:
=================
No errors


Additional info:
Comment 3 Ashish Pandey 2015-09-07 04:00:19 EDT
I tried to recreate this bug and found following - 

Call for df -h on nfs mount is going to svc_stat through server, nfs, io_stat and snap client translators. 
svc_stat validate loc->inode, which is null, and gives in valid argument error.

It looks like loc->inode is NOT getting resolved and linked by nfs which passes it to translators below that. As soon as svc_stat validates it, it throws error.

Following is the back trace - 
------------------------------------------------

[Switching to Thread 0x7fd76db68700 (LWP 27806)]

Breakpoint 1, svc_stat (frame=0x7fd778752678, this=0x7fd76803ecb0, loc=0x7fd7631934a4, xdata=0x0) at snapview-client.c:567
567	{
(gdb) p *loc
$4 = {path = 0x0, name = 0x0, inode = 0x0, parent = 0x0, gfid = '\000' <repeats 15 times>, pargfid = '\000' <repeats 15 times>}
(gdb) bt
#0  svc_stat (frame=0x7fd778752678, this=0x7fd76803ecb0, loc=0x7fd7631934a4, xdata=0x0) at snapview-client.c:567
#1  0x00007fd76c6147c9 in io_stats_stat (frame=0x7fd778752d30, this=0x7fd76803fb10, loc=0x7fd7631934a4, xdata=0x0) at io-stats.c:1947
#2  0x00007fd76c3b564c in nfs_fop_stat (nfsx=<optimized out>, xl=0x7fd76803fb10, nfu=nfu@entry=0x7fd76db67250, loc=loc@entry=0x7fd7631934a4, cbk=cbk@entry=0x7fd76c3d5ea0 <nfs3_fsstat_stat_cbk>, 
    local=local@entry=0x7fd76319306c) at nfs-fops.c:551
#3  0x00007fd76c3bd259 in nfs_stat (nfsx=<optimized out>, xl=<optimized out>, nfu=nfu@entry=0x7fd76db67250, pathloc=pathloc@entry=0x7fd7631934a4, cbk=cbk@entry=0x7fd76c3d5ea0 <nfs3_fsstat_stat_cbk>, 
    local=local@entry=0x7fd76319306c) at nfs-generics.c:63
#4  0x00007fd76c3d6089 in nfs3_fsstat_statfs_cbk (frame=<optimized out>, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=<optimized out>, buf=<optimized out>, xdata=0x0) at nfs3.c:4665
#5  0x00007fd76c3b2280 in nfs_fop_statfs_cbk (frame=0x7fd7785936d0, cookie=<optimized out>, this=<optimized out>, op_ret=<optimized out>, op_errno=<optimized out>, buf=<optimized out>, xdata=0x0)
    at nfs-fops.c:757
#6  0x00007fd76c61f930 in io_stats_statfs_cbk (frame=0x7fd77875206c, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=117, buf=0x7fd760fc43a4, xdata=0x0) at io-stats.c:1656
#7  0x00007fd76cc77f33 in dht_statfs_cbk (frame=0x7fd7787531e4, cookie=<optimized out>, this=<optimized out>, op_ret=<optimized out>, op_errno=<optimized out>, statvfs=<optimized out>, xdata=0x0)
    at dht-common.c:4343
#8  0x00007fd77ac4d4bc in default_statfs_cbk (frame=0x7fd778753ea8, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, buf=0x7fd76052f65c, xdata=0x0) at defaults.c:1080
#9  0x00007fd76ced1850 in ec_manager_statfs (fop=0x7fd760d6bcfc, state=<optimized out>) at ec-generic.c:1245
#10 0x00007fd76cecc027 in __ec_manager (fop=0x7fd760d6bcfc, error=0) at ec-common.c:1949
#11 0x00007fd76cecc178 in ec_resume (fop=0x7fd760d6bcfc, error=0) at ec-common.c:307
#12 0x00007fd76cecc2bc in ec_complete (fop=0x7fd760d6bcfc) at ec-common.c:380
#13 0x00007fd76cecf9f3 in ec_statfs_cbk (frame=<optimized out>, cookie=0x9, this=0x7fd76803aaa0, op_ret=0, op_errno=0, buf=0x7fd76db67a80, xdata=0x0) at ec-generic.c:1171
#14 0x00007fd76d134145 in client3_3_statfs_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7fd778753540) at client-rpc-fops.c:844
#15 0x00007fd77aa14a80 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7fd768146380, pollin=pollin@entry=0x7fd7680a55d0) at rpc-clnt.c:766
#16 0x00007fd77aa14d3f in rpc_clnt_notify (trans=<optimized out>, mydata=0x7fd7681463b0, event=<optimized out>, data=0x7fd7680a55d0) at rpc-clnt.c:907
#17 0x00007fd77aa10873 in rpc_transport_notify (this=this@entry=0x7fd768156000, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7fd7680a55d0) at rpc-transport.c:543
#18 0x00007fd76f830bb6 in socket_event_poll_in (this=this@entry=0x7fd768156000) at socket.c:2290
#19 0x00007fd76f833aa4 in socket_event_handler (fd=fd@entry=26, idx=idx@entry=14, data=0x7fd768156000, poll_in=1, poll_out=0, poll_err=0) at socket.c:2403
#20 0x00007fd77aca76aa in event_dispatch_epoll_handler (event=0x7fd76db67e80, event_pool=0x7fd77c380c90) at event-epoll.c:575
#21 event_dispatch_epoll_worker (data=0x7fd77c3ccc10) at event-epoll.c:678
#22 0x00007fd779aaedf5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fd7793f51ad in clone () from /lib64/libc.so.6
(gdb) fr 4
#4  0x00007fd76c3d6089 in nfs3_fsstat_statfs_cbk (frame=<optimized out>, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=<optimized out>, buf=<optimized out>, xdata=0x0) at nfs3.c:4665
4665	        ret = nfs_stat (cs->nfsx, cs->vol, &nfu, &cs->resolvedloc,
(gdb) p cs->resolvedloc
$5 = {path = 0x0, name = 0x0, inode = 0x0, parent = 0x0, gfid = '\000' <repeats 15 times>, pargfid = '\000' <repeats 15 times>}
(gdb) bt
#0  svc_stat (frame=0x7fd778752678, this=0x7fd76803ecb0, loc=0x7fd7631934a4, xdata=0x0) at snapview-client.c:567
#1  0x00007fd76c6147c9 in io_stats_stat (frame=0x7fd778752d30, this=0x7fd76803fb10, loc=0x7fd7631934a4, xdata=0x0) at io-stats.c:1947
#2  0x00007fd76c3b564c in nfs_fop_stat (nfsx=<optimized out>, xl=0x7fd76803fb10, nfu=nfu@entry=0x7fd76db67250, loc=loc@entry=0x7fd7631934a4, cbk=cbk@entry=0x7fd76c3d5ea0 <nfs3_fsstat_stat_cbk>, 
    local=local@entry=0x7fd76319306c) at nfs-fops.c:551
#3  0x00007fd76c3bd259 in nfs_stat (nfsx=<optimized out>, xl=<optimized out>, nfu=nfu@entry=0x7fd76db67250, pathloc=pathloc@entry=0x7fd7631934a4, cbk=cbk@entry=0x7fd76c3d5ea0 <nfs3_fsstat_stat_cbk>, 
    local=local@entry=0x7fd76319306c) at nfs-generics.c:63
#4  0x00007fd76c3d6089 in nfs3_fsstat_statfs_cbk (frame=<optimized out>, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=<optimized out>, buf=<optimized out>, xdata=0x0) at nfs3.c:4665
#5  0x00007fd76c3b2280 in nfs_fop_statfs_cbk (frame=0x7fd7785936d0, cookie=<optimized out>, this=<optimized out>, op_ret=<optimized out>, op_errno=<optimized out>, buf=<optimized out>, xdata=0x0)
    at nfs-fops.c:757
#6  0x00007fd76c61f930 in io_stats_statfs_cbk (frame=0x7fd77875206c, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=117, buf=0x7fd760fc43a4, xdata=0x0) at io-stats.c:1656
#7  0x00007fd76cc77f33 in dht_statfs_cbk (frame=0x7fd7787531e4, cookie=<optimized out>, this=<optimized out>, op_ret=<optimized out>, op_errno=<optimized out>, statvfs=<optimized out>, xdata=0x0)
    at dht-common.c:4343
#8  0x00007fd77ac4d4bc in default_statfs_cbk (frame=0x7fd778753ea8, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, buf=0x7fd76052f65c, xdata=0x0) at defaults.c:1080
#9  0x00007fd76ced1850 in ec_manager_statfs (fop=0x7fd760d6bcfc, state=<optimized out>) at ec-generic.c:1245
#10 0x00007fd76cecc027 in __ec_manager (fop=0x7fd760d6bcfc, error=0) at ec-common.c:1949
#11 0x00007fd76cecc178 in ec_resume (fop=0x7fd760d6bcfc, error=0) at ec-common.c:307
#12 0x00007fd76cecc2bc in ec_complete (fop=0x7fd760d6bcfc) at ec-common.c:380
#13 0x00007fd76cecf9f3 in ec_statfs_cbk (frame=<optimized out>, cookie=0x9, this=0x7fd76803aaa0, op_ret=0, op_errno=0, buf=0x7fd76db67a80, xdata=0x0) at ec-generic.c:1171
#14 0x00007fd76d134145 in client3_3_statfs_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7fd778753540) at client-rpc-fops.c:844
#15 0x00007fd77aa14a80 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7fd768146380, pollin=pollin@entry=0x7fd7680a55d0) at rpc-clnt.c:766
#16 0x00007fd77aa14d3f in rpc_clnt_notify (trans=<optimized out>, mydata=0x7fd7681463b0, event=<optimized out>, data=0x7fd7680a55d0) at rpc-clnt.c:907
#17 0x00007fd77aa10873 in rpc_transport_notify (this=this@entry=0x7fd768156000, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7fd7680a55d0) at rpc-transport.c:543
#18 0x00007fd76f830bb6 in socket_event_poll_in (this=this@entry=0x7fd768156000) at socket.c:2290
#19 0x00007fd76f833aa4 in socket_event_handler (fd=fd@entry=26, idx=idx@entry=14, data=0x7fd768156000, poll_in=1, poll_out=0, poll_err=0) at socket.c:2403
#20 0x00007fd77aca76aa in event_dispatch_epoll_handler (event=0x7fd76db67e80, event_pool=0x7fd77c380c90) at event-epoll.c:575
#21 event_dispatch_epoll_worker (data=0x7fd77c3ccc10) at event-epoll.c:678
#22 0x00007fd779aaedf5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fd7793f51ad in clone () from /lib64/libc.so.6


------------------------------------------------

nfs.log - 


[2015-09-07 06:21:20.847976] E [snapview-client.c:578:svc_stat] (-->/usr/lib64/glusterfs/3.7.1/xlator/nfs/server.so(nfs_fop_stat+0x1ec) [0x7fd76c3b564c] -->/usr/lib64/glusterfs/3.7.1/xlator/debug/io-stats.so(io_stats_stat+0x149) [0x7fd76c6147c9] -->/usr/lib64/glusterfs/3.7.1/xlator/features/snapview-client.so(+0x8da1) [0x7fd76c835da1] ) 0-gfsa-snapview-client: invalid argument: loc->inode [Invalid argument]
[2015-09-07 06:21:20.857524] W [MSGID: 112041] [nfs3.c:4626:nfs3_fsstat_stat_cbk] 0-nfs: df7cb3d4: (null) => -1 (Invalid argument) [Invalid argument]
[2015-09-07 06:21:20.857562] W [MSGID: 112199] [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: df7cb3d4, FSSTAT: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [Invalid argument]


---------------------------


I have observed that this issue is happening only when USS is on.
Comment 4 Niels de Vos 2015-09-07 05:47:48 EDT
This sounds very much like bug 1176837, not sure if those patches have been backported to RHGS yet.

Also, does this only happen with Gluster/NFS, and not with a FUSE mount?
Comment 5 Soumya Koduri 2015-09-07 08:44:51 EDT
Seems like 'cs->resolvedloc' stored by nfs xlator is wiped out by the time it gets response for the fop.

Breakpoint 1, nfs3_fsstat_resume (carg=0x7fd76319306c) at nfs3.c:4684
4684	{
(gdb) b nfs_fop_statfs
Breakpoint 4 at 0x7fd76c3b6920: file nfs-fops.c, line 767.
(gdb) c
Continuing.

Breakpoint 4, nfs_fop_statfs (nfsx=0x7fd768040970, xl=0x7fd76803fb10, nfu=nfu@entry=0x7fd76290ec70, pathloc=pathloc@entry=0x7fd7631934a4, 
    cbk=cbk@entry=0x7fd76c3d5f80 <nfs3_fsstat_statfs_cbk>, local=local@entry=0x7fd76319306c) at nfs-fops.c:767
767	{
(gdb) p (nfs3_call_state_t)local
Invalid cast.
(gdb) p (nfs3_call_state_t *)local
$10 = (nfs3_call_state_t *) 0x7fd76319306c
(gdb) p $10->resolvedloc
$11 = {path = 0x7fd754009900 "/", name = 0x7fd754009901 "", inode = 0x7fd766e8e06c, parent = 0x0, gfid = '\000' <repeats 15 times>, "\001", 
  pargfid = '\000' <repeats 15 times>}
(gdb) bt
#0  nfs_fop_statfs (nfsx=0x7fd768040970, xl=0x7fd76803fb10, nfu=nfu@entry=0x7fd76290ec70, pathloc=pathloc@entry=0x7fd7631934a4, 
    cbk=cbk@entry=0x7fd76c3d5f80 <nfs3_fsstat_statfs_cbk>, local=local@entry=0x7fd76319306c) at nfs-fops.c:767
#1  0x00007fd76c3bd589 in nfs_statfs (nfsx=<optimized out>, xl=<optimized out>, nfu=nfu@entry=0x7fd76290ec70, pathloc=pathloc@entry=0x7fd7631934a4, 
    cbk=cbk@entry=0x7fd76c3d5f80 <nfs3_fsstat_statfs_cbk>, local=local@entry=0x7fd76319306c) at nfs-generics.c:332
#2  0x00007fd76c3d6212 in nfs3_fsstat_resume (carg=0x7fd76319306c) at nfs3.c:4697
#3  0x00007fd76c3e006c in nfs3_fh_resolve_inode_done (cs=cs@entry=0x7fd76319306c, inode=inode@entry=0x7fd766e8e06c) at nfs3-helpers.c:3597
#4  0x00007fd76c3e0844 in nfs3_fh_resolve_inode (cs=0x7fd76319306c) at nfs3-helpers.c:3799
#5  0x00007fd76c3e0905 in nfs3_fh_resolve_resume (cs=cs@entry=0x7fd76319306c) at nfs3-helpers.c:3831
#6  0x00007fd76c3e0b20 in nfs3_fh_resolve_root (cs=cs@entry=0x7fd76319306c) at nfs3-helpers.c:3886
#7  0x00007fd76c3e0d71 in nfs3_fh_resolve_and_resume (cs=cs@entry=0x7fd76319306c, fh=fh@entry=0x7fd76290fae0, entry=entry@entry=0x0, 
    resum_fn=resum_fn@entry=0x7fd76c3d61a0 <nfs3_fsstat_resume>) at nfs3-helpers.c:3982
#8  0x00007fd76c3d65ae in nfs3_fsstat (req=req@entry=0x7fd76c0dc06c, fh=fh@entry=0x7fd76290fae0) at nfs3.c:4737
#9  0x00007fd76c3d687a in nfs3svc_fsstat (req=0x7fd76c0dc06c) at nfs3.c:4774
#10 0x00007fd77aa0e549 in rpcsvc_handle_rpc_call (svc=0x7fd768051830, trans=trans@entry=0x7fd75400f6d0, msg=msg@entry=0x7fd75400b360) at rpcsvc.c:703
#11 0x00007fd77aa0e7ab in rpcsvc_notify (trans=0x7fd75400f6d0, mydata=<optimized out>, event=<optimized out>, data=0x7fd75400b360) at rpcsvc.c:797
#12 0x00007fd77aa10873 in rpc_transport_notify (this=this@entry=0x7fd75400f6d0, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7fd75400b360)
    at rpc-transport.c:543
#13 0x00007fd76f830bb6 in socket_event_poll_in (this=this@entry=0x7fd75400f6d0) at socket.c:2290
#14 0x00007fd76f833aa4 in socket_event_handler (fd=fd@entry=46, idx=idx@entry=40, data=0x7fd75400f6d0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2403
#15 0x00007fd77aca76aa in event_dispatch_epoll_handler (event=0x7fd76290fe80, event_pool=0x7fd77c380c90) at event-epoll.c:575
#16 event_dispatch_epoll_worker (data=0x7fd7680b5f10) at event-epoll.c:678
#17 0x00007fd779aaedf5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fd7793f51ad in clone () from /lib64/libc.so.6
(gdb) f 5
#5  0x00007fd76c3e0905 in nfs3_fh_resolve_resume (cs=cs@entry=0x7fd76319306c) at nfs3-helpers.c:3831
3831	                ret = nfs3_fh_resolve_inode (cs);
(gdb) p cs->resolvedloc
$12 = {path = 0x7fd754009900 "/", name = 0x7fd754009901 "", inode = 0x7fd766e8e06c, parent = 0x0, gfid = '\000' <repeats 15 times>, "\001", 
  pargfid = '\000' <repeats 15 times>}
(gdb) l
3826	
3827	        if (cs->resolve_ret < 0)
3828	                goto err_resume_call;
3829	
3830	        if (!cs->resolventry)
3831	                ret = nfs3_fh_resolve_inode (cs);
3832	        else
3833	                ret = nfs3_fh_resolve_entry (cs);
3834	
3835	err_resume_call:
(gdb) p local
No symbol "local" in current context.
(gdb) n
772	        if ((!nfsx) || (!xl) || (!pathloc) || (!nfu))
(gdb) 
775	        gf_msg_trace (GF_NFS, 0, "Statfs: %s", pathloc->path);
(gdb) 
776	        nfs_fop_handle_frame_create (frame, nfsx, nfu, ret, err);
(gdb) 
777	        nfs_fop_handle_local_init (frame, nfsx, nfl, cbk, local, ret, err);
(gdb) 
779	        STACK_WIND_COOKIE (frame, nfs_fop_statfs_cbk, xl, xl,
(gdb) p frame->local
$13 = (void *) 0x7fd77af31710
(gdb) p (nfs_fop_local *)frame->local
No symbol "nfs_fop_local" in current context.
(gdb) p (nfs_fop_local_t *)frame->local
No symbol "nfs_fop_local_t" in current context.
(gdb) p (nfs_fop_local *)$13
No symbol "nfs_fop_local" in current context.
(gdb) p (struct nfs_fop_local *)$13
$14 = (struct nfs_fop_local *) 0x7fd77af31710
(gdb) p $14->proglocal
$15 = (void *) 0x7fd76319306c
(gdb) p (nfs3_call_state_t *)$15->resolvedloca
Attempt to dereference a generic pointer.
(gdb) p (nfs3_call_state_t *)$15->resolvedloc
Attempt to dereference a generic pointer.
(gdb) p (nfs3_call_state_t *)$15 
$16 = (nfs3_call_state_t *) 0x7fd76319306c
(gdb) p $16->resolvedloc
$17 = {path = 0x7fd754009900 "/", name = 0x7fd754009901 "", inode = 0x7fd766e8e06c, parent = 0x0, gfid = '\000' <repeats 15 times>, "\001", 
  pargfid = '\000' <repeats 15 times>}
(gdb) b  nfs_fop_statfs_cbk
Note: breakpoint 3 also set at pc 0x7fd76c3b2250.
Breakpoint 5 at 0x7fd76c3b2250: file nfs-fops.c, line 750.
(gdb) c



In cbk path,
Breakpoint 3, nfs_fop_statfs_cbk (frame=0x7fd77858d994, cookie=0x7fd76803fb10, this=0x7fd768040970, op_ret=0, op_errno=117, buf=0x7fd760fc539c, xdata=0x0)
    at nfs-fops.c:750
750	{
(gdb) p (struct nfs_fop_local *)frame->local
$18 = (struct nfs_fop_local *) 0x7fd77af31710
(gdb) p (nfs3_call_state_t *) $18->proglocal
$19 = (nfs3_call_state_t *) 0x7fd76319306c
(gdb) p $19->resolvedloc
$20 = {path = 0x0, name = 0x0, inode = 0x0, parent = 0x0, gfid = '\000' <repeats 15 times>, pargfid = '\000' <repeats 15 times>}
(gdb) p p $19->nfsx
No symbol "p" in current context.
(gdb) p  $19->nfsx
$21 = (xlator_t *) 0x7fd768040970
(gdb) p  $19->vol
$22 = (xlator_t *) 0x7fd76803fb10
(gdb) p &$19->resolvedoc
There is no member named resolvedoc.
(gdb) p &$19->resolvedloc
$23 = (loc_t *) 0x7fd7631934a4
(gdb) p *$
$24 = {path = 0x0, name = 0x0, inode = 0x0, parent = 0x0, gfid = '\000' <repeats 15 times>, pargfid = '\000' <repeats 15 times>}
(gdb) bt



On further investigating the code with Pranith & Ashish, we found that 
svc_statfs(..) does loc_wipe after STACK_WIND 



int32_t
svc_statfs (call_frame_t *frame, xlator_t *this, loc_t *loc,
            dict_t *xdata)
{
        xlator_t      *subvolume  = NULL;
        int32_t        ret        = -1;
        int            inode_type = -1;
        int32_t        op_ret     = -1;
        int32_t        op_errno   = EINVAL;
        gf_boolean_t   wind       = _gf_false;
        svc_private_t  *priv      = NULL;
        const char     *path      = NULL;

...
...
...
      path_len = strlen (loc->path);
        snap_len = strlen (priv->path);
        temp_loc = loc;

        if (path_len >= snap_len && inode_type == VIRTUAL_INODE) {
                path = &loc->path[path_len - snap_len];
                if (!strcmp (path, priv->path)) {
                        /*
                         * statfs call for virtual snap directory.
                         * Sent the fops to parent volume by removing
                         * virtual directory from path
                         */
                        subvolume = FIRST_CHILD (this);
                        root_loc.path = gf_strdup("/");
                        gf_uuid_clear(root_loc.gfid);
                        root_loc.gfid[15] = 1;
                        root_loc.inode = inode_ref (loc->inode->table->root);
                        temp_loc = &root_loc;
                }
        }

        STACK_WIND_TAIL (frame, subvolume, subvolume->fops->statfs,
                         temp_loc, xdata);
        if (temp_loc)
                loc_wipe (temp_loc);
....

}

Hence this issue was seen only when USS is on. This needs a fix.
Comment 9 Ashish Pandey 2015-09-10 04:29:36 EDT

Patch for upstream - 
http://review.gluster.org/#/c/12123/

Patch for downstream - 
http://review.gluster.org/#/c/12147/

Patch for rhgs 3.1 -
https://code.engineering.redhat.com/gerrit/#/c/57439/
Comment 10 Bhaskarakiran 2015-09-18 02:45:59 EDT
verified this on 3.7.1-15 and it works.


[root@rhs-client17 files]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
vertigo:/vol1   1.9T  1.6G  1.9T   1% /mnt/nfs
[root@rhs-client17 files]# 
[root@rhs-client17 files]# 
[root@rhs-client17 files]# pwd
/mnt/nfs/files
[root@rhs-client17 files]# mount |grep nfs
vertigo:/vol1 on /mnt/nfs type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.70.34.56,mountvers=3,mountport=38465,mountproto=tcp,local_lock=none,addr=10.70.34.56)
[root@rhs-client17 files]#
Comment 12 errata-xmlrpc 2015-10-05 03:24:56 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html

Note You need to log in before you can comment on or make changes to this bug.