+++ This bug was initially created as a clone of Bug #1245636 +++ Description of problem: ======================= In a nfs ganesha set up,created some snapshots and activated them. cd to .snaps and any further operation on the mount point hangs thereafter. Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.7.1-10.el6rhs.x86_64 nfs-ganesha-gluster-2.2.0-5.el6rhs.x86_64 How reproducible: ================ 2/2 Steps to Reproduce: =================== 1.Created a 6x2 volume and mounted it 2.Scheduled some snapshots and activated them. 3.cd to .snaps from the mount point , it hangs . Further any operation on the mount point hangs nfs server times out for the showmount command but ganesha is still running on that server [root@rhs-client21 ~]# ps -eaf | grep ganesha root 5650 13568 0 17:27 pts/0 00:00:00 grep ganesha root 14065 1 0 Jul17 ? 00:08:59 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -p /var/run/ganesha.nfsd.pid [root@rhs-client21 ~]# [root@rhs-client21 ~]# [root@rhs-client21 ~]# [root@rhs-client21 ~]# showmount -e localhost rpc mount export: RPC: Timed out [nfs server timing out for the showmount command was reported in this bug - https://bugzilla.redhat.com/show_bug.cgi?id=1224618] I am seeing this issue while testing snapshots in nfs ganesha set up. Actual results: Expected results: Additional info: --- Additional comment from Soumya Koduri on 2015-07-30 14:51:52 CEST --- This issue is always reproducible. The issue here is that the lookup fop of snapview-client xlator 'svc_lookup' has the same name as one of the routines in libntirpc used by NFS-Ganesha. Hence while loading snapview xlator, the xlator->fops->lookup symbol incorrectly gets resolved to ntiprc 'svc_lookup'. Breakpoint 1, meta_lookup (frame=0x7f7668016520, this=0x7f76480160d0, loc=0x7f7668daea80, xdata=0x0) at meta.c:22 22 { Missing separate debuginfos, use: debuginfo-install libacl-2.2.49-6.el6.x86_64 libuuid-2.17.2-12.14.el6_5.x86_64 openssl-1.0.1e-16.el6_5.7.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 meta_lookup (frame=0x7f7668016520, this=0x7f76480160d0, loc=0x7f7668daea80, xdata=0x0) at meta.c:22 #1 0x00007f7661173a33 in syncop_lookup (subvol=0x7f76480160d0, loc=0x7f7668daea80, iatt=0x0, parent=0x0, xdata_in=0x0, xdata_out=0x0) at syncop.c:1223 #2 0x00007f76613dc6eb in glfs_first_lookup_safe (subvol=0x7f76480160d0) at glfs-resolve.c:46 #3 0x00007f76613dc787 in __glfs_first_lookup (fs=0x7f7664007f70, subvol=0x7f76480160d0) at glfs-resolve.c:63 #4 0x00007f76613dc7fb in __glfs_active_subvol (fs=0x7f7664007f70) at glfs-resolve.c:815 #5 0x00007f76613dcaff in priv_glfs_active_subvol (fs=0x7f7664007f70) at glfs-resolve.c:896 #6 0x00007f76613d3f00 in pub_glfs_chdir (fs=0x7f7664007f70, path=0x7f76613e0e56 "/") at glfs-fops.c:3468 #7 0x00007f76615ec08b in glusterfs_create_export (fsal_hdl=0x7f7664007ca0, parse_node=0x7f7664007560, err_type=0x7f7668daf210, up_ops=0x81ca00) at /home/skoduri/nfs-ganesha-new/src/FSAL/FSAL_GLUSTER/export.c:651 #8 0x000000000051a8e5 in fsal_commit (node=0x7f7664007560, link_mem=0x7f7664002ed8, self_struct=0x7f7664002c70, err_type=0x7f7668daf210) at /home/skoduri/nfs-ganesha-new/src/support/exports.c:585 #9 0x0000000000548967 in proc_block (node=0x7f7664007560, item=0x821c00, link_mem=0x7f7664002ed8, err_type=0x7f7668daf210) at /home/skoduri/nfs-ganesha-new/src/config_parsing/config_parsing.c:1377 #10 0x000000000054841f in do_block_load (blk=0x7f7664007ad0, params=0x8214c0, relax=false, param_struct=0x7f7664002e08, err_type=0x7f7668daf210) at /home/skoduri/nfs-ganesha-new/src/config_parsing/config_parsing.c:1235 #11 0x0000000000548891 in proc_block (node=0x7f7664007ad0, item=0x821ce8, link_mem=0x0, err_type=0x7f7668daf210) at /home/skoduri/nfs-ganesha-new/src/config_parsing/config_parsing.c:1361 #12 0x00000000005496d1 in load_config_from_node (tree_node=0x7f7664007ad0, conf_blk=0x821ce0, param=0x0, unique=false, err_type=0x7f7668daf210) at /home/skoduri/nfs-ganesha-new/src/config_parsing/config_parsing.c:1868 #13 0x0000000000530607 in gsh_export_addexport (args=0x7f7668daf2e0, reply=0x1dbf410, error=0x7f7668daf330) at /home/skoduri/nfs-ganesha-new/src/support/export_mgr.c:948 #14 0x00000000005430e2 in dbus_message_entrypoint (conn=0x1dbef40, msg=0x1dbf250, user_data=0x822870) at /home/skoduri/nfs-ganesha-new/src/dbus/dbus_server.c:518 #15 0x000000366061cefe in ?? () from /lib64/libdbus-1.so.3 #16 0x0000003660610b4c in dbus_connection_dispatch () from /lib64/libdbus-1.so.3 #17 0x0000003660610dd9 in ?? () from /lib64/libdbus-1.so.3 ---Type <return> to continue, or q <return> to quit---q Quit (gdb) f 1 #1 0x00007f7661173a33 in syncop_lookup (subvol=0x7f76480160d0, loc=0x7f7668daea80, iatt=0x0, parent=0x0, xdata_in=0x0, xdata_out=0x0) at syncop.c:1223 1223 SYNCOP (subvol, (&args), syncop_lookup_cbk, subvol->fops->lookup, (gdb) p subvol->children->xlator->name $2 = 0x7f76480134c0 "lvm_vol" (gdb) p subvol->children->xlator->children->xlator->name $3 = 0x7f76480131c0 "lvm_vol-snapview-client" (gdb) p subvol->children->xlator->children->xlator->fops $4 = (struct xlator_fops *) 0x7f763911c960 (gdb) p subvol->children->xlator->children->xlator->fop There is no member named fop. (gdb) p subvol->children->xlator->children->xlator->lookup There is no member named lookup. (gdb) p subvol->children->xlator->children->xlator->fops.lookup $5 = (fop_lookup_t) 0x561ec8 <svc_lookup> (gdb) p svc_lookup $6 = {svc_lookup_result_t (svc_rec_t **, svc_vers_range_t *, rpcprog_t, rpcvers_t, char *, u_int)} 0x561ec8 <svc_lookup> (gdb) p *svc_lookup $7 = {svc_lookup_result_t (svc_rec_t **, svc_vers_range_t *, rpcprog_t, rpcvers_t, char *, u_int)} 0x561ec8 <svc_lookup> (gdb) Marking these fops static fixed this issue. Fix is posted upstream -http://review.gluster.org/#/c/11805/
REVIEW: http://review.gluster.org/12008 (snapshot: Make fops static for correct resolution of symbols) posted (#1) for review on release-3.7 by soumya k (skoduri)
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.4, please open a new bug report. glusterfs-3.7.4 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/12496 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user