+++ This bug was initially created as a clone of Bug #1460514 +++ +++ This bug was initially created as a clone of Bug #1457183 +++ Description of problem: ----------------------- 4 node cluster,4 clients accessing the export via v4. Kill NFS-Ganesha on any node to simulate failover. Ganesha crashes in a minute when failover is about to get over. Try restarting Ganesha to simulate failback. Ganesha dumps the same core again. (gdb) bt #0 0x00007fb042dac8a0 in glusterfs_normalize_dentry () from /lib64/libglusterfs.so.0 #1 0x00007fb04307eac8 in glfs_resolve_at () from /lib64/libgfapi.so.0 #2 0x00007fb0430805c4 in glfs_h_lookupat () from /lib64/libgfapi.so.0 #3 0x00007fb04349d6df in lookup (parent=0x7faf74003fa8, path=0x55a9dec3a482 "..", handle=0x7fb019db0af8, attrs_out=0x0) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/FSAL_GLUSTER/handle.c:113 #4 0x000055a9dec28b7f in mdc_get_parent (export=export@entry=0x55a9e066bbf0, entry=0x7faf74004260) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:287 #5 0x000055a9dec257f5 in mdcache_create_handle (exp_hdl=0x55a9e066bbf0, hdl_desc=<optimized out>, handle=0x7fb019db0be8, attrs_out=0x0) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1768 #6 0x000055a9deb91daa in nfs4_mds_putfh (data=data@entry=0x7fb019db1180) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_op_putfh.c:211 #7 0x000055a9deb922c0 in nfs4_op_putfh (op=0x7faf880017e0, data=0x7fb019db1180, resp=0x7faf74000ad0) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_op_putfh.c:281 #8 0x000055a9deb81bbd in nfs4_Compound (arg=<optimized out>, req=<optimized out>, res=0x7faf740009c0) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_Compound.c:734 #9 0x000055a9deb72d6c in nfs_rpc_execute (reqdata=reqdata@entry=0x7faf880008c0) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_worker_thread.c:1281 #10 0x000055a9deb743ca in worker_run (ctx=0x55a9e0758ec0) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_worker_thread.c:1548 #11 0x000055a9debfd999 in fridgethr_start_routine (arg=0x55a9e0758ec0) at /usr/src/debug/nfs-ganesha-2.4.4/src/support/fridgethr.c:550 #12 0x00007fb046401e25 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fb045acf34d in clone () from /lib64/libc.so.6 Version-Release number of selected component (if applicable): -------------------------------------------------------------- [root@gqas013 tmp]# rpm -qa|grep ganesha glusterfs-ganesha-3.8.4-26.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.4-6.el7rhgs.x86_64 nfs-ganesha-debuginfo-2.4.4-6.el7rhgs.x86_64 nfs-ganesha-2.4.4-6.el7rhgs.x86_64 [root@gqas013 tmp]# [root@gqas013 tmp]# [root@gqas013 tmp]# rpm -qa|grep libnti libntirpc-1.4.3-1.el7rhgs.x86_64 libntirpc-devel-1.4.3-1.el7rhgs.x86_64 [root@gqas013 tmp]# [root@gqas013 tmp]# rpm -qa|grep pacem pacemaker-cluster-libs-1.1.16-10.el7.x86_64 pacemaker-cli-1.1.16-10.el7.x86_64 pacemaker-1.1.16-10.el7.x86_64 pacemaker-libs-1.1.16-10.el7.x86_64 [root@gqas013 tmp]# [root@gqas013 tmp]# rpm -qa|grep coros corosynclib-2.4.0-9.el7.x86_64 corosync-2.4.0-9.el7.x86_64 [root@gqas013 tmp]# [root@gqas013 tmp]# rpm -qa|grep resource-ag resource-agents-3.9.5-100.el7.x86_64 [root@gqas013 tmp]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.4 Beta (Maipo) [root@gqas013 tmp]# How reproducible: ----------------- 3/3 --- Additional comment from Red Hat Bugzilla Rules Engine on 2017-05-31 Clearer BT : #0 glusterfs_normalize_dentry (parent=parent@entry=0x7fc99fb96778, component=component@entry=0x7fc99fb96780, dentry_name=dentry_name@entry=0x7fc99fb96840 "") at inode.c:2646 #1 0x00007fca36f12ac8 in priv_glfs_resolve_at (fs=fs@entry=0x55f5f8b01060, subvol=subvol@entry=0x7fca1c01fe00, at=at@entry=0x7fc734020090, origpath=origpath@entry=0x55f5f6f1b482 "..", loc=loc@entry=0x7fc99fb978d0, iatt=iatt@entry=0x7fc99fb97910, follow=follow@entry=0, reval=reval@entry=0) at glfs-resolve.c:412 #2 0x00007fca36f145c4 in pub_glfs_h_lookupat (fs=0x55f5f8b01060, parent=<optimized out>, path=path@entry=0x55f5f6f1b482 "..", stat=stat@entry=0x7fc99fb979f0, follow=follow@entry=0) at glfs-handleops.c:102 #3 0x00007fca36f146a8 in pub_glfs_h_lookupat34 (fs=<optimized out>, parent=<optimized out>, path=path@entry=0x55f5f6f1b482 "..", stat=stat@entry=0x7fc99fb979f0) at glfs-handleops.c:133 #4 0x00007fca373316df in lookup (parent=0x7fc734031fa8, path=0x55f5f6f1b482 "..", handle=0x7fc99fb97af8, attrs_out=0x0) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/FSAL_GLUSTER/handle.c:113 #5 0x000055f5f6f09b7f in mdc_get_parent (export=export@entry=0x55f5f8b00bf0, entry=0x7fc734033ee0) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:287 #6 0x000055f5f6f067f5 in mdcache_create_handle (exp_hdl=0x55f5f8b00bf0, hdl_desc=<optimized out>, handle=0x7fc99fb97be8, attrs_out=0x0) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1768 #7 0x000055f5f6e72daa in nfs4_mds_putfh (data=data@entry=0x7fc99fb98180) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_op_putfh.c:211 #8 0x000055f5f6e732c0 in nfs4_op_putfh (op=0x7fc97c243bd0, data=0x7fc99fb98180, resp=0x7fc734031bd0) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_op_putfh.c:281 #9 0x000055f5f6e62bbd in nfs4_Compound (arg=<optimized out>, req=<optimized out>, res=0x7fc734029050) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_Compound.c:734 #10 0x000055f5f6e53d6c in nfs_rpc_execute (reqdata=reqdata@entry=0x7fc97c243400) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_worker_thread.c:1281 #11 0x000055f5f6e553ca in worker_run (ctx=0x55f5f8c2d870) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_worker_thread.c:1548 #12 0x000055f5f6ede999 in fridgethr_start_routine (arg=0x55f5f8c2d870) at /usr/src/debug/nfs-ganesha-2.4.4/src/support/fridgethr.c:550 #13 0x00007fca3a295e25 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fca3996334d in clone () from /lib64/libc.so.6 (gdb) --- Additional comment from Manisha Saini on 2017-05-31 11:18:30 EDT --- Crash is not specific to rhel 7.4 Its even observed in Rhel 7.3 - After doing node reboot,node is unable to come up. bt- (no debugging symbols found)...done. Loaded symbols for /usr/lib64/gssproxy/proxymech.so Reading symbols from /lib64/libgssrpc.so.4...Reading symbols from /lib64/libgssrpc.so.4...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /lib64/libgssrpc.so.4 0x00007f5fe2aceef7 in pthread_join () from /lib64/libpthread.so.0 Missing separate debuginfos, use: debuginfo-install nfs-ganesha-2.4.4-6.el7rhgs.x86_64 (gdb) c Continuing. [New Thread 0x7f5f002bb700 (LWP 14671)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f5f81bbe700 (LWP 11392)] 0x00007f5fdf2758a0 in glusterfs_normalize_dentry () from /lib64/libglusterfs.so.0 (gdb) bt #0 0x00007f5fdf2758a0 in glusterfs_normalize_dentry () from /lib64/libglusterfs.so.0 #1 0x00007f5fdf547ac8 in glfs_resolve_at () from /lib64/libgfapi.so.0 #2 0x00007f5fdf5495c4 in glfs_h_lookupat () from /lib64/libgfapi.so.0 #3 0x00007f5fdf9666df in lookup () from /usr/lib64/ganesha/libfsalgluster.so #4 0x00007f5fe4617b7f in mdc_get_parent () #5 0x00007f5fe46147f5 in mdcache_create_handle () #6 0x00007f5fe4580daa in nfs4_mds_putfh () #7 0x00007f5fe45812c0 in nfs4_op_putfh () #8 0x00007f5fe4570bbd in nfs4_Compound () #9 0x00007f5fe4561d6c in nfs_rpc_execute () #10 0x00007f5fe45633ca in worker_run () #11 0x00007f5fe45ec999 in fridgethr_start_routine () #12 0x00007f5fe2acddc5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f5fe219c76d in clone () from /lib64/libc.so.6 Version- nfs-ganesha-2.4.4-6.el7rhgs.x86_64 --- Additional comment from Jiffin on 2017-06-02 09:33:52 EDT --- RCA : In above case, following should have happened linux untar was running on four different servers on four different directories using four different servers, let's say client1 was creating file a/b/c/file and failover happened(server1 got killed), so now it sends requests to server2. Usually the in nfs world all the communication happen via file handles. First server will try to create handle for that file using given gfid, but server does not have context about a/b/c till now. After creation of handle, it tries do lookup on parent using "..". During parent inode resolution the glusterfs_normalize_dentry() tries replcae ".." with its name and then it crashes, because this function has assumption inode of parent already linked to table. But in above case inode of parent never linked to table. Finally ending up in killing all the ganesha servers There are two solution to resolve this issue [1] handle inode_parent failures in glusterfs_normalise_dentry() or [2] revert the changes made to glfs_resolve_component() in https://review.gluster.org/#/c/17177 and then call glusterfs_normalise_dentry() followed by it Also thanks Soumya & Rafi for help in debugging the issue
REVIEW: https://review.gluster.org/17966 (tests/gfapi : add test case for nameless lookups in glfs_resolve_component()) posted (#1) for review on release-3.12 by jiffin tony Thottan (jthottan)
COMMIT: https://review.gluster.org/17966 committed in release-3.12 by Shyamsundar Ranganathan (srangana) ------ commit b683109485da3a2681ff769ab40b512fb9a59c4a Author: Jiffin Tony Thottan <jthottan> Date: Fri Jul 21 12:14:25 2017 +0530 tests/gfapi : add test case for nameless lookups in glfs_resolve_component() Plus address pending comment to add check for entry "" in glfs_resolve_component() Upstream reference : >Change-Id: I6063f776ce1cd76cb4c1b1f621b064f3dcc91e5c >BUG: 1460514 >Signed-off-by: Jiffin Tony Thottan <jthottan> >Reviewed-on: https://review.gluster.org/17844 >Smoke: Gluster Build System <jenkins.org> >CentOS-regression: Gluster Build System <jenkins.org> >Reviewed-by: Niels de Vos <ndevos> >Reviewed-by: soumya k <skoduri> >(cherry picked from commit 5c433f8f5834a4cae62d0375bfdb273242630f01) Change-Id: I6063f776ce1cd76cb4c1b1f621b064f3dcc91e5c BUG: 1477994 Signed-off-by: Jiffin Tony Thottan <jthottan> Reviewed-on: https://review.gluster.org/17966 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Shyamsundar Ranganathan <srangana>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.12.0, please open a new bug report. glusterfs-3.12.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2017-September/000082.html [2] https://www.gluster.org/pipermail/gluster-users/