Description of problem: ----------------------- I hit this while repro'ing https://bugzilla.redhat.com/show_bug.cgi?id=1481040 in QE labs. Ganesha crashed on 3 of my nodes with BT mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1492695. When I did a service start if NFS-Ganesha on the nodes it crashed,it crashed again and dumped a core with the following BT : (gdb) bt #0 0x00007f7f323ecad5 in list_del_init (old=0x7f7f4435f858) at list.h:87 #1 __iobuf_arena_destroy_iobufs (iobuf_arena=iobuf_arena@entry=0x7f7f4435f4d0) at iobuf.c:132 #2 0x00007f7f323ecc3a in __iobuf_arena_destroy (iobuf_pool=<optimized out>, iobuf_arena=0x7f7f4435f4d0) at iobuf.c:153 #3 0x00007f7f323ed2b7 in iobuf_pool_destroy (iobuf_pool=0x7f7f44399ae0) at iobuf.c:331 #4 0x00007f7f326967ce in glusterfs_ctx_destroy (ctx=0x7f7f4435f9e0) at glfs.c:1055 #5 pub_glfs_fini (fs=fs@entry=0x7f7f44f59a60) at glfs.c:1232 #6 0x00007f7f32ac3d9b in glusterfs_create_export (fsal_hdl=<optimized out>, parse_node=<optimized out>, err_type=<optimized out>, up_ops=0x7f7f377cb800 <fsal_up_top>) at /usr/src/debug/nfs-ganesha-2.4.1/src/FSAL/FSAL_GLUSTER/export.c:772 #7 0x00007f7f375418e0 in fsal_cfg_commit (node=0x7f7f38bd4c50, link_mem=0x7f7f4435f388, self_struct=<optimized out>, err_type=0x7ffe149180a0) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/exports.c:751 #8 0x00007f7f3757a848 in proc_block (node=<optimized out>, item=<optimized out>, link_mem=<optimized out>, err_type=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.1/src/config_parsing/config_parsing.c:1337 #9 0x00007f7f37579cc0 in do_block_load (err_type=<optimized out>, param_struct=<optimized out>, relax=<optimized out>, params=<optimized out>, blk=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.1/src/config_parsing/config_parsing.c:1195 #10 proc_block (node=<optimized out>, item=<optimized out>, link_mem=<optimized out>, err_type=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.1/src/config_parsing/config_parsing.c:1321 #11 0x00007f7f3757b130 in load_config_from_parse (config=config@entry=0x7f7f38bbd350 "Pӻ8\177\177", conf_blk=conf_blk@entry=0x7f7f377cf2a0 <export_param>, param=param@entry=0x0, unique=unique@entry=false, err_type=err_type@entry=0x7ffe149180a0) at /usr/src/debug/nfs-ganesha-2.4.1/src/config_parsing/config_parsing.c:1908 #12 0x00007f7f3754215b in ReadExports (in_config=0x7f7f38bbd350 "Pӻ8\177\177", err_type=err_type@entry=0x7ffe149180a0) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/exports.c:1970 #13 0x00007f7f37491291 in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.1/src/MainNFSD/nfs_main.c:454 (gdb) q It crashed on all the 3 nodes with the same BT. Version-Release number of selected component (if applicable): ------------------------------------------------------------- nfs-ganesha-2.4.1-11.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-18.4.el7rhgs.x86_64 How reproducible: ----------------- 1/1
Similar(but not same) crash on another node while restarting Ganesha : (gdb) bt #0 0x00007f6d7cccbad5 in list_del_init (old=0x7f6d889bbf78) at list.h:87 #1 __iobuf_arena_destroy_iobufs (iobuf_arena=iobuf_arena@entry=0x7f6d8898d020) at iobuf.c:132 #2 0x00007f6d7cccbc3a in __iobuf_arena_destroy (iobuf_pool=<optimized out>, iobuf_arena=0x7f6d8898d020) at iobuf.c:153 #3 0x00007f6d7cccc2b7 in iobuf_pool_destroy (iobuf_pool=0x7f6d889c7750) at iobuf.c:331 #4 0x00007f6d7cf757ce in glusterfs_ctx_destroy (ctx=0x7f6d8898d650) at glfs.c:1055 #5 pub_glfs_fini (fs=0x7f6d895876d0) at glfs.c:1232 #6 0x00007f6d7d3a27a1 in export_release (exp_hdl=0x7f6d895a2f70) at /usr/src/debug/nfs-ganesha-2.4.1/src/FSAL/FSAL_GLUSTER/export.c:88 #7 0x00007f6d81e419dd in mdcache_exp_release (exp_hdl=0x7f6d889bb840) at /usr/src/debug/nfs-ganesha-2.4.1/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_export.c:180 #8 0x00007f6d81e2179b in free_export_resources (export=export@entry=0x7f6d8898cdf8) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/exports.c:2068 #9 0x00007f6d81e314a9 in free_export (export=0x7f6d8898cdf8) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/export_mgr.c:252 #10 0x00007f6d81e329b5 in put_gsh_export (export=export@entry=0x7f6d8898cdf8) at /usr/src/debug/nfs-ganesha-2.4.1/src/support/export_mgr.c:632 #11 0x00007f6d81e33a4f in remove_all_exports () at /usr/src/debug/nfs-ganesha-2.4.1/src/support/export_mgr.c:762 #12 0x00007f6d81da6f1a in do_shutdown () at /usr/src/debug/nfs-ganesha-2.4.1/src/MainNFSD/nfs_admin_thread.c:433 #13 admin_thread (UnusedArg=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.1/src/MainNFSD/nfs_admin_thread.c:466 #14 0x00007f6d80305dc5 in start_thread (arg=0x7f6939a7e700) at pthread_create.c:308 #15 0x00007f6d7f9d473d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Same fixes mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1492695#c9 address this issue as well.
Since this BZ was observed while restarting ganesha service which got crashed while performing steps in BZ 1492695 BZ 1492695 has been now verified.Since the reproducer steps are same for both these bugs,moving this BZ also to verified state. # rpm -qa | grep ganesha nfs-ganesha-2.5.5-9.el7rhgs.x86_64 nfs-ganesha-debuginfo-2.5.5-9.el7rhgs.x86_64 glusterfs-ganesha-3.12.2-15.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-9.el7rhgs.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2610