Red Hat Bugzilla – Bug 1003917
smbd crashes in libglusterfs under heavy load
Last modified: 2013-09-23 18:32:17 EDT
Description of problem: While running the glusterfs_build : git clone followed by autogen and configure.As soon as we run configure the smbd crashes and dumps core. Version-Release number of selected component (if applicable): glusterfs-3.4.0.30rhs-2.el6rhs.x86_64 How reproducible: Always Steps to Reproduce: 1.Create a volume . 2.Mount via cifs on Linux client 3.Do a git clone git://github.com/gluster/glusterfs.git glusterfs.git 4.cd glusterfs.git/ 5../autogen.sh 6 ./configure The core will get generated and configure will throw following error: root@darrel [Sep-03-2013-13:16:32] >cd /mnt/cifs_mount1 root@darrel [Sep-03-2013-13:16:38] >git clone git://github.com/gluster/glusterfs.git glusterfs.git Initialized empty Git repository in /mnt/cifs_mount1/glusterfs.git/.git/ remote: Counting objects: 67920, done. remote: Compressing objects: 100% (23806/23806), done. remote: Total 67920 (delta 45556), reused 66010 (delta 43843) Receiving objects: 100% (67920/67920), 23.92 MiB | 250 KiB/s, done. Resolving deltas: 100% (45556/45556), done. root@darrel [Sep-03-2013-13:18:14] > root@darrel [Sep-03-2013-13:18:16] >ls glusterfs.git root@darrel [Sep-03-2013-13:18:22] >cd glusterfs.git/ root@darrel [Sep-03-2013-13:18:34] >ls api configure.ac error-codes.json glusterfsd Makefile.am tests argp-standalone contrib extras glusterfs-hadoop NEWS THANKS AUTHORS CONTRIBUTING gen-headers.py glusterfs.spec.in README xlators autogen.sh COPYING-GPLV2 geo-replication INSTALL rfc.sh ChangeLog COPYING-LGPLV3 gf-error-codes.h.template libgfchangelog.pc.in rpc cli doc glusterfs-api.pc.in libglusterfs run-tests.sh root@darrel [Sep-03-2013-13:18:35] >./autogen.sh ... GlusterFS autogen ... Generate gf-error-codes.h ... `gf-error-codes.h' -> `libglusterfs/src/gf-error-codes.h' Running aclocal... configure.ac:493: warning: macro `AM_PATH_XML2' not found in library Running autoheader... Running libtoolize... Running autoconf... configure.ac:493: error: possibly undefined macro: AM_PATH_XML2 If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. Running automake... configure.ac:168: installing `./config.guess' configure.ac:168: installing `./config.sub' configure.ac:11: installing `./install-sh' configure.ac:11: installing `./missing' api/src/Makefile.am: installing `./depcomp' geo-replication/syncdaemon/Makefile.am:3: installing `./py-compile' configure.ac:10: installing `./install-sh' configure.ac:10: installing `./missing' Makefile.am: installing `./depcomp' Please proceed with configuring, compiling, and installing. root@darrel [Sep-03-2013-13:19:24] >./configure Core was generated: Core was generated by `smbd'. Program terminated with signal 11, Segmentation fault. #0 0x00007f1b767f363d in _gf_log () from /usr/lib64/libglusterfs.so.0 Missing separate debuginfos, use: debuginfo-install samba-3.6.9-160.3.el6rhs.x86_64 (gdb) bt #0 0x00007f1b767f363d in _gf_log () from /usr/lib64/libglusterfs.so.0 #1 0x00007f1b765cfd6a in rpc_clnt_record_build_header () from /usr/lib64/libgfrpc.so.0 #2 0x00007f1b765d0096 in rpc_clnt_record_build_record () from /usr/lib64/libgfrpc.so.0 #3 0x00007f1b765d02f6 in rpc_clnt_record () from /usr/lib64/libgfrpc.so.0 #4 0x00007f1b765d07c9 in rpc_clnt_submit () from /usr/lib64/libgfrpc.so.0 #5 0x00007f1b66df8723 in client_submit_request () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/protocol/client.so #6 0x00007f1b66e031d8 in client3_3_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/protocol/client.so #7 0x00007f1b66df845c in client_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/protocol/client.so #8 0x00007f1b66bce961 in afr_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/cluster/replicate.so #9 0x00007f1b66954638 in dht_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/cluster/distribute.so #10 0x00007f1b767f48ad in default_lookup () from /usr/lib64/libglusterfs.so.0 #11 0x00007f1b665164ec in ioc_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/performance/io-cache.so #12 0x00007f1b6630cc47 in qr_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/performance/quick-read.so #13 0x00007f1b65ee7600 in trace_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/debug/trace.so #14 0x00007f1b65cce337 in io_stats_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/debug/io-stats.so #15 0x00007f1b768240ea in syncop_lookup () from /usr/lib64/libglusterfs.so.0 Actual results: The smbd crashes. Expected results: The glusterfs_build should run successfully. Additional info:
Update: The stack trace given above does not pass through the EA handling. This is a crash under heavy load. The load, in this case, is building Gluster itself. The stack trace shows that the crash occurs in the Gluster layer. SMBD crashes because it is linked to the Gluster client library.
Posted patch for review at https://code.engineering.redhat.com/gerrit/#/c/12520/
Verified the bz with latest version glusterfs-server-3.4.0.31rhs-1.el6rhs.x86_64 Executing following steps doesn't generates core and tests ran successfully. 1.Create a volume . 2.Mount via cifs on Linux client 3.Do a git clone git://github.com/gluster/glusterfs.git glusterfs.git 4.cd glusterfs.git/ 5../autogen.sh 6 ./configure [root@RHEL6 stat-prefetch]# cd glusterfs.git/ [root@RHEL6 glusterfs.git]# ./autogen.sh ... GlusterFS autogen ... Running automake... Please proceed with configuring, compiling, and installing. Please proceed with configuring, compiling, and installing. [root@RHEL6 glusterfs.git]# ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... /usr/bin/ld checking if the linker (/usr/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B checking the name lister (/usr/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1966080 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking for /usr/bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for ar... ar checking for strip... strip checking for ranlib... ranlib checking command to parse /usr/bin/nm -B output from gcc object... ok checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html