Bug 1003917 - smbd crashes in libglusterfs under heavy load
Summary: smbd crashes in libglusterfs under heavy load
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: samba
Version: 2.1
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: ---
Assignee: Raghavendra Talur
QA Contact: surabhi
URL:
Whiteboard:
Depends On:
Blocks: 1004100
TreeView+ depends on / blocked
 
Reported: 2013-09-03 14:06 UTC by surabhi
Modified: 2013-09-23 22:32 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1004100 (view as bug list)
Environment:
Last Closed: 2013-09-23 22:32:17 UTC
Embargoed:


Attachments (Terms of Use)

Description surabhi 2013-09-03 14:06:26 UTC
Description of problem:
While running the glusterfs_build : git clone followed by autogen and configure.As soon as we run configure the smbd crashes and dumps core.

Version-Release number of selected component (if applicable):
glusterfs-3.4.0.30rhs-2.el6rhs.x86_64

How reproducible:
Always

Steps to Reproduce:
1.Create a volume .
2.Mount via cifs on Linux client
3.Do a git clone git://github.com/gluster/glusterfs.git glusterfs.git
4.cd glusterfs.git/
5../autogen.sh
6 ./configure

The core will get generated and configure will throw following error:
root@darrel [Sep-03-2013-13:16:32] >cd /mnt/cifs_mount1
root@darrel [Sep-03-2013-13:16:38] >git clone git://github.com/gluster/glusterfs.git glusterfs.git
Initialized empty Git repository in /mnt/cifs_mount1/glusterfs.git/.git/
remote: Counting objects: 67920, done.
remote: Compressing objects: 100% (23806/23806), done.
remote: Total 67920 (delta 45556), reused 66010 (delta 43843)
Receiving objects: 100% (67920/67920), 23.92 MiB | 250 KiB/s, done.
Resolving deltas: 100% (45556/45556), done.
root@darrel [Sep-03-2013-13:18:14] >
root@darrel [Sep-03-2013-13:18:16] >ls
glusterfs.git
root@darrel [Sep-03-2013-13:18:22] >cd glusterfs.git/
root@darrel [Sep-03-2013-13:18:34] >ls
api              configure.ac    error-codes.json           glusterfsd            Makefile.am   tests
argp-standalone  contrib         extras                     glusterfs-hadoop      NEWS          THANKS
AUTHORS          CONTRIBUTING    gen-headers.py             glusterfs.spec.in     README        xlators
autogen.sh       COPYING-GPLV2   geo-replication            INSTALL               rfc.sh
ChangeLog        COPYING-LGPLV3  gf-error-codes.h.template  libgfchangelog.pc.in  rpc
cli              doc             glusterfs-api.pc.in        libglusterfs          run-tests.sh
root@darrel [Sep-03-2013-13:18:35] >./autogen.sh 

... GlusterFS autogen ...

Generate gf-error-codes.h ...
`gf-error-codes.h' -> `libglusterfs/src/gf-error-codes.h'
Running aclocal...
configure.ac:493: warning: macro `AM_PATH_XML2' not found in library
Running autoheader...
Running libtoolize...
Running autoconf...
configure.ac:493: error: possibly undefined macro: AM_PATH_XML2
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
Running automake...
configure.ac:168: installing `./config.guess'
configure.ac:168: installing `./config.sub'
configure.ac:11: installing `./install-sh'
configure.ac:11: installing `./missing'
api/src/Makefile.am: installing `./depcomp'
geo-replication/syncdaemon/Makefile.am:3: installing `./py-compile'
configure.ac:10: installing `./install-sh'
configure.ac:10: installing `./missing'
Makefile.am: installing `./depcomp'

Please proceed with configuring, compiling, and installing.
root@darrel [Sep-03-2013-13:19:24] >./configure 

Core was generated:

Core was generated by `smbd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f1b767f363d in _gf_log () from /usr/lib64/libglusterfs.so.0
Missing separate debuginfos, use: debuginfo-install samba-3.6.9-160.3.el6rhs.x86_64
(gdb) bt
#0  0x00007f1b767f363d in _gf_log () from /usr/lib64/libglusterfs.so.0
#1  0x00007f1b765cfd6a in rpc_clnt_record_build_header () from /usr/lib64/libgfrpc.so.0
#2  0x00007f1b765d0096 in rpc_clnt_record_build_record () from /usr/lib64/libgfrpc.so.0
#3  0x00007f1b765d02f6 in rpc_clnt_record () from /usr/lib64/libgfrpc.so.0
#4  0x00007f1b765d07c9 in rpc_clnt_submit () from /usr/lib64/libgfrpc.so.0
#5  0x00007f1b66df8723 in client_submit_request () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/protocol/client.so
#6  0x00007f1b66e031d8 in client3_3_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/protocol/client.so
#7  0x00007f1b66df845c in client_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/protocol/client.so
#8  0x00007f1b66bce961 in afr_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/cluster/replicate.so
#9  0x00007f1b66954638 in dht_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/cluster/distribute.so
#10 0x00007f1b767f48ad in default_lookup () from /usr/lib64/libglusterfs.so.0
#11 0x00007f1b665164ec in ioc_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/performance/io-cache.so
#12 0x00007f1b6630cc47 in qr_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/performance/quick-read.so
#13 0x00007f1b65ee7600 in trace_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/debug/trace.so
#14 0x00007f1b65cce337 in io_stats_lookup () from /usr/lib64/glusterfs/3.4.0.30rhs/xlator/debug/io-stats.so
#15 0x00007f1b768240ea in syncop_lookup () from /usr/lib64/libglusterfs.so.0


Actual results:
The smbd crashes.

Expected results:
The glusterfs_build should run successfully.

Additional info:

Comment 2 Christopher R. Hertel 2013-09-04 00:16:50 UTC
Update:  The stack trace given above does not pass through the EA handling.  This is a crash under heavy load.

The load, in this case, is building Gluster itself.  The stack trace shows that the crash occurs in the Gluster layer.  SMBD crashes because it is linked to the Gluster client library.

Comment 4 Raghavendra Talur 2013-09-05 11:16:11 UTC
Posted patch for review at 
https://code.engineering.redhat.com/gerrit/#/c/12520/

Comment 5 surabhi 2013-09-06 11:22:32 UTC
Verified the bz with latest version
glusterfs-server-3.4.0.31rhs-1.el6rhs.x86_64

Executing following steps doesn't generates core and tests ran successfully.

1.Create a volume .
2.Mount via cifs on Linux client
3.Do a git clone git://github.com/gluster/glusterfs.git glusterfs.git
4.cd glusterfs.git/
5../autogen.sh
6 ./configure

[root@RHEL6 stat-prefetch]# cd glusterfs.git/
[root@RHEL6 glusterfs.git]# ./autogen.sh

... GlusterFS autogen ...

Running automake...

Please proceed with configuring, compiling, and installing.

Please proceed with configuring, compiling, and installing.
[root@RHEL6 glusterfs.git]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1966080
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so

Comment 6 Scott Haines 2013-09-23 22:32:17 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html


Note You need to log in before you can comment on or make changes to this bug.