Bug 689483

Summary: crash in ganglia moddisk.so
Product: [Fedora] Fedora Reporter: Terje Røsten <terje.rosten>
Component: gangliaAssignee: Kostas Georgiou <k.georgiou>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 15CC: bernard, cjg9411, k.georgiou, kjell.m.randa
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: ganglia-3.1.7-4.fc15 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-07-15 01:24:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Terje Røsten 2011-03-21 16:19:18 UTC
Description of problem:

I see a crash when running gmond on Fedora 15 (and rawhide):

*** stack smashing detected ***: gmond terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x3e2c4f7a87]
/lib64/libc.so.6(__fortify_fail+0x0)[0x3e2c4f7a50]
/usr/lib64/ganglia/moddisk.so(disk_free_func+0x0)[0x7f82ec31fef0]
/usr/lib64/ganglia/moddisk.so(disk_total_func+0x20)[0x7f82ec31ff40]
/usr/lib64/ganglia/moddisk.so(+0x25aa)[0x7f82ec31d5aa]
gmond(Ganglia_collection_group_collect+0xa2)[0x407d52]
gmond(process_collection_groups+0x52)[0x408342]
gmond(main+0x3f2)[0x404132]
/lib64/libc.so.6(__libc_start_main+0xed)[0x3e2c42131d]
gmond[0x404641]
======= Memory map: ========
00400000-0040e000 r-xp 00000000 fc:01 399138                             /usr/sbin/gmond
0060e000-0060f000 rw-p 0000e000 fc:01 399138                             /usr/sbin/gmond
0060f000-00610000 rw-p 00000000 00:00 0 
0182e000-01874000 rw-p 00000000 00:00 0                                  [heap]
3e2c000000-3e2c020000 r-xp 00000000 fc:01 264731                         /lib64/ld-2.13.90.so
3e2c220000-3e2c221000 r--p 00020000 fc:01 264731                         /lib64/ld-2.13.90.so
3e2c221000-3e2c222000 rw-p 00021000 fc:01 264731                         /lib64/ld-2.13.90.so
3e2c222000-3e2c223000 rw-p 00000000 00:00 0 

0182e000-01874000 rw-p 00000000 00:00 0                                  [heap]
3e2c000000-3e2c020000 r-xp 00000000 fc:01 264731                         /lib64/ld-2.13.90.so
3e2c220000-3e2c221000 r--p 00020000 fc:01 264731                         /lib64/ld-2.13.90.so
3e2c221000-3e2c222000 rw-p 00021000 fc:01 264731                         /lib64/ld-2.13.90.so
3e2c222000-3e2c223000 rw-p 00000000 00:00 0 
3e2c400000-3e2c592000 r-xp 00000000 fc:01 264732                         /lib64/libc-2.13.90.so
3e2c592000-3e2c792000 ---p 00192000 fc:01 264732                         /lib64/libc-2.13.90.so
3e2c792000-3e2c796000 r--p 00192000 fc:01 264732                         /lib64/libc-2.13.90.so
3e2c796000-3e2c797000 rw-p 00196000 fc:01 264732                         /lib64/libc-2.13.90.so
3e2c797000-3e2c79d000 rw-p 00000000 00:00 0 
3e2c800000-3e2c817000 r-xp 00000000 fc:01 264733                         /lib64/libpthread-2.13.90.so
3e2c817000-3e2ca16000 ---p 00017000 fc:01 264733                         /lib64/libpthread-2.13.90.so
3e2ca16000-3e2ca17000 r--p 00016000 fc:01 264733                         /lib64/libpthread-2.13.90.so
3e2ca17000-3e2ca18000 rw-p 00017000 fc:01 264733                         /lib64/libpthread-2.13.90.so
3e2ca18000-3e2ca1c000 rw-p 00000000 00:00 0 
3e2cc00000-3e2cc02000 r-xp 00000000 fc:01 264735                         /lib64/libdl-2.13.90.so
3e2cc02000-3e2ce02000 ---p 00002000 fc:01 264735                         /lib64/libdl-2.13.90.so
3e2ce02000-3e2ce03000 r--p 00002000 fc:01 264735                         /lib64/libdl-2.13.90.so
3e2ce03000-3e2ce04000 rw-p 00003000 fc:01 264735                         /lib64/libdl-2.13.90.so
3e2dc00000-3e2dc15000 r-xp 00000000 fc:01 264749                         /lib64/libgcc_s-4.6.0-20110318.so.1
3e2dc15000-3e2de14000 ---p 00015000 fc:01 264749                         /lib64/libgcc_s-4.6.0-20110318.so.1
3e2de14000-3e2de15000 rw-p 00014000 fc:01 264749                         /lib64/libgcc_s-4.6.0-20110318.so.1
3e2e800000-3e2e817000 r-xp 00000000 fc:01 264740                         /lib64/libresolv-2.13.90.so
3e2e817000-3e2ea17000 ---p 00017000 fc:01 264740                         /lib64/libresolv-2.13.90.so
3e2ea17000-3e2ea18000 r--p 00017000 fc:01 264740                         /lib64/libresolv-2.13.90.so
3e2ea18000-3e2ea19000 rw-p 00018000 fc:01 264740                         /lib64/libresolv-2.13.90.so
3e2ea19000-3e2ea1b000 rw-p 00000000 00:00 0 
3e30800000-3e30827000 r-xp 00000000 fc:01 264757                         /lib64/libexpat.so.1.5.2
3e30827000-3e30a26000 ---p 00027000 fc:01 264757                         /lib64/libexpat.so.1.5.2
3e30a26000-3e30a29000 rw-p 00026000 fc:01 264757                         /lib64/libexpat.so.1.5.2
3e36c00000-3e36c5c000 r-xp 00000000 fc:01 264736                         /lib64/libfreebl3.so
3e36c5c000-3e36e5c000 ---p 0005c000 fc:01 264736                         /lib64/libfreebl3.so
3e36e5c000-3e36e5d000 rw-p 0005c000 fc:01 264736                         /lib64/libfreebl3.so
3e36e5d000-3e36e62000 rw-p 00000000 00:00 0 
3e37000000-3e37008000 r-xp 00000000 fc:01 264737                         /lib64/libcrypt-2.13.90.so
3e37008000-3e37207000 ---p 00008000 fc:01 264737                         /lib64/libcrypt-2.13.90.so
3e37207000-3e37208000 r--p 00007000 fc:01 264737                         /lib64/libcrypt-2.13.90.so
3e37208000-3e37209000 rw-p 00008000 fc:01 264737                         /lib64/libcrypt-2.13.90.so
3e37209000-3e37237000 rw-p 00000000 00:00 0 
3e38000000-3e38004000 r-xp 00000000 fc:01 264758                         /lib64/libuuid.so.1.3.0
3e38004000-3e38203000 ---p 00004000 fc:01 264758                         /lib64/libuuid.so.1.3.0
3e38203000-3e38204000 rw-p 00003000 fc:01 264758                         /lib64/libuuid.so.1.3.0
3e39000000-3e3903c000 r-xp 00000000 fc:01 261588                         /lib64/libpcre.so.0.0.1
3e3903c000-3e3923b000 ---p 0003c000 fc:01 261588                         /lib64/libpcre.so.0.0.1
3e3923b000-3e3923c000 rw-p 0003b000 fc:01 261588                         /lib64/libpcre.so.0.0.1
3e3e000000-3e3e017000 r-xp 00000000 fc:01 261177                         /lib64/libnsl-2.13.90.so
3e3e017000-3e3e216000 ---p 00017000 fc:01 261177                         /lib64/libnsl-2.13.90.so
3e3e216000-3e3e217000 r--p 00016000 fc:01 261177                         /lib64/libnsl-2.13.90.so
3e3e217000-3e3e218000 rw-p 00017000 fc:01 261177                         /lib64/libnsl-2.13.90.so
3e3e218000-3e3e21a000 rw-p 00000000 00:00 0 
7f82eb467000-7f82eb473000 r-xp 00000000 fc:01 261153                     /lib64/libnss_files-2.13.90.so
7f82eb473000-7f82eb672000 ---p 0000c000 fc:01 261153                     /lib64/libnss_files-2.13.90.so
7f82eb672000-7f82eb673000 r--p 0000b000 fc:01 261153                     /lib64/libnss_files-2.13.90.so
7f82eb673000-7f82eb674000 rw-p 0000c000 fc:01 261153                     /lib64/libnss_files-2.13.90.so
7f82eb674000-7f82eb677000 r-xp 00000000 fc:01 396419                     /usr/lib64/ganglia/modmulticpu.so
7f82eb677000-7f82eb876000 ---p 00003000 fc:01 396419                     /usr/lib64/ganglia/modmulticpu.so
7f82eb876000-7f82eb87b000 rw-p 00002000 fc:01 396419                     /usr/lib64/ganglia/modmulticpu.so
f82eb87b000-7f82eb882000 r-xp 00000000 fc:01 396464                     /usr/lib64/ganglia/modsys.so
7f82eb882000-7f82eba81000 ---p 00007000 fc:01 396464                     /usr/lib64/ganglia/modsys.so
7f82eba81000-7f82eba92000 rw-p 00006000 fc:01 396464                     /usr/lib64/ganglia/modsys.so
7f82eba92000-7f82eba9b000 rw-p 00000000 00:00 0 
7f82eba9b000-7f82ebaa2000 r-xp 00000000 fc:01 396462                     /usr/lib64/ganglia/modproc.soAvbrutt (SIGABRT)

Version-Release number of selected component (if applicable):

$ rpm -q ganglia-gmond ganglia glibc ; uname -r
ganglia-gmond-3.1.7-3.fc15.x86_64
ganglia-3.1.7-3.fc15.x86_64
glibc-2.13.90-6.x86_64
2.6.38-1.fc15.x86_64

How reproducible:

Install ganglia-gmond.

start gmond on the command:

$ gmond -f

By just doing 

$ mv /usr/lib64/ganglia/moddisk.so /usr/lib64/ganglia/moddisk.so.broken 

gmond is running fine (with some missing info of course).

Comment 1 Kostas Georgiou 2011-03-22 21:47:12 UTC
I can't reproduce localy unfortunately :(
Would it be possible to install the debuginfo packages and try again from inside gdb?

Comment 2 Terje Røsten 2011-03-22 22:09:58 UTC
Hm, strange I can't reproduce any longer. 

From yum history I see that glibc has been updated to glibc-2.13.90-7.x86_64.

Do you use glibc-2.13.90-7 too?

Comment 3 Bernard Li 2011-03-22 23:16:36 UTC
I tested on a VM running Fedora 15 Alpha x86_64 and couldn't reproduce it either.

Also using glibc-2.13.90-7.x86_64.

Comment 4 Kostas Georgiou 2011-03-22 23:26:40 UTC
I tested with glibc-2.13.90-6 actually. We can't rule out a stack overflow somewhere but until it shows up again I'll put it down as a "random" rawhide failure.

Comment 5 Terje Røsten 2011-03-23 07:30:28 UTC
Yeah, you can close it, I will reopen if I see the problem again. Thanks for the quick reply.

Comment 6 Kostas Georgiou 2011-03-24 18:36:35 UTC
Reopening since I just saw the problem.

(gdb) where
#0  0x0000003d9cc362c5 in raise () from /lib64/libc.so.6
#1  0x0000003d9cc37bdb in abort () from /lib64/libc.so.6
#2  0x0000003d9cc722c3 in __libc_message () from /lib64/libc.so.6
#3  0x0000003d9ccf7a87 in __fortify_fail () from /lib64/libc.so.6
#4  0x0000003d9ccf7a50 in __stack_chk_fail () from /lib64/libc.so.6
#5  0x00007ffff1d02ef0 in find_disk_space (total_size=0x7fffffffe2e8, total_free=0x7fffffffe2e0) at metrics.c:1264
#6  0x00007ffff1d02f40 in disk_total_func () at metrics.c:1289
#7  0x00007ffff1d005aa in disk_metric_handler (metric_index=<optimized out>) at mod_disk.c:36
#8  disk_metric_handler (metric_index=<optimized out>) at mod_disk.c:27

Unfortunately after I setup some breakpoints and tried to find where it dies it magically fixed itself!

My bet for the cause is the sscanf in find_disk_space. I can't see anything else
that could overwrite the stack in the function.

sscanf(procline, "%s %s %s %s ", device, mount, type, mode);

Comment 7 Kostas Georgiou 2011-03-25 13:11:43 UTC
Got it, here is an entry from /proc/mounts (which was getting expired and removed by the automounter everytime I was trying to debug the problem, so I missed it the first few times)

foobar:/a/b/c /vols/c nfs rw,nosuid,nodev,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.10.10,mountvers=3,mountport=22194,mountproto=udp,local_lock=none,addr=10.10.10.10

char mount[128], device[128], type[32], mode[128];
rc=sscanf(procline, "%s %s %s %s ", device, mount, type, mode);

Not f15 specific it seems, it's just better at detecting stack corruptions.

Comment 8 Terje Røsten 2011-06-28 19:02:43 UTC
Any fix available for this? It's beginning to annoy me a little :-)

Comment 9 Fedora Update System 2011-07-05 12:22:58 UTC
ganglia-3.1.7-4.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/ganglia-3.1.7-4.fc15

Comment 10 Fedora Update System 2011-07-06 21:41:03 UTC
Package ganglia-3.1.7-4.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing ganglia-3.1.7-4.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/ganglia-3.1.7-4.fc15
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2011-07-15 01:23:54 UTC
ganglia-3.1.7-4.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.