Description of problem: I see a crash when running gmond on Fedora 15 (and rawhide): *** stack smashing detected ***: gmond terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x3e2c4f7a87] /lib64/libc.so.6(__fortify_fail+0x0)[0x3e2c4f7a50] /usr/lib64/ganglia/moddisk.so(disk_free_func+0x0)[0x7f82ec31fef0] /usr/lib64/ganglia/moddisk.so(disk_total_func+0x20)[0x7f82ec31ff40] /usr/lib64/ganglia/moddisk.so(+0x25aa)[0x7f82ec31d5aa] gmond(Ganglia_collection_group_collect+0xa2)[0x407d52] gmond(process_collection_groups+0x52)[0x408342] gmond(main+0x3f2)[0x404132] /lib64/libc.so.6(__libc_start_main+0xed)[0x3e2c42131d] gmond[0x404641] ======= Memory map: ======== 00400000-0040e000 r-xp 00000000 fc:01 399138 /usr/sbin/gmond 0060e000-0060f000 rw-p 0000e000 fc:01 399138 /usr/sbin/gmond 0060f000-00610000 rw-p 00000000 00:00 0 0182e000-01874000 rw-p 00000000 00:00 0 [heap] 3e2c000000-3e2c020000 r-xp 00000000 fc:01 264731 /lib64/ld-2.13.90.so 3e2c220000-3e2c221000 r--p 00020000 fc:01 264731 /lib64/ld-2.13.90.so 3e2c221000-3e2c222000 rw-p 00021000 fc:01 264731 /lib64/ld-2.13.90.so 3e2c222000-3e2c223000 rw-p 00000000 00:00 0 0182e000-01874000 rw-p 00000000 00:00 0 [heap] 3e2c000000-3e2c020000 r-xp 00000000 fc:01 264731 /lib64/ld-2.13.90.so 3e2c220000-3e2c221000 r--p 00020000 fc:01 264731 /lib64/ld-2.13.90.so 3e2c221000-3e2c222000 rw-p 00021000 fc:01 264731 /lib64/ld-2.13.90.so 3e2c222000-3e2c223000 rw-p 00000000 00:00 0 3e2c400000-3e2c592000 r-xp 00000000 fc:01 264732 /lib64/libc-2.13.90.so 3e2c592000-3e2c792000 ---p 00192000 fc:01 264732 /lib64/libc-2.13.90.so 3e2c792000-3e2c796000 r--p 00192000 fc:01 264732 /lib64/libc-2.13.90.so 3e2c796000-3e2c797000 rw-p 00196000 fc:01 264732 /lib64/libc-2.13.90.so 3e2c797000-3e2c79d000 rw-p 00000000 00:00 0 3e2c800000-3e2c817000 r-xp 00000000 fc:01 264733 /lib64/libpthread-2.13.90.so 3e2c817000-3e2ca16000 ---p 00017000 fc:01 264733 /lib64/libpthread-2.13.90.so 3e2ca16000-3e2ca17000 r--p 00016000 fc:01 264733 /lib64/libpthread-2.13.90.so 3e2ca17000-3e2ca18000 rw-p 00017000 fc:01 264733 /lib64/libpthread-2.13.90.so 3e2ca18000-3e2ca1c000 rw-p 00000000 00:00 0 3e2cc00000-3e2cc02000 r-xp 00000000 fc:01 264735 /lib64/libdl-2.13.90.so 3e2cc02000-3e2ce02000 ---p 00002000 fc:01 264735 /lib64/libdl-2.13.90.so 3e2ce02000-3e2ce03000 r--p 00002000 fc:01 264735 /lib64/libdl-2.13.90.so 3e2ce03000-3e2ce04000 rw-p 00003000 fc:01 264735 /lib64/libdl-2.13.90.so 3e2dc00000-3e2dc15000 r-xp 00000000 fc:01 264749 /lib64/libgcc_s-4.6.0-20110318.so.1 3e2dc15000-3e2de14000 ---p 00015000 fc:01 264749 /lib64/libgcc_s-4.6.0-20110318.so.1 3e2de14000-3e2de15000 rw-p 00014000 fc:01 264749 /lib64/libgcc_s-4.6.0-20110318.so.1 3e2e800000-3e2e817000 r-xp 00000000 fc:01 264740 /lib64/libresolv-2.13.90.so 3e2e817000-3e2ea17000 ---p 00017000 fc:01 264740 /lib64/libresolv-2.13.90.so 3e2ea17000-3e2ea18000 r--p 00017000 fc:01 264740 /lib64/libresolv-2.13.90.so 3e2ea18000-3e2ea19000 rw-p 00018000 fc:01 264740 /lib64/libresolv-2.13.90.so 3e2ea19000-3e2ea1b000 rw-p 00000000 00:00 0 3e30800000-3e30827000 r-xp 00000000 fc:01 264757 /lib64/libexpat.so.1.5.2 3e30827000-3e30a26000 ---p 00027000 fc:01 264757 /lib64/libexpat.so.1.5.2 3e30a26000-3e30a29000 rw-p 00026000 fc:01 264757 /lib64/libexpat.so.1.5.2 3e36c00000-3e36c5c000 r-xp 00000000 fc:01 264736 /lib64/libfreebl3.so 3e36c5c000-3e36e5c000 ---p 0005c000 fc:01 264736 /lib64/libfreebl3.so 3e36e5c000-3e36e5d000 rw-p 0005c000 fc:01 264736 /lib64/libfreebl3.so 3e36e5d000-3e36e62000 rw-p 00000000 00:00 0 3e37000000-3e37008000 r-xp 00000000 fc:01 264737 /lib64/libcrypt-2.13.90.so 3e37008000-3e37207000 ---p 00008000 fc:01 264737 /lib64/libcrypt-2.13.90.so 3e37207000-3e37208000 r--p 00007000 fc:01 264737 /lib64/libcrypt-2.13.90.so 3e37208000-3e37209000 rw-p 00008000 fc:01 264737 /lib64/libcrypt-2.13.90.so 3e37209000-3e37237000 rw-p 00000000 00:00 0 3e38000000-3e38004000 r-xp 00000000 fc:01 264758 /lib64/libuuid.so.1.3.0 3e38004000-3e38203000 ---p 00004000 fc:01 264758 /lib64/libuuid.so.1.3.0 3e38203000-3e38204000 rw-p 00003000 fc:01 264758 /lib64/libuuid.so.1.3.0 3e39000000-3e3903c000 r-xp 00000000 fc:01 261588 /lib64/libpcre.so.0.0.1 3e3903c000-3e3923b000 ---p 0003c000 fc:01 261588 /lib64/libpcre.so.0.0.1 3e3923b000-3e3923c000 rw-p 0003b000 fc:01 261588 /lib64/libpcre.so.0.0.1 3e3e000000-3e3e017000 r-xp 00000000 fc:01 261177 /lib64/libnsl-2.13.90.so 3e3e017000-3e3e216000 ---p 00017000 fc:01 261177 /lib64/libnsl-2.13.90.so 3e3e216000-3e3e217000 r--p 00016000 fc:01 261177 /lib64/libnsl-2.13.90.so 3e3e217000-3e3e218000 rw-p 00017000 fc:01 261177 /lib64/libnsl-2.13.90.so 3e3e218000-3e3e21a000 rw-p 00000000 00:00 0 7f82eb467000-7f82eb473000 r-xp 00000000 fc:01 261153 /lib64/libnss_files-2.13.90.so 7f82eb473000-7f82eb672000 ---p 0000c000 fc:01 261153 /lib64/libnss_files-2.13.90.so 7f82eb672000-7f82eb673000 r--p 0000b000 fc:01 261153 /lib64/libnss_files-2.13.90.so 7f82eb673000-7f82eb674000 rw-p 0000c000 fc:01 261153 /lib64/libnss_files-2.13.90.so 7f82eb674000-7f82eb677000 r-xp 00000000 fc:01 396419 /usr/lib64/ganglia/modmulticpu.so 7f82eb677000-7f82eb876000 ---p 00003000 fc:01 396419 /usr/lib64/ganglia/modmulticpu.so 7f82eb876000-7f82eb87b000 rw-p 00002000 fc:01 396419 /usr/lib64/ganglia/modmulticpu.so f82eb87b000-7f82eb882000 r-xp 00000000 fc:01 396464 /usr/lib64/ganglia/modsys.so 7f82eb882000-7f82eba81000 ---p 00007000 fc:01 396464 /usr/lib64/ganglia/modsys.so 7f82eba81000-7f82eba92000 rw-p 00006000 fc:01 396464 /usr/lib64/ganglia/modsys.so 7f82eba92000-7f82eba9b000 rw-p 00000000 00:00 0 7f82eba9b000-7f82ebaa2000 r-xp 00000000 fc:01 396462 /usr/lib64/ganglia/modproc.soAvbrutt (SIGABRT) Version-Release number of selected component (if applicable): $ rpm -q ganglia-gmond ganglia glibc ; uname -r ganglia-gmond-3.1.7-3.fc15.x86_64 ganglia-3.1.7-3.fc15.x86_64 glibc-2.13.90-6.x86_64 2.6.38-1.fc15.x86_64 How reproducible: Install ganglia-gmond. start gmond on the command: $ gmond -f By just doing $ mv /usr/lib64/ganglia/moddisk.so /usr/lib64/ganglia/moddisk.so.broken gmond is running fine (with some missing info of course).
I can't reproduce localy unfortunately :( Would it be possible to install the debuginfo packages and try again from inside gdb?
Hm, strange I can't reproduce any longer. From yum history I see that glibc has been updated to glibc-2.13.90-7.x86_64. Do you use glibc-2.13.90-7 too?
I tested on a VM running Fedora 15 Alpha x86_64 and couldn't reproduce it either. Also using glibc-2.13.90-7.x86_64.
I tested with glibc-2.13.90-6 actually. We can't rule out a stack overflow somewhere but until it shows up again I'll put it down as a "random" rawhide failure.
Yeah, you can close it, I will reopen if I see the problem again. Thanks for the quick reply.
Reopening since I just saw the problem. (gdb) where #0 0x0000003d9cc362c5 in raise () from /lib64/libc.so.6 #1 0x0000003d9cc37bdb in abort () from /lib64/libc.so.6 #2 0x0000003d9cc722c3 in __libc_message () from /lib64/libc.so.6 #3 0x0000003d9ccf7a87 in __fortify_fail () from /lib64/libc.so.6 #4 0x0000003d9ccf7a50 in __stack_chk_fail () from /lib64/libc.so.6 #5 0x00007ffff1d02ef0 in find_disk_space (total_size=0x7fffffffe2e8, total_free=0x7fffffffe2e0) at metrics.c:1264 #6 0x00007ffff1d02f40 in disk_total_func () at metrics.c:1289 #7 0x00007ffff1d005aa in disk_metric_handler (metric_index=<optimized out>) at mod_disk.c:36 #8 disk_metric_handler (metric_index=<optimized out>) at mod_disk.c:27 Unfortunately after I setup some breakpoints and tried to find where it dies it magically fixed itself! My bet for the cause is the sscanf in find_disk_space. I can't see anything else that could overwrite the stack in the function. sscanf(procline, "%s %s %s %s ", device, mount, type, mode);
Got it, here is an entry from /proc/mounts (which was getting expired and removed by the automounter everytime I was trying to debug the problem, so I missed it the first few times) foobar:/a/b/c /vols/c nfs rw,nosuid,nodev,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.10.10,mountvers=3,mountport=22194,mountproto=udp,local_lock=none,addr=10.10.10.10 char mount[128], device[128], type[32], mode[128]; rc=sscanf(procline, "%s %s %s %s ", device, mount, type, mode); Not f15 specific it seems, it's just better at detecting stack corruptions.
Any fix available for this? It's beginning to annoy me a little :-)
ganglia-3.1.7-4.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/ganglia-3.1.7-4.fc15
Package ganglia-3.1.7-4.fc15: * should fix your issue, * was pushed to the Fedora 15 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing ganglia-3.1.7-4.fc15' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/ganglia-3.1.7-4.fc15 then log in and leave karma (feedback).
ganglia-3.1.7-4.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.