Bug 1656316

Can you please provide the complete backtrace (thread apply all bt) of the core from gdb session?

(In reply to Atin Mukherjee from comment #2)
> Can you please provide the complete backtrace (thread apply all bt) of the
> core from gdb session?

Hi Atin,
How can I get the complete backtrace?

Hi Atin,
Can you please explain how to get the complete backetrace?
We're in production enviroment and need to resolv this problem.

Regards

cat /proc/sys/kernel/core_pattern

This will let you know the location of the core file. 

And then do the following to get to the gdb session (you may have to install gdb and glusterfs-debuginfo package for the interim)

gdb glusterd <core file with path> 

t a a bt

The Core is already uploaded as attachment (is was created in the working folder with the pid of the processes ended), 
here's the command's output;

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/glusterfsd...Reading symbols from /usr/lib/debug/usr/sbin/glusterfsd.debug...done.
done.

warning: core file may not match specified executable file.
[New LWP 28964]
[New LWP 28962]
[New LWP 28961]
[New LWP 28965]
[New LWP 28960]
Core was generated by `gluster volume statedump inode'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f6a89786f01 in ?? ()

Any news?

Hi,
I've rerunned the gdb command, I didn't noticed that the command executable was glusterd instead of gluster.
Here's the new output:

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/gluster...Reading symbols from /usr/lib/debug/usr/sbin/gluster.debug...done.
done.
[New LWP 4052]
[New LWP 4049]
[New LWP 4053]
[New LWP 4048]
[New LWP 4050]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `gluster volume statedump gv0'.
Program terminated with signal 11, Segmentation fault.
#0  __strstr_sse2 (haystack_start=0x0, needle_start=0x55c2d1b1063b "nfs") at ../string/strstr.c:63
63	  while (*haystack && *needle)

Hi,
I've rerunned the gdb command, I didn't noticed that the command executable was glusterd instead of gluster.
Here's the new output:

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/gluster...Reading symbols from /usr/lib/debug/usr/sbin/gluster.debug...done.
done.
[New LWP 4052]
[New LWP 4049]
[New LWP 4053]
[New LWP 4048]
[New LWP 4050]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `gluster volume statedump gv0'.
Program terminated with signal 11, Segmentation fault.
#0  __strstr_sse2 (haystack_start=0x0, needle_start=0x55c2d1b1063b "nfs") at ../string/strstr.c:63
63	  while (*haystack && *needle)

Unfortunately I don't have a centos machine at this moment to debug this crash and this is why I was requesting you to install glusterfs-debuginfo package and then do the following:

gdb gluster core.28960
(gdb) thread apply all bt

The backtrace you're pointing is basically a strstr call, but we need to track the gluster functions to see what's wrong here. From the core we can definitely figure out that this is a crash in cli.

I've already installed glusterfs-debuginfo:
rpm -qa |grep glusterfs
glusterfs-4.1.1-1.el7.x86_64
glusterfs-fuse-4.1.1-1.el7.x86_64
glusterfs-debuginfo-4.1.1-1.el7.x86_64
glusterfs-api-4.1.1-1.el7.x86_64
glusterfs-libs-4.1.1-1.el7.x86_64
glusterfs-cli-4.1.1-1.el7.x86_64
glusterfs-server-4.1.1-1.el7.x86_64
glusterfs-client-xlators-4.1.1-1.el7.x86_64


I've launched the thread apply all bt, and this is the complete output:

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/gluster...Reading symbols from /usr/lib/debug/usr/sbin/gluster.debug...done.
done.
[New LWP 28964]
[New LWP 28962]
[New LWP 28961]
[New LWP 28965]
[New LWP 28960]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `gluster volume statedump inode'.
Program terminated with signal 11, Segmentation fault.
#0  __strstr_sse2 (haystack_start=0x0, needle_start=0x557e3786063b "nfs") at ../string/strstr.c:63
63	  while (*haystack && *needle)
(gdb) thread apply all bt

Thread 5 (Thread 0x7f6a8bfab780 (LWP 28960)):
#0  0x00007f6a89f18f47 in pthread_join (threadid=140095378020096, thread_return=thread_return@entry=0x0) at pthread_join.c:90
#1  0x00007f6a8bb186c8 in event_dispatch_epoll (event_pool=0x557e391903e0) at event-epoll.c:750
#2  0x0000557e3780fa53 in main (argc=<optimized out>, argv=0x7fff400439b8) at cli.c:790

Thread 4 (Thread 0x7f6a7f3dd700 (LWP 28965)):
#0  0x00007f6a897e0483 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f6a8bb17f62 in event_dispatch_epoll_worker (data=0x557e391d6370) at event-epoll.c:653
#2  0x00007f6a89f17dd5 in start_thread (arg=0x7f6a7f3dd700) at pthread_create.c:307
#3  0x00007f6a897dfead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f6a894bb700 (LWP 28961)):
#0  0x00007f6a897a6e2d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f6a897a6cc4 in __sleep (seconds=0, seconds@entry=30) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007f6a8badf75d in pool_sweeper (arg=<optimized out>) at mem-pool.c:470
#3  0x00007f6a89f17dd5 in start_thread (arg=0x7f6a894bb700) at pthread_create.c:307
#4  0x00007f6a897dfead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f6a803df700 (LWP 28962)):
#0  0x00007f6a89f1ee3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f6a8bac2bf6 in gf_timer_proc (data=0x557e391d29b0) at timer.c:202
#2  0x00007f6a89f17dd5 in start_thread (arg=0x7f6a803df700) at pthread_create.c:307
#3  0x00007f6a897dfead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f6a7fbde700 (LWP 28964)):
#0  __strstr_sse2 (haystack_start=0x0, needle_start=0x557e3786063b "nfs") at ../string/strstr.c:63
#1  0x0000557e37845eb4 in cli_cmd_volume_statedump_options_parse (words=words@entry=0x7fff400439c0, wordcount=<optimized out>, options=options@entry=0x7f6a7fbdde20) at cli-cmd-parser.c:3604
#2  0x0000557e37814c97 in cli_cmd_volume_statedump_cbk (state=<optimized out>, word=0x557e391d50e0, words=0x7fff400439c0, wordcount=<optimized out>) at cli-cmd-volume.c:2984
#3  0x0000557e378119a3 in cli_cmd_process (state=0x7fff400437c0, argc=3, argv=0x7fff400439c0) at cli-cmd.c:135
#4  0x0000557e37811320 in cli_batch (d=<optimized out>) at input.c:29
#5  0x00007f6a89f17dd5 in start_thread (arg=0x7f6a7fbde700) at pthread_create.c:307
#6  0x00007f6a897dfead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

This is fixed through https://review.gluster.org/#/c/glusterfs/+/20867/ which was one of the several fixes we did as part of coverity defect fixes.

The fix is already available in the release 5 branch but not in release 4 which you're using. Would you mind upgrading to the latest version of Gluster?

We have installed gluster via yum repo:
[centos-gluster41]
name=CentOS-$releasever - Gluster 4.1 (Long Term Maintanance)
baseurl=http://mirror.centos.org/centos/$releasever/storage/$basearch/gluster-4.1/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage

Is this patch already available via this repo?
The latest version available is 4.1.6-1

In case I need to do a manual upgrade, where can I find a procedure? 
We're in production environment, I would like to broke anything

No, the fix is only available from glusterfs-5 release onwards.

If you're looking for upgrade document please refer https://docs.gluster.org/en/latest/Upgrade-Guide/upgrade_to_4.1/ . Even though it says upgrade to 4.1, the same steps would be applicable to upgrade to glusterfs-5.x as well.

Ok, 
We will upgrade the server software, but only after the 10 of January 2019,
We're now in release/upgrade freeze.

I think we could close the ticket now.

Regards