Bug 1273385 - tiering + nfs-ganesha: tiering has a segfault
tiering + nfs-ganesha: tiering has a segfault
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
unspecified
x86_64 Linux
high Severity high
: ---
: RHGS 3.1.2
Assigned To: Joseph Elwin Fernandes
Apeksha
: ZStream
Depends On: 1273272
Blocks: 1260783 1276742
  Show dependency treegraph
 
Reported: 2015-10-20 06:50 EDT by nchilaka
Modified: 2016-09-17 11:44 EDT (History)
11 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-8
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1273272
Environment:
Last Closed: 2016-03-01 00:43:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description nchilaka 2015-10-20 06:50:56 EDT
+++ This bug was initially created as a clone of Bug #1273272 +++

Description of problem:
While executing the fs-sanity test suite, the tiering volume throws a segfault

Version-Release number of selected component (if applicable):
glusterfs-3.7.5-1.el7.x86_64
nfs-ganesha-2.3-0.rc6.el7.centos.x86_64

How reproducible:
happen to be first time

Steps to Reproduce:
1. create volume of dist-rep type with tiering enabled.
2. export the volume for nfs-ganesha and mount it with vers=4
3. start executing the fs-sanity test suite

Actual results:
[2015-10-19 16:14:14.701640] E [MSGID: 109037] [tier.c:951:tier_start] 0-vol2-tier-dht: Demotion failed!
[2015-10-19 16:16:00.720267] I [MSGID: 109038] [tier.c:586:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove /var/run/gluster/vol2-tier-dht/demotequeryfile-vol2-tier-dht
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2015-10-19 16:16:00
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.5
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7fc0dab7c002]
/lib64/libglusterfs.so.0(gf_print_trace+0x31d)[0x7fc0dab9848d]
/lib64/libc.so.6(+0x35650)[0x7fc0d926a650]
/lib64/libc.so.6(+0x86811)[0x7fc0d92bb811]
/usr/lib64/libgfdb.so.0(gf_sql_query_function+0x101)[0x7fc0cc3e4001]
/usr/lib64/libgfdb.so.0(gf_sqlite3_find_unchanged_for_time+0xd0)[0x7fc0cc3e55d0]
/usr/lib64/libgfdb.so.0(find_unchanged_for_time+0x41)[0x7fc0cc3dfbd1]
/usr/lib64/glusterfs/3.7.5/xlator/cluster/tier.so(+0x57947)[0x7fc0cca0f947]
/lib64/libpthread.so.0(+0x7df5)[0x7fc0d99e4df5]
/lib64/libc.so.6(clone+0x6d)[0x7fc0d932b1ad]
---------
(END)



(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/x86_64/strlen.S:31
#1  0x00007fc0cc3e4001 in gf_sql_query_function (prep_stmt=0x7fc0a042b288, query_callback=query_callback@entry=0x7fc0cca0d640 <tier_gf_query_callback>, _query_cbk_args=_query_cbk_args@entry=0x7fc0b17f9e70)
    at gfdb_sqlite3_helper.c:1157
#2  0x00007fc0cc3e55d0 in gf_sqlite3_find_unchanged_for_time (db_conn=0x7fc0a0445440, query_callback=0x7fc0cca0d640 <tier_gf_query_callback>, query_cbk_args=0x7fc0b17f9e70, for_time=0x7fc0b17f9e60)
    at gfdb_sqlite3.c:809
#3  0x00007fc0cc3dfbd1 in find_unchanged_for_time (_conn_node=<optimized out>, query_callback=0x7fc0cca0d640 <tier_gf_query_callback>, _query_cbk_args=0x7fc0b17f9e70, for_time=0x7fc0b17f9e60)
    at gfdb_data_store.c:506
#4  0x00007fc0cca0f947 in tier_process_brick_cbk (args=<synthetic pointer>, local_brick=0x7fc0b8003680) at tier.c:501
#5  tier_build_migration_qfile (is_promotion=_gf_false, query_cbk_args=0x7fc0b17f9e70, args=0x7fc0c501ecc0) at tier.c:606
#6  tier_demote (args=0x7fc0c501ecc0) at tier.c:666
#7  0x00007fc0d99e4df5 in start_thread (arg=0x7fc0b17fa700) at pthread_create.c:308
#8  0x00007fc0d932b1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113


# gluster volume status vol2
Status of volume: vol2
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.46.60:/rhs/brick2/cold-d2r21    49158     0          Y       16128
Brick 10.70.46.63:/rhs/brick2/cold-d2r11    49159     0          Y       30149
Brick 10.70.46.64:/rhs/brick2/cold-d1r21    49159     0          Y       23716
Brick 10.70.46.59:/rhs/brick2/cold-d1r11    49159     0          Y       19532
Cold Bricks:
Brick 10.70.46.59:/rhs/brick1/d1r11         49158     0          Y       19412
Brick 10.70.46.64:/rhs/brick1/d1r21         49158     0          Y       23610
Brick 10.70.46.63:/rhs/brick1/d2r11         49158     0          Y       30053
Brick 10.70.46.60:/rhs/brick1/d2r21         49157     0          Y       16026
 
Task Status of Volume vol2
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : 1dd3374a-f0d5-4aa0-8a79-c1a2776ed72e
Status               : in progress         



Expected results:
no coredump expected

Additional info:
Not very which test in the test-suite caused the issue

--- Additional comment from Saurabh on 2015-10-20 02:39 EDT ---
Comment 3 Vivek Agarwal 2015-11-05 05:06:27 EST
Do not plan to support ganesha with tiering for 3.1.2, removing from the list.
Comment 6 Joseph Elwin Fernandes 2015-11-10 20:33:50 EST
the Code where we are seeing the core dump has changed now, with the following fix

"tier/libgfdb: Replacing ASCII query file with binary"
https://code.engineering.redhat.com/gerrit/61006.

Moving this On QA.
Comment 7 Bhaskarakiran 2015-12-02 05:18:45 EST
pls update the fixed in version
Comment 10 Apeksha 2015-12-21 00:50:13 EST
I ran the fsanity with ganesha on glusterfs-3.7.5-11.el7rhgs.x86_64 with a tiered volume and i dint see any crash, hence marking it as verified.
Comment 12 errata-xmlrpc 2016-03-01 00:43:16 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html

Note You need to log in before you can comment on or make changes to this bug.