Bug 1273272

Summary: tiering + nfs-ganesha: tiering has a segfault
Product: [Community] GlusterFS Reporter: Saurabh <saujain>
Component: tieringAssignee: sankarshan <sankarshan>
Status: CLOSED EOL QA Contact: bugs <bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.7.5CC: bugs, jthottan, kkeithle, mzywusko, nbalacha, ndevos, skoduri
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1273385 (view as bug list) Environment:
Last Closed: 2017-03-08 10:50:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1273385    
Attachments:
Description Flags
coredump from vm1 none

Description Saurabh 2015-10-20 06:34:32 UTC
Description of problem:
While executing the fs-sanity test suite, the tiering volume throws a segfault

Version-Release number of selected component (if applicable):
glusterfs-3.7.5-1.el7.x86_64
nfs-ganesha-2.3-0.rc6.el7.centos.x86_64

How reproducible:
happen to be first time

Steps to Reproduce:
1. create volume of dist-rep type with tiering enabled.
2. export the volume for nfs-ganesha and mount it with vers=4
3. start executing the fs-sanity test suite

Actual results:
[2015-10-19 16:14:14.701640] E [MSGID: 109037] [tier.c:951:tier_start] 0-vol2-tier-dht: Demotion failed!
[2015-10-19 16:16:00.720267] I [MSGID: 109038] [tier.c:586:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove /var/run/gluster/vol2-tier-dht/demotequeryfile-vol2-tier-dht
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2015-10-19 16:16:00
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.5
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7fc0dab7c002]
/lib64/libglusterfs.so.0(gf_print_trace+0x31d)[0x7fc0dab9848d]
/lib64/libc.so.6(+0x35650)[0x7fc0d926a650]
/lib64/libc.so.6(+0x86811)[0x7fc0d92bb811]
/usr/lib64/libgfdb.so.0(gf_sql_query_function+0x101)[0x7fc0cc3e4001]
/usr/lib64/libgfdb.so.0(gf_sqlite3_find_unchanged_for_time+0xd0)[0x7fc0cc3e55d0]
/usr/lib64/libgfdb.so.0(find_unchanged_for_time+0x41)[0x7fc0cc3dfbd1]
/usr/lib64/glusterfs/3.7.5/xlator/cluster/tier.so(+0x57947)[0x7fc0cca0f947]
/lib64/libpthread.so.0(+0x7df5)[0x7fc0d99e4df5]
/lib64/libc.so.6(clone+0x6d)[0x7fc0d932b1ad]
---------
(END)



(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/x86_64/strlen.S:31
#1  0x00007fc0cc3e4001 in gf_sql_query_function (prep_stmt=0x7fc0a042b288, query_callback=query_callback@entry=0x7fc0cca0d640 <tier_gf_query_callback>, _query_cbk_args=_query_cbk_args@entry=0x7fc0b17f9e70)
    at gfdb_sqlite3_helper.c:1157
#2  0x00007fc0cc3e55d0 in gf_sqlite3_find_unchanged_for_time (db_conn=0x7fc0a0445440, query_callback=0x7fc0cca0d640 <tier_gf_query_callback>, query_cbk_args=0x7fc0b17f9e70, for_time=0x7fc0b17f9e60)
    at gfdb_sqlite3.c:809
#3  0x00007fc0cc3dfbd1 in find_unchanged_for_time (_conn_node=<optimized out>, query_callback=0x7fc0cca0d640 <tier_gf_query_callback>, _query_cbk_args=0x7fc0b17f9e70, for_time=0x7fc0b17f9e60)
    at gfdb_data_store.c:506
#4  0x00007fc0cca0f947 in tier_process_brick_cbk (args=<synthetic pointer>, local_brick=0x7fc0b8003680) at tier.c:501
#5  tier_build_migration_qfile (is_promotion=_gf_false, query_cbk_args=0x7fc0b17f9e70, args=0x7fc0c501ecc0) at tier.c:606
#6  tier_demote (args=0x7fc0c501ecc0) at tier.c:666
#7  0x00007fc0d99e4df5 in start_thread (arg=0x7fc0b17fa700) at pthread_create.c:308
#8  0x00007fc0d932b1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113


# gluster volume status vol2
Status of volume: vol2
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.46.60:/rhs/brick2/cold-d2r21    49158     0          Y       16128
Brick 10.70.46.63:/rhs/brick2/cold-d2r11    49159     0          Y       30149
Brick 10.70.46.64:/rhs/brick2/cold-d1r21    49159     0          Y       23716
Brick 10.70.46.59:/rhs/brick2/cold-d1r11    49159     0          Y       19532
Cold Bricks:
Brick 10.70.46.59:/rhs/brick1/d1r11         49158     0          Y       19412
Brick 10.70.46.64:/rhs/brick1/d1r21         49158     0          Y       23610
Brick 10.70.46.63:/rhs/brick1/d2r11         49158     0          Y       30053
Brick 10.70.46.60:/rhs/brick1/d2r21         49157     0          Y       16026
 
Task Status of Volume vol2
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : 1dd3374a-f0d5-4aa0-8a79-c1a2776ed72e
Status               : in progress         



Expected results:
no coredump expected

Additional info:
Not very which test in the test-suite caused the issue

Comment 1 Saurabh 2015-10-20 06:39:05 UTC
Created attachment 1084611 [details]
coredump from vm1

Comment 5 Kaushal 2017-03-08 10:50:28 UTC
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life.

Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS.
If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.