Bug 1258149 - Data Tiering:sqlite db database lock and crash while querying on tier volume bricks while IO was going on
Data Tiering:sqlite db database lock and crash while querying on tier volume ...
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: tiering (Show other bugs)
3.7.5
Unspecified Unspecified
urgent Severity urgent
: ---
: ---
Assigned To: Joseph Elwin Fernandes
bugs@gluster.org
: Triaged
Depends On:
Blocks: 1260923
  Show dependency treegraph
 
Reported: 2015-08-29 12:41 EDT by nchilaka
Modified: 2016-06-19 20:01 EDT (History)
3 users (show)

See Also:
Fixed In Version: glusterfs-3.7.6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-14 01:58:36 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description nchilaka 2015-08-29 12:41:43 EDT
Description of problem:
======================
On a hot replica pair over EC volume, While I was doing IOs on fuse mount(quite a lot of mp3 files), I issued a sql query.
This crashed the database as below on one node
==>>>>>>>>>>>>>>>>>>>>>>>>>> COLD BRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==
efebba2e-e22b-494e-b4ed-a031878219fa|0|0|0|0|0|0|0|0|1|1
71ded0ac-64ed-47da-a327-136fab01c4c1|0|0|0|0|0|0|0|0|1|1
6774b9d1-8486-4a33-ba66-7da9af41a8e7|0|0|0|0|0|0|0|0|1|1
7d6fbf0a-5043-418c-99af-2668ff6781d0|0|0|0|0|0|0|0|0|1|1
3885ff39-2616-481a-ad8b-81d2f40af4d1|0|0|0|0|0|0|0|0|1|1
efebba2e-e22b-494e-b4ed-a031878219fa|00000000-0000-0000-0000-000000000001|f1|/f1|0|0
71ded0ac-64ed-47da-a327-136fab01c4c1|00000000-0000-0000-0000-000000000001|f2|/f2|0|0
6774b9d1-8486-4a33-ba66-7da9af41a8e7|00000000-0000-0000-0000-000000000001|f3|/f3|0|0
7d6fbf0a-5043-418c-99af-2668ff6781d0|00000000-0000-0000-0000-000000000001|f4|/f4|0|0
3885ff39-2616-481a-ad8b-81d2f40af4d1|00000000-0000-0000-0000-000000000001|f5|/f5|0|0
==>>>>>>>>>>>>>>>>>>>>>>>>>> HOTBRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==
Error: near line 1: database is locked

Crash was seen:
[2015-08-29 21:38:00.172183] E [MSGID: 109037] [tier.c:194:tier_migrate_using_query_file] 0-ecvol-tier-dht: failed parsing Ooh

[2015-08-29 21:38:00.172196] E [MSGID: 109037] [tier.c:194:tier_migrate_using_query_file] 0-ecvol-tier-dht: failed parsing La

[2015-08-29 21:38:00.172209] E [MSGID: 109037] [tier.c:194:tier_migrate_using_query_file] 0-ecvol-tier-dht: failed parsing Laa

pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 
2015-08-29 21:38:00
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.3
/usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb6)[0x3560c25936]
/usr/lib64/libglusterfs.so.0(gf_print_trace+0x32f)[0x3560c4549f]
/lib64/libc.so.6[0x340e8326a0]
/lib64/libc.so.6(gsignal+0x35)[0x340e832625]
/lib64/libc.so.6(abort+0x175)[0x340e833e05]
/lib64/libc.so.6[0x340e870537]
/lib64/libc.so.6(__fortify_fail+0x37)[0x340e9025d7]
/lib64/libc.so.6[0x340e9004c0]
/usr/lib64/glusterfs/3.7.3/xlator/cluster/tier.so(+0x55110)[0x7f92bfb87110]
/usr/lib64/glusterfs/3.7.3/xlator/cluster/tier.so(+0x55ebf)[0x7f92bfb87ebf]
/lib64/libpthread.so.0[0x340ec07a51]
/lib64/libc.so.6(clone+0x6d)[0x340e8e89ad]
---------
(END) 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.mounted a replica over ec volume(already mounted on nfs) using fuse
2.did some IOs
3.issued a sql query on both nodes

4. crash was seen after database lock was seen on one node

[root@nag-manual-node1 ~]#  echo "===========Date=====================";date;echo "================================ ColdBrick#1==============================" ;  echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick1/ecvol/.glusterfs/ecvol.db;echo "===================== ColdBrick#2 ======================" ;  echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick2/ecvol/.glusterfs/ecvol.db;echo "==>>>>>>>>>>>>>>>>>>>>>>>>>> COLD BRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==" ;  echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick3/ecvol/.glusterfs/ecvol.db ;echo "==>>>>>>>>>>>>>>>>>>>>>>>>>> HOTBRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==" ;  echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick4/ecvolhot/.glusterfs/ecvolhot.db 
===========Date=====================
Sun Aug 30 03:06:23 IST 2015

sosreports will be updated
Comment 1 nchilaka 2015-08-29 12:50:39 EDT
sosreports @ rhsqe-repo bug.1258149]# pwd
/home/repo/sosreports/bug.1258149
Comment 2 Joseph Elwin Fernandes 2016-02-14 01:58:36 EST
https://bugzilla.redhat.com/show_bug.cgi?id=1270123 fixes this issue

Note You need to log in before you can comment on or make changes to this bug.