Bug 1258149 - Data Tiering:sqlite db database lock and crash while querying on tier volume bricks while IO was going on
Summary: Data Tiering:sqlite db database lock and crash while querying on tier volume ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: tiering
Version: 3.7.5
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
Assignee: Joseph Elwin Fernandes
QA Contact: bugs@gluster.org
URL:
Whiteboard:
Depends On:
Blocks: 1260923
TreeView+ depends on / blocked
 
Reported: 2015-08-29 16:41 UTC by Nag Pavan Chilakam
Modified: 2016-06-20 00:01 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.7.6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-02-14 06:58:36 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Nag Pavan Chilakam 2015-08-29 16:41:43 UTC
Description of problem:
======================
On a hot replica pair over EC volume, While I was doing IOs on fuse mount(quite a lot of mp3 files), I issued a sql query.
This crashed the database as below on one node
==>>>>>>>>>>>>>>>>>>>>>>>>>> COLD BRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==
efebba2e-e22b-494e-b4ed-a031878219fa|0|0|0|0|0|0|0|0|1|1
71ded0ac-64ed-47da-a327-136fab01c4c1|0|0|0|0|0|0|0|0|1|1
6774b9d1-8486-4a33-ba66-7da9af41a8e7|0|0|0|0|0|0|0|0|1|1
7d6fbf0a-5043-418c-99af-2668ff6781d0|0|0|0|0|0|0|0|0|1|1
3885ff39-2616-481a-ad8b-81d2f40af4d1|0|0|0|0|0|0|0|0|1|1
efebba2e-e22b-494e-b4ed-a031878219fa|00000000-0000-0000-0000-000000000001|f1|/f1|0|0
71ded0ac-64ed-47da-a327-136fab01c4c1|00000000-0000-0000-0000-000000000001|f2|/f2|0|0
6774b9d1-8486-4a33-ba66-7da9af41a8e7|00000000-0000-0000-0000-000000000001|f3|/f3|0|0
7d6fbf0a-5043-418c-99af-2668ff6781d0|00000000-0000-0000-0000-000000000001|f4|/f4|0|0
3885ff39-2616-481a-ad8b-81d2f40af4d1|00000000-0000-0000-0000-000000000001|f5|/f5|0|0
==>>>>>>>>>>>>>>>>>>>>>>>>>> HOTBRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==
Error: near line 1: database is locked

Crash was seen:
[2015-08-29 21:38:00.172183] E [MSGID: 109037] [tier.c:194:tier_migrate_using_query_file] 0-ecvol-tier-dht: failed parsing Ooh

[2015-08-29 21:38:00.172196] E [MSGID: 109037] [tier.c:194:tier_migrate_using_query_file] 0-ecvol-tier-dht: failed parsing La

[2015-08-29 21:38:00.172209] E [MSGID: 109037] [tier.c:194:tier_migrate_using_query_file] 0-ecvol-tier-dht: failed parsing Laa

pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 
2015-08-29 21:38:00
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.3
/usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb6)[0x3560c25936]
/usr/lib64/libglusterfs.so.0(gf_print_trace+0x32f)[0x3560c4549f]
/lib64/libc.so.6[0x340e8326a0]
/lib64/libc.so.6(gsignal+0x35)[0x340e832625]
/lib64/libc.so.6(abort+0x175)[0x340e833e05]
/lib64/libc.so.6[0x340e870537]
/lib64/libc.so.6(__fortify_fail+0x37)[0x340e9025d7]
/lib64/libc.so.6[0x340e9004c0]
/usr/lib64/glusterfs/3.7.3/xlator/cluster/tier.so(+0x55110)[0x7f92bfb87110]
/usr/lib64/glusterfs/3.7.3/xlator/cluster/tier.so(+0x55ebf)[0x7f92bfb87ebf]
/lib64/libpthread.so.0[0x340ec07a51]
/lib64/libc.so.6(clone+0x6d)[0x340e8e89ad]
---------
(END) 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.mounted a replica over ec volume(already mounted on nfs) using fuse
2.did some IOs
3.issued a sql query on both nodes

4. crash was seen after database lock was seen on one node

[root@nag-manual-node1 ~]#  echo "===========Date=====================";date;echo "================================ ColdBrick#1==============================" ;  echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick1/ecvol/.glusterfs/ecvol.db;echo "===================== ColdBrick#2 ======================" ;  echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick2/ecvol/.glusterfs/ecvol.db;echo "==>>>>>>>>>>>>>>>>>>>>>>>>>> COLD BRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==" ;  echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick3/ecvol/.glusterfs/ecvol.db ;echo "==>>>>>>>>>>>>>>>>>>>>>>>>>> HOTBRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==" ;  echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick4/ecvolhot/.glusterfs/ecvolhot.db 
===========Date=====================
Sun Aug 30 03:06:23 IST 2015

sosreports will be updated

Comment 1 Nag Pavan Chilakam 2015-08-29 16:50:39 UTC
sosreports @ rhsqe-repo bug.1258149]# pwd
/home/repo/sosreports/bug.1258149

Comment 2 Joseph Elwin Fernandes 2016-02-14 06:58:36 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1270123 fixes this issue


Note You need to log in before you can comment on or make changes to this bug.