Bug 1260730 - Database locking due to write contention between CTR sql connection and tier migrator sql connection
Summary: Database locking due to write contention between CTR sql connection and tier ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: tiering
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Dan Lambright
QA Contact: bugs@gluster.org
URL:
Whiteboard:
Depends On:
Blocks: 1260923 1261039 1261140 1262341
TreeView+ depends on / blocked
 
Reported: 2015-09-07 14:24 UTC by Joseph Elwin Fernandes
Modified: 2016-06-20 00:01 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.8rc2
Clone Of:
: 1261039 1261140 1262341 (view as bug list)
Environment:
Last Closed: 2016-06-16 13:35:09 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Joseph Elwin Fernandes 2015-09-07 14:24:31 UTC
Description of problem:
The DB on the brick is been accessed by CTR, for write and
tier migrator, for read and write. The write from tier migrator is reseting
the heat counters after a cycle. Since we are using sqlite, two connections
trying to write would cause a db lock contention. As a result CTR used to fail
to update the db.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Vijay Bellur 2015-09-07 14:27:02 UTC
REVIEW: http://review.gluster.org/12031 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#5) for review on master by Joseph Fernandes

Comment 2 Vijay Bellur 2015-09-08 12:13:06 UTC
COMMIT: http://review.gluster.org/12031 committed in master by Dan Lambright (dlambrig) 
------
commit 96af474045c9ba5ab74ca76daa823d91a0a0c610
Author: Joseph Fernandes <josferna>
Date:   Thu Aug 27 17:23:07 2015 +0530

    tier/ctr: Solving DB Lock issue due to write contention from db connections
    
    Problem: The DB on the brick is been accessed by CTR, for write and
    tier migrator, for read and write. The write from tier migrator is reseting
    the heat counters after a cycle. Since we are using sqlite, two connections
    trying to write would cause a db lock contention. As a result CTR used to fail
    to update the db.
    
    Solution: Using the same db connection of CTR for reseting the heat counters.
    1) Introducted a new IPC FOP for CTR
    2) After the query do a ipc syncop to the underlying client xlator associated
       to the brick.
    3) CTR in brick will catch the IPC FOP and cleat the heat counters.
    
    Change-Id: I53306bfc08dcdba479deb4ccc154896521336150
    BUG: 1260730
    Signed-off-by: Joseph Fernandes <josferna>
    Reviewed-on: http://review.gluster.org/12031
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Dan Lambright <dlambrig>

Comment 3 Dan Lambright 2015-09-10 19:23:47 UTC
reopening in order to fix freebsd bug 


> /usr/home/jenkins/root/workspace/freebsd-smoke/libglusterfs/src/gfdb/gfdb_sqlite3.h:15:10: fatal error: 'sqlite3.h' file not found
> 
> https://build.gluster.org/job/freebsd-smoke/9859/console
>

Comment 4 Dan Lambright 2015-09-10 19:23:47 UTC
reopening in order to fix freebsd bug 


> /usr/home/jenkins/root/workspace/freebsd-smoke/libglusterfs/src/gfdb/gfdb_sqlite3.h:15:10: fatal error: 'sqlite3.h' file not found
> 
> https://build.gluster.org/job/freebsd-smoke/9859/console
>

Comment 5 Vijay Bellur 2015-09-10 19:28:00 UTC
REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#1) for review on master by Dan Lambright (dlambrig)

Comment 6 Vijay Bellur 2015-09-11 12:28:50 UTC
REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#2) for review on master by Dan Lambright (dlambrig)

Comment 7 Vijay Bellur 2015-09-11 13:50:03 UTC
REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#3) for review on master by Dan Lambright (dlambrig)

Comment 8 Vijay Bellur 2015-09-11 17:15:01 UTC
REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#4) for review on master by Dan Lambright (dlambrig)

Comment 9 Vijay Bellur 2015-09-11 23:08:14 UTC
REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#5) for review on master by Dan Lambright (dlambrig)

Comment 10 Vijay Bellur 2015-09-12 00:13:26 UTC
COMMIT: http://review.gluster.org/12155 committed in master by Dan Lambright (dlambrig) 
------
commit c90c03e9a22d4e21ac7570482255ab42d7ff188e
Author: Dan Lambright <dlambrig>
Date:   Thu Sep 10 15:25:55 2015 -0400

    cluster/tier fix bug with sql includes introduced by 12031
    
    We accidentally introduced a bug where client translators have a
    dependency on sql. This broke freebsd smoke tests. Fix is to
    abstract from the client those dependencies.
    
    Change-Id: I7152573a489bacc8f32e6eb139f9ff4408288f5b
    BUG: 1260730
    Signed-off-by: Dan Lambright <dlambrig>
    Reviewed-on: http://review.gluster.org/12155
    Tested-by: NetBSD Build System <jenkins.org>

Comment 11 Nagaprasad Sathyanarayana 2015-10-25 14:44:52 UTC
Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.

Comment 12 Niels de Vos 2016-06-16 13:35:09 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.