Bug 1262341 - Database locking due to write contention between CTR sql connection and tier migrator sql connection
Database locking due to write contention between CTR sql connection and tier ...
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: tiering (Show other bugs)
3.7.5
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: bugs@gluster.org
bugs@gluster.org
: Triaged
: 1261140 (view as bug list)
Depends On: 1260730
Blocks: 1260923 1261039 1261140
  Show dependency treegraph
 
Reported: 2015-09-11 08:52 EDT by Dan Lambright
Modified: 2015-10-30 13:32 EDT (History)
3 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1260730
Environment:
Last Closed: 2015-10-14 06:29:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan Lambright 2015-09-11 08:52:37 EDT
+++ This bug was initially created as a clone of Bug #1260730 +++

Description of problem:
The DB on the brick is been accessed by CTR, for write and
tier migrator, for read and write. The write from tier migrator is reseting
the heat counters after a cycle. Since we are using sqlite, two connections
trying to write would cause a db lock contention. As a result CTR used to fail
to update the db.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Vijay Bellur on 2015-09-07 10:27:02 EDT ---

REVIEW: http://review.gluster.org/12031 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#5) for review on master by Joseph Fernandes

--- Additional comment from Vijay Bellur on 2015-09-08 08:13:06 EDT ---

COMMIT: http://review.gluster.org/12031 committed in master by Dan Lambright (dlambrig@redhat.com) 
------
commit 96af474045c9ba5ab74ca76daa823d91a0a0c610
Author: Joseph Fernandes <josferna@redhat.com>
Date:   Thu Aug 27 17:23:07 2015 +0530

    tier/ctr: Solving DB Lock issue due to write contention from db connections
    
    Problem: The DB on the brick is been accessed by CTR, for write and
    tier migrator, for read and write. The write from tier migrator is reseting
    the heat counters after a cycle. Since we are using sqlite, two connections
    trying to write would cause a db lock contention. As a result CTR used to fail
    to update the db.
    
    Solution: Using the same db connection of CTR for reseting the heat counters.
    1) Introducted a new IPC FOP for CTR
    2) After the query do a ipc syncop to the underlying client xlator associated
       to the brick.
    3) CTR in brick will catch the IPC FOP and cleat the heat counters.
    
    Change-Id: I53306bfc08dcdba479deb4ccc154896521336150
    BUG: 1260730
    Signed-off-by: Joseph Fernandes <josferna@redhat.com>
    Reviewed-on: http://review.gluster.org/12031
    Tested-by: NetBSD Build System <jenkins@build.gluster.org>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Dan Lambright <dlambrig@redhat.com>
    Tested-by: Dan Lambright <dlambrig@redhat.com>

--- Additional comment from Dan Lambright on 2015-09-10 15:23:47 EDT ---

reopening in order to fix freebsd bug 


> /usr/home/jenkins/root/workspace/freebsd-smoke/libglusterfs/src/gfdb/gfdb_sqlite3.h:15:10: fatal error: 'sqlite3.h' file not found
> 
> https://build.gluster.org/job/freebsd-smoke/9859/console
>

--- Additional comment from Dan Lambright on 2015-09-10 15:23:47 EDT ---

reopening in order to fix freebsd bug 


> /usr/home/jenkins/root/workspace/freebsd-smoke/libglusterfs/src/gfdb/gfdb_sqlite3.h:15:10: fatal error: 'sqlite3.h' file not found
> 
> https://build.gluster.org/job/freebsd-smoke/9859/console
>

--- Additional comment from Vijay Bellur on 2015-09-10 15:28:00 EDT ---

REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#1) for review on master by Dan Lambright (dlambrig@redhat.com)

--- Additional comment from Vijay Bellur on 2015-09-11 08:28:50 EDT ---

REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#2) for review on master by Dan Lambright (dlambrig@redhat.com)
Comment 1 Vijay Bellur 2015-09-11 08:59:08 EDT
REVIEW: http://review.gluster.org/12161 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#1) for review on release-3.7 by Dan Lambright (dlambrig@redhat.com)
Comment 2 Vijay Bellur 2015-09-11 09:36:58 EDT
REVIEW: http://review.gluster.org/12161 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#2) for review on release-3.7 by Dan Lambright (dlambrig@redhat.com)
Comment 3 Vijay Bellur 2015-09-11 09:49:58 EDT
REVIEW: http://review.gluster.org/12161 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#3) for review on release-3.7 by Dan Lambright (dlambrig@redhat.com)
Comment 4 Dan Lambright 2015-09-12 16:50:21 EDT
*** Bug 1261140 has been marked as a duplicate of this bug. ***
Comment 5 Pranith Kumar K 2015-10-14 06:29:23 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-glusterfs-3.7.5, please open a new bug report.

glusterfs-glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
Comment 6 Pranith Kumar K 2015-10-14 06:38:21 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.5, please open a new bug report.

glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.