Bug 1262341

Summary:	Database locking due to write contention between CTR sql connection and tier migrator sql connection
Product:	[Community] GlusterFS	Reporter:	Dan Lambright <dlambrig>
Component:	tiering	Assignee:	bugs <bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:	bugs <bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.7.5	CC:	bugs, dlambrig, josferna
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.5	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1260730	Environment:
Last Closed:	2015-10-14 10:29:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1260730
Bug Blocks:	1260923, 1261039, 1261140

Description Dan Lambright 2015-09-11 12:52:37 UTC

+++ This bug was initially created as a clone of Bug #1260730 +++

Description of problem:
The DB on the brick is been accessed by CTR, for write and
tier migrator, for read and write. The write from tier migrator is reseting
the heat counters after a cycle. Since we are using sqlite, two connections
trying to write would cause a db lock contention. As a result CTR used to fail
to update the db.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Vijay Bellur on 2015-09-07 10:27:02 EDT ---

REVIEW: http://review.gluster.org/12031 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#5) for review on master by Joseph Fernandes

--- Additional comment from Vijay Bellur on 2015-09-08 08:13:06 EDT ---

COMMIT: http://review.gluster.org/12031 committed in master by Dan Lambright (dlambrig) 
------
commit 96af474045c9ba5ab74ca76daa823d91a0a0c610
Author: Joseph Fernandes <josferna>
Date:   Thu Aug 27 17:23:07 2015 +0530

    tier/ctr: Solving DB Lock issue due to write contention from db connections
    
    Problem: The DB on the brick is been accessed by CTR, for write and
    tier migrator, for read and write. The write from tier migrator is reseting
    the heat counters after a cycle. Since we are using sqlite, two connections
    trying to write would cause a db lock contention. As a result CTR used to fail
    to update the db.
    
    Solution: Using the same db connection of CTR for reseting the heat counters.
    1) Introducted a new IPC FOP for CTR
    2) After the query do a ipc syncop to the underlying client xlator associated
       to the brick.
    3) CTR in brick will catch the IPC FOP and cleat the heat counters.
    
    Change-Id: I53306bfc08dcdba479deb4ccc154896521336150
    BUG: 1260730
    Signed-off-by: Joseph Fernandes <josferna>
    Reviewed-on: http://review.gluster.org/12031
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Dan Lambright <dlambrig>

--- Additional comment from Dan Lambright on 2015-09-10 15:23:47 EDT ---

reopening in order to fix freebsd bug 


> /usr/home/jenkins/root/workspace/freebsd-smoke/libglusterfs/src/gfdb/gfdb_sqlite3.h:15:10: fatal error: 'sqlite3.h' file not found
> 
> https://build.gluster.org/job/freebsd-smoke/9859/console
>

--- Additional comment from Dan Lambright on 2015-09-10 15:23:47 EDT ---

reopening in order to fix freebsd bug 


> /usr/home/jenkins/root/workspace/freebsd-smoke/libglusterfs/src/gfdb/gfdb_sqlite3.h:15:10: fatal error: 'sqlite3.h' file not found
> 
> https://build.gluster.org/job/freebsd-smoke/9859/console
>

--- Additional comment from Vijay Bellur on 2015-09-10 15:28:00 EDT ---

REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#1) for review on master by Dan Lambright (dlambrig)

--- Additional comment from Vijay Bellur on 2015-09-11 08:28:50 EDT ---

REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#2) for review on master by Dan Lambright (dlambrig)

Comment 1 Vijay Bellur 2015-09-11 12:59:08 UTC

REVIEW: http://review.gluster.org/12161 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#1) for review on release-3.7 by Dan Lambright (dlambrig)

Comment 2 Vijay Bellur 2015-09-11 13:36:58 UTC

REVIEW: http://review.gluster.org/12161 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#2) for review on release-3.7 by Dan Lambright (dlambrig)

Comment 3 Vijay Bellur 2015-09-11 13:49:58 UTC

REVIEW: http://review.gluster.org/12161 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#3) for review on release-3.7 by Dan Lambright (dlambrig)

Comment 4 Dan Lambright 2015-09-12 20:50:21 UTC

*** Bug 1261140 has been marked as a duplicate of this bug. ***

Comment 5 Pranith Kumar K 2015-10-14 10:29:23 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-glusterfs-3.7.5, please open a new bug report.

glusterfs-glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 6 Pranith Kumar K 2015-10-14 10:38:21 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.5, please open a new bug report.

glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user