Bug 1260730

Summary:	Database locking due to write contention between CTR sql connection and tier migrator sql connection
Product:	[Community] GlusterFS	Reporter:	Joseph Elwin Fernandes <josferna>
Component:	tiering	Assignee:	Dan Lambright <dlambrig>
Status:	CLOSED CURRENTRELEASE	QA Contact:	bugs <bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, dlambrig, sankarshan
Target Milestone:	---	Keywords:	Reopened, Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.8rc2	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1261039 1261140 1262341 (view as bug list)		Environment:
Last Closed:	2016-06-16 13:35:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1260923, 1261039, 1261140, 1262341

Description Joseph Elwin Fernandes 2015-09-07 14:24:31 UTC

Description of problem:
The DB on the brick is been accessed by CTR, for write and
tier migrator, for read and write. The write from tier migrator is reseting
the heat counters after a cycle. Since we are using sqlite, two connections
trying to write would cause a db lock contention. As a result CTR used to fail
to update the db.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Vijay Bellur 2015-09-07 14:27:02 UTC

REVIEW: http://review.gluster.org/12031 (tier/ctr: Solving DB Lock issue due to write contention from db connections) posted (#5) for review on master by Joseph Fernandes

Comment 2 Vijay Bellur 2015-09-08 12:13:06 UTC

COMMIT: http://review.gluster.org/12031 committed in master by Dan Lambright (dlambrig) 
------
commit 96af474045c9ba5ab74ca76daa823d91a0a0c610
Author: Joseph Fernandes <josferna>
Date:   Thu Aug 27 17:23:07 2015 +0530

    tier/ctr: Solving DB Lock issue due to write contention from db connections
    
    Problem: The DB on the brick is been accessed by CTR, for write and
    tier migrator, for read and write. The write from tier migrator is reseting
    the heat counters after a cycle. Since we are using sqlite, two connections
    trying to write would cause a db lock contention. As a result CTR used to fail
    to update the db.
    
    Solution: Using the same db connection of CTR for reseting the heat counters.
    1) Introducted a new IPC FOP for CTR
    2) After the query do a ipc syncop to the underlying client xlator associated
       to the brick.
    3) CTR in brick will catch the IPC FOP and cleat the heat counters.
    
    Change-Id: I53306bfc08dcdba479deb4ccc154896521336150
    BUG: 1260730
    Signed-off-by: Joseph Fernandes <josferna>
    Reviewed-on: http://review.gluster.org/12031
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Dan Lambright <dlambrig>

Comment 3 Dan Lambright 2015-09-10 19:23:47 UTC

reopening in order to fix freebsd bug 


> /usr/home/jenkins/root/workspace/freebsd-smoke/libglusterfs/src/gfdb/gfdb_sqlite3.h:15:10: fatal error: 'sqlite3.h' file not found
> 
> https://build.gluster.org/job/freebsd-smoke/9859/console
>

Comment 4 Dan Lambright 2015-09-10 19:23:47 UTC

reopening in order to fix freebsd bug 


> /usr/home/jenkins/root/workspace/freebsd-smoke/libglusterfs/src/gfdb/gfdb_sqlite3.h:15:10: fatal error: 'sqlite3.h' file not found
> 
> https://build.gluster.org/job/freebsd-smoke/9859/console
>

Comment 5 Vijay Bellur 2015-09-10 19:28:00 UTC

REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#1) for review on master by Dan Lambright (dlambrig)

Comment 6 Vijay Bellur 2015-09-11 12:28:50 UTC

REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#2) for review on master by Dan Lambright (dlambrig)

Comment 7 Vijay Bellur 2015-09-11 13:50:03 UTC

REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#3) for review on master by Dan Lambright (dlambrig)

Comment 8 Vijay Bellur 2015-09-11 17:15:01 UTC

REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#4) for review on master by Dan Lambright (dlambrig)

Comment 9 Vijay Bellur 2015-09-11 23:08:14 UTC

REVIEW: http://review.gluster.org/12155 (cluster/tier fix bug with sql includes introduced by 12031) posted (#5) for review on master by Dan Lambright (dlambrig)

Comment 10 Vijay Bellur 2015-09-12 00:13:26 UTC

COMMIT: http://review.gluster.org/12155 committed in master by Dan Lambright (dlambrig) 
------
commit c90c03e9a22d4e21ac7570482255ab42d7ff188e
Author: Dan Lambright <dlambrig>
Date:   Thu Sep 10 15:25:55 2015 -0400

    cluster/tier fix bug with sql includes introduced by 12031
    
    We accidentally introduced a bug where client translators have a
    dependency on sql. This broke freebsd smoke tests. Fix is to
    abstract from the client those dependencies.
    
    Change-Id: I7152573a489bacc8f32e6eb139f9ff4408288f5b
    BUG: 1260730
    Signed-off-by: Dan Lambright <dlambrig>
    Reviewed-on: http://review.gluster.org/12155
    Tested-by: NetBSD Build System <jenkins.org>

Comment 11 Nagaprasad Sathyanarayana 2015-10-25 14:44:52 UTC

Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.

Comment 12 Niels de Vos 2016-06-16 13:35:09 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user