Bug 1219608

Summary:	IO touched a file undergoing migration fails for tiered volumes
Product:	[Community] GlusterFS	Reporter:	Dan Lambright <dlambrig>
Component:	tiering	Assignee:	bugs <bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:	bugs <bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.7.0	CC:	bugs, josferna, nchilaka, nlevinki, rhs-bugs, rwheeler, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1214048	Environment:
Last Closed:	2015-05-14 17:27:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1214048
Bug Blocks:	1186580, 1199352, 1212084, 1260923

Description Dan Lambright 2015-05-07 18:51:31 UTC

+++ This bug was initially created as a clone of Bug #1214048 +++

Description of problem:

If a read IO occurs against a file that has reached rebalance phase 2, we redirect the IO to the destination. For tiered volumes, when we try to reopen the file (on the destination), the lower level DHT receives the open call and fails; it does not have a "cached subvol". Fix it to "teach" the lower level DHT of the new location by sending a locate first in this case.

--- Additional comment from Anand Avati on 2015-04-21 16:52:13 EDT ---

REVIEW: http://review.gluster.org/10324 (uster/dht: tiered volumes may not allow access to files undergoing migration [WIP]) posted (#1) for review on master by Dan Lambright (dlambrig)

--- Additional comment from Anand Avati on 2015-04-22 06:16:33 EDT ---

REVIEW: http://review.gluster.org/10324 (guster/dht: tiered volumes may not allow access to files undergoing migration [WIP]) posted (#2) for review on master by Dan Lambright (dlambrig)

--- Additional comment from Anand Avati on 2015-04-23 12:16:04 EDT ---

REVIEW: http://review.gluster.org/10324 (guster/dht: tiered volumes may not allow access to files undergoing migration) posted (#3) for review on master by Dan Lambright (dlambrig)

--- Additional comment from Anand Avati on 2015-04-29 15:39:19 EDT ---

REVIEW: http://review.gluster.org/10324 (guster/dht: tiered volumes may not allow access to files undergoing migration) posted (#4) for review on master by Dan Lambright (dlambrig)

--- Additional comment from Anand Avati on 2015-05-04 10:38:27 EDT ---

REVIEW: http://review.gluster.org/10324 (guster/dht: tiered volumes may not allow access to files undergoing migration) posted (#5) for review on master by Dan Lambright (dlambrig)

--- Additional comment from Anand Avati on 2015-05-05 12:46:32 EDT ---

COMMIT: http://review.gluster.org/10324 committed in master by Raghavendra G (rgowdapp) 
------
commit 8965ac78bfd26e7f45c9579894191817afafa15e
Author: Dan Lambright <dlambrig>
Date:   Wed Apr 29 15:35:56 2015 -0400

    guster/dht: tiered volumes may not allow access to files undergoing migration
    
    If a read IO occurs against a file that has reached rebalance
    phase 2, we redirect the IO to the destination. For tiered
    volumes, when we try to reopen the file (on the destination),
    the lower level DHT receives the open call and fails; it does
    not have a "cached subvol". Fix is to "teach" the lower level
    DHT of the new location by sending a locate before the open.
    
    Change-Id: Ia4acb0035ff1da15f6a8f9ed54f43c76e8b98f5f
    BUG: 1214048
    Signed-off-by: Dan Lambright <dlambrig>
    Signed-off-by: root <root.lab.eng.bos.redhat.com>
    Signed-off-by: Dan Lambright <dlambrig>
    Reviewed-on: http://review.gluster.org/10324
    Tested-by: NetBSD Build System
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>
    Tested-by: Raghavendra G <rgowdapp>

Comment 1 Anand Avati 2015-05-08 08:55:46 UTC

COMMIT: http://review.gluster.org/10654 committed in release-3.7 by Vijay Bellur (vbellur) 
------
commit eee703473f17e3784e15dba291cc3df00fde9f96
Author: Dan Lambright <dlambrig>
Date:   Thu May 7 14:59:51 2015 -0400

    guster/dht: tiered volumes may not allow access to files undergoing migration
    
    This is a backport of fix 10324 to Gluster 3.7.
    
    If a read IO occurs against a file that has reached rebalance
    phase 2, we redirect the IO to the destination. For tiered
    volumes, when we try to reopen the file (on the destination),
    the lower level DHT receives the open call and fails; it does
    not have a "cached subvol". Fix is to "teach" the lower level
    DHT of the new location by sending a locate before the open.
    
    > http://review.gluster.org/#/c/10324/
    > Change-Id: Ia4acb0035ff1da15f6a8f9ed54f43c76e8b98f5f
    > BUG: 1214048
    > Signed-off-by: Dan Lambright <dlambrig>
    > Signed-off-by: root <root.lab.eng.bos.redhat.com>
    > Signed-off-by: Dan Lambright <dlambrig>
    > Reviewed-on: http://review.gluster.org/10324
    > Tested-by: NetBSD Build System
    > Tested-by: Gluster Build System <jenkins.com>
    > Reviewed-by: Raghavendra G <rgowdapp>
    > Tested-by: Raghavendra G <rgowdapp>
    > Signed-off-by: Dan Lambright <dlambrig>
    
    Change-Id: Ia4acb0035ff1da15f6a8f9ed54f43c76e8b98f5f
    BUG: 1219608
    Signed-off-by: Dan Lambright <dlambrig>
    Reviewed-on: http://review.gluster.org/10654
    Tested-by: Gluster Build System <jenkins.com>
    Tested-by: NetBSD Build System
    Reviewed-by: Joseph Fernandes
    Reviewed-by: Vijay Bellur <vbellur>

Comment 2 Niels de Vos 2015-05-14 17:27:35 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 3 Niels de Vos 2015-05-14 17:28:59 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 4 Niels de Vos 2015-05-14 17:35:21 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user