Bug 1603056

Summary:	When reserve limits are reached, append on an existing file after truncate operation results to hang
Product:	[Community] GlusterFS	Reporter:	Ravishankar N <ravishankar>
Component:	replicate	Assignee:	Ravishankar N <ravishankar>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.1	CC:	bugs, moagrawa, pkarampu, ravishankar
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-4.1.2	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1602236	Environment:
Last Closed:	2018-07-30 19:02:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1599998, 1602236, 1602241
Bug Blocks:

Description Ravishankar N 2018-07-19 04:22:33 UTC

+++ This bug was initially created as a clone of Bug #1602236 +++

+++ This bug was initially created as a clone of Bug #1599998 +++

Description of problem:
=======================
When reserve limits are reached, append on an existing file after truncate operation results to hang.

Version-Release number of selected component (if applicable):
3.12.2-13.el7rhgs.x86_64

How reproducible:
always

Steps to Reproduce:
===================
1) Create a Distributed-Replicate volume and start it.
2) Set the storage.reserve limits to 50 using below command,
 gluster v set distrep storage.reserve 50
3) FUSE mount the volume on a client.
4) Write data till the limits reaches reserve limits(you will see 100% mount point fill in df -h client output). I have used fallocate to quickly fill the disk.
5) Pick an existing file and truncate that file.
truncate -s 0 fallocate_99
It will throw ENOSPC error.
6) Now on the same file try to append some data.
cat /etc/redhat-release > fallocate_99

Actual results:
===============
append will hang. cat command hung here.

Expected results:
=================
No hungs.

--- Additional comment from Worker Ant on 2018-07-17 23:57:10 EDT ---

REVIEW: https://review.gluster.org/20527 (afr: switch lk_owner only when pre-op succeeds) posted (#1) for review on master by Ravishankar N

--- Additional comment from Worker Ant on 2018-07-18 12:50:31 EDT ---

COMMIT: https://review.gluster.org/20527 committed in master by "Ravishankar N" <ravishankar> with a commit message- afr: switch lk_owner only when pre-op succeeds

Problem:
In a disk full scenario, we take a failure path in afr_transaction_perform_fop()
and go to unlock phase. But we change the lk-owner before that, causing unlock
to fail. When mount issues another fop that takes locks on that file, it hangs.

Fix:
Change lk-owner only when we are about to perform the fop phase.
Also fix the same issue for arbiters when afr_txn_arbitrate_fop() fails the fop.

Also removed the DISK_SPACE_CHECK_AND_GOTO in posix_xattrop. Otherwise truncate
to zero will fail pre-op phase with ENOSPC when the user is actually trying to
freee up space.

Change-Id: Ic4c8a596b4cdf4a7fc189bf00b561113cf114353
fixes: bz#1602236
Signed-off-by: Ravishankar N <ravishankar>

Comment 1 Ravishankar N 2018-07-19 04:25:03 UTC

https://review.gluster.org/#/c/20532/

Comment 2 Worker Ant 2018-07-19 04:26:56 UTC

REVIEW: https://review.gluster.org/20532 (afr: switch lk_owner only when pre-op succeeds) posted (#1) for review on release-4.1 by Ravishankar N

Comment 3 Worker Ant 2018-07-23 23:42:40 UTC

COMMIT: https://review.gluster.org/20532 committed in release-4.1 by "Shyamsundar Ranganathan" <srangana> with a commit message- afr: switch lk_owner only when pre-op succeeds

Problem:
In a disk full scenario, we take a failure path in afr_transaction_perform_fop()
and go to unlock phase. But we change the lk-owner before that, causing unlock
to fail. When mount issues another fop that takes locks on that file, it hangs.

Fix:
Change lk-owner only when we are about to perform the fop phase.
Also fix the same issue for arbiters when afr_txn_arbitrate_fop() fails the fop.

Also removed the DISK_SPACE_CHECK_AND_GOTO in posix_xattrop. Otherwise truncate
to zero will fail pre-op phase with ENOSPC when the user is actually trying to
freee up space.

Change-Id: Ic4c8a596b4cdf4a7fc189bf00b561113cf114353
fixes: bz#1603056
Signed-off-by: Ravishankar N <ravishankar>
(cherry picked from commit ec0d7d77de3e4bd485a4fa2e53c9137e25c71ce7)

Comment 4 Shyamsundar 2018-07-30 19:02:48 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.2, please open a new bug report.

glusterfs-4.1.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2018-July/000106.html
[2] https://www.gluster.org/pipermail/gluster-users/