Bug 1039674

Summary:	quota: ENOTCONN parodically seen in logs when setting hard/soft timeout during I/O.
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Ben Turner <bturner>
Component:	quota	Assignee:	Vijaikumar Mallikarjuna <vmallika>
Status:	CLOSED ERRATA	QA Contact:	Ben Turner <bturner>
Severity:	high	Docs Contact:
Priority:	medium
Version:	2.1	CC:	ashah, bmohanra, nsathyan, rgowdapp, rhs-bugs, smohan, storage-qa-internal, vagarwal, vbellur, vmallika
Target Milestone:	---
Target Release:	RHGS 3.1.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.1-1	Doc Type:	Bug Fix
Doc Text:	Previously, when quotad was restarted as part of add/remove brick, resulted in 'Transport endpoint Not Connected' error in the I/O path. With this fix, this issue is resolved.	Story Points:	---
Clone Of:
Clones:	1211220 (view as bug list)		Environment:
Last Closed:	2015-07-29 04:30:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1186580, 1202842

Description Ben Turner 2013-12-09 18:20:38 UTC

Description of problem:

When running quota automation I occasionally(1 in 10 runs?) see the following testcase fail:

1. create a 6x2 volume, start it.
2. gluster volume quota <vol-name> enable
3. gluster volume quota <vol-name> limit-usage / 5GB
4. gluster volume quota <vol-name> list
5. mount -t nfs/glusterfs/(or mount using SMB) <server-ip>:<vol-name> <mount-point>
6. start creating data inside the mount-point, till limit is reached. files of size 2MB meanwhile:
7. gluster volume quota <vol-name> soft-timeout 30s
8. gluster volume quota <vol-name> hard-timeout 60s after data creation is completed.
10. gluster volume quota <vol-name> list

Client side I see:

dd: opening `/quota-mount/tcms_285026/test.file': Transport endpoint is not connected

And in the brick logs I see:

/var/log/glusterfs/bricks/bricks-quota-test-setup_brick2.log:[2013-12-06 17:59:02.743336] W [quota-enforcer-client.c:187:quota_enforcer_lookup_cbk] 0-quota-test-setup-quota: remote operation failed: Transport endpoint is not connected. Path: /tcms_285026 (d892ce24-7e59-4eeb-b86f-7c7d34c71317)
/var/log/glusterfs/bricks/bricks-quota-test-setup_brick2.log:[2013-12-06 17:59:02.743377] I [server-rpc-fops.c:1618:server_create_cbk] 0-quota-test-setup-server: 26: CREATE /tcms_285026/test.file (d892ce24-7e59-4eeb-b86f-7c7d34c71317/test.file) ==> (Transport endpoint is not connected)

Version-Release number of selected component (if applicable):

glusterfs-server-3.4.0.44.1u2rhs-1.el6rhs.x86_64

How reproducible:

So far this looks to be about 1 in 10 runs.

Steps to Reproduce:
1. create a 6x2 volume, start it.
2. gluster volume quota <vol-name> enable
3. gluster volume quota <vol-name> limit-usage / 5GB
4. gluster volume quota <vol-name> list
5. mount -t nfs/glusterfs/(or mount using SMB) <server-ip>:<vol-name> <mount-point>
6. start creating data inside the mount-point, till limit is reached. files of size 2MB meanwhile:
7. gluster volume quota <vol-name> soft-timeout 30s
8. gluster volume quota <vol-name> hard-timeout 60s after data creation is completed.
10. gluster volume quota <vol-name> list

Actual results:

I/O errors are occasionally hit when the hard/soft timeout is modified with data in flight.

Expected results:

I/Os complete successfully when timeouts are modified.

Additional info:

I'll try to provide a more concrete reproducer.

Comment 1 Vijaikumar Mallikarjuna 2015-03-03 08:59:22 UTC

Hi Ben,

I am not able to re-create this issue with 3.6 release.

Comment 2 Vijaikumar Mallikarjuna 2015-04-13 10:41:06 UTC

Whenever a new volume is created, quotad gets restarted. This can cause ENOTCONN in the others volumes IO path

Comment 3 Vijaikumar Mallikarjuna 2015-04-14 08:55:59 UTC

submitted upstream patch: http://review.gluster.org/10230

Comment 4 Ben Turner 2015-07-07 17:13:45 UTC

Verified on glusterfs-3.7.1-7.

Comment 5 Bhavana 2015-07-25 07:45:09 UTC

Hi Vijai,

The doc text is updated. Please review the same and share your technical review comments. If it looks ok, then sign-off on the same.

Regards,
Bhavana

Comment 6 Vijaikumar Mallikarjuna 2015-07-27 04:32:28 UTC

Doc-text looks good to me

Comment 8 errata-xmlrpc 2015-07-29 04:30:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1495.html