Bug 1266863

Summary: Data Tiering:Getting "Transport endpoint is not connected" error when writing new files after stopping quota deamon after quotas are exceeded
Product: [Community] GlusterFS Reporter: nchilaka <nchilaka>
Component: quotaAssignee: Vijaikumar Mallikarjuna <vmallika>
Status: CLOSED NOTABUG QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 3.7.4CC: bugs, dlambrig, gluster-bugs, josferna, mselvaga, nbalacha, nsathyan, rgowdapp, smohan
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-21 04:18:49 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description nchilaka 2015-09-28 05:26:58 EDT
Description of problem:
On a tiered volume, After setting quotas I did some IOs . After quotas exceeded, I killed the quota deamon on local node. Now when I tried do new file creates, I am getting transport end point error on fuse mount.
I tried and  deleted some files and retried , new files got created till probably till disk quota got reached and again started to throw transport end point error

NOTE: This may not be a tiering problem alone. It has to be more to do with quotas.

[root@zod ~]# rpm -qa|grep gluster
[root@zod ~]# gluster --version
glusterfs 3.7.4 built on Sep 19 2015 01:30:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod ~]# 

Steps to Reproduce:
1.create a tier volume 
2.enable quota and set hardlimit of root volume / to say 10gb
3. Now enable ctr and set the demote freq to say 1500sec
3.Now mount from glustefs-fuse and now fill the hardlimit to full
4. Now kill the quota deamon on local node
5. Now do new file creates and you can see transport end point error.

fuse mount log sample:
[root@localhost togo]# tail /var/log/glusterfs/mnt-togo.log 
[2015-09-28 09:10:12.329534] W [fuse-bridge.c:1978:fuse_create_cbk] 0-glusterfs-fuse: 168844: /file.205 => -1 (Transport endpoint is not connected)
[2015-09-28 09:10:12.371421] W [MSGID: 114031] [client-rpc-fops.c:2399:client3_3_create_cbk] 0-togo-client-7: remote operation failed. Path: /file.206 [Disk quota exceeded]

deleted some files and retried(file 131-151 got created and then started to error out)
[root@localhost togo]# for i in {131..230};do cp file.1 file.$i;done
	cp: error writing ‘file.152’: Disk quota exceeded
cp: failed to extend ‘file.152’: Disk quota exceeded
cp: failed to close ‘file.152’: Disk quota exceeded
cp: cannot create regular file ‘file.153’: Transport endpoint is not connected
cp: cannot create regular file ‘file.154’: Transport endpoint is not connected
cp: cannot create regular file ‘file.155’: Transport endpoint is not connected
cp: cannot create regular file ‘file.156’: Transport endpoint is not connected
Comment 1 nchilaka 2015-09-28 05:34:57 EDT
Comment 2 Vijaikumar Mallikarjuna 2015-12-21 04:18:49 EST
Hi Nag Pavan,

When quotad is down, it is expected to get 'Transport endpoint is not connected' in the IO path. so this is not a bug