Bug 1266863 - Data Tiering:Getting "Transport endpoint is not connected" error when writing new files after stopping quota deamon after quotas are exceeded
Summary: Data Tiering:Getting "Transport endpoint is not connected" error when writing...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: GlusterFS
Classification: Community
Component: quota
Version: 3.7.4
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
Assignee: Vijaikumar Mallikarjuna
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-28 09:26 UTC by Nag Pavan Chilakam
Modified: 2016-05-11 22:49 UTC (History)
9 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-12-21 09:18:49 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Nag Pavan Chilakam 2015-09-28 09:26:58 UTC
Description of problem:
========================
On a tiered volume, After setting quotas I did some IOs . After quotas exceeded, I killed the quota deamon on local node. Now when I tried do new file creates, I am getting transport end point error on fuse mount.
I tried and  deleted some files and retried , new files got created till probably till disk quota got reached and again started to throw transport end point error


NOTE: This may not be a tiering problem alone. It has to be more to do with quotas.


[root@zod ~]# rpm -qa|grep gluster
glusterfs-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-api-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-client-xlators-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-server-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-cli-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-libs-3.7.4-0.43.gitf139283.el7.centos.x86_64
[root@zod ~]# gluster --version
glusterfs 3.7.4 built on Sep 19 2015 01:30:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod ~]# 


Steps to Reproduce:
==================
1.create a tier volume 
2.enable quota and set hardlimit of root volume / to say 10gb
3. Now enable ctr and set the demote freq to say 1500sec
3.Now mount from glustefs-fuse and now fill the hardlimit to full
4. Now kill the quota deamon on local node
5. Now do new file creates and you can see transport end point error.





Logs:
====
fuse mount log sample:
[root@localhost togo]# tail /var/log/glusterfs/mnt-togo.log 
[2015-09-28 09:10:12.329534] W [fuse-bridge.c:1978:fuse_create_cbk] 0-glusterfs-fuse: 168844: /file.205 => -1 (Transport endpoint is not connected)
[2015-09-28 09:10:12.371421] W [MSGID: 114031] [client-rpc-fops.c:2399:client3_3_create_cbk] 0-togo-client-7: remote operation failed. Path: /file.206 [Disk quota exceeded]




deleted some files and retried(file 131-151 got created and then started to error out)
[root@localhost togo]# for i in {131..230};do cp file.1 file.$i;done
	cp: error writing ‘file.152’: Disk quota exceeded
cp: failed to extend ‘file.152’: Disk quota exceeded
cp: failed to close ‘file.152’: Disk quota exceeded
cp: cannot create regular file ‘file.153’: Transport endpoint is not connected
cp: cannot create regular file ‘file.154’: Transport endpoint is not connected
cp: cannot create regular file ‘file.155’: Transport endpoint is not connected
cp: cannot create regular file ‘file.156’: Transport endpoint is not connected

Comment 1 Nag Pavan Chilakam 2015-09-28 09:34:57 UTC
sosreports.eng.blr.redhat.com:/home/repo/sosreports/bug.1266863

Comment 2 Vijaikumar Mallikarjuna 2015-12-21 09:18:49 UTC
Hi Nag Pavan,

When quotad is down, it is expected to get 'Transport endpoint is not connected' in the IO path. so this is not a bug


Note You need to log in before you can comment on or make changes to this bug.