Description of problem: ======================== On a tiered volume, After setting quotas I did some IOs . After quotas exceeded, I killed the quota deamon on local node. Now when I tried do new file creates, I am getting transport end point error on fuse mount. I tried and deleted some files and retried , new files got created till probably till disk quota got reached and again started to throw transport end point error NOTE: This may not be a tiering problem alone. It has to be more to do with quotas. [root@zod ~]# rpm -qa|grep gluster glusterfs-3.7.4-0.43.gitf139283.el7.centos.x86_64 glusterfs-fuse-3.7.4-0.43.gitf139283.el7.centos.x86_64 glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64 glusterfs-api-3.7.4-0.43.gitf139283.el7.centos.x86_64 glusterfs-client-xlators-3.7.4-0.43.gitf139283.el7.centos.x86_64 glusterfs-server-3.7.4-0.43.gitf139283.el7.centos.x86_64 glusterfs-cli-3.7.4-0.43.gitf139283.el7.centos.x86_64 glusterfs-libs-3.7.4-0.43.gitf139283.el7.centos.x86_64 [root@zod ~]# gluster --version glusterfs 3.7.4 built on Sep 19 2015 01:30:43 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@zod ~]# Steps to Reproduce: ================== 1.create a tier volume 2.enable quota and set hardlimit of root volume / to say 10gb 3. Now enable ctr and set the demote freq to say 1500sec 3.Now mount from glustefs-fuse and now fill the hardlimit to full 4. Now kill the quota deamon on local node 5. Now do new file creates and you can see transport end point error. Logs: ==== fuse mount log sample: [root@localhost togo]# tail /var/log/glusterfs/mnt-togo.log [2015-09-28 09:10:12.329534] W [fuse-bridge.c:1978:fuse_create_cbk] 0-glusterfs-fuse: 168844: /file.205 => -1 (Transport endpoint is not connected) [2015-09-28 09:10:12.371421] W [MSGID: 114031] [client-rpc-fops.c:2399:client3_3_create_cbk] 0-togo-client-7: remote operation failed. Path: /file.206 [Disk quota exceeded] deleted some files and retried(file 131-151 got created and then started to error out) [root@localhost togo]# for i in {131..230};do cp file.1 file.$i;done cp: error writing ‘file.152’: Disk quota exceeded cp: failed to extend ‘file.152’: Disk quota exceeded cp: failed to close ‘file.152’: Disk quota exceeded cp: cannot create regular file ‘file.153’: Transport endpoint is not connected cp: cannot create regular file ‘file.154’: Transport endpoint is not connected cp: cannot create regular file ‘file.155’: Transport endpoint is not connected cp: cannot create regular file ‘file.156’: Transport endpoint is not connected
sosreports.eng.blr.redhat.com:/home/repo/sosreports/bug.1266863
Hi Nag Pavan, When quotad is down, it is expected to get 'Transport endpoint is not connected' in the IO path. so this is not a bug