Bug 1266863 - Data Tiering:Getting "Transport endpoint is not connected" error when writing new files after stopping quota deamon after quotas are exceeded
Data Tiering:Getting "Transport endpoint is not connected" error when writing...
Status: CLOSED NOTABUG
Product: GlusterFS
Classification: Community
Component: quota (Show other bugs)
3.7.4
Unspecified Unspecified
low Severity medium
: ---
: ---
Assigned To: Vijaikumar Mallikarjuna
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-28 05:26 EDT by nchilaka
Modified: 2016-05-11 18:49 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-21 04:18:49 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description nchilaka 2015-09-28 05:26:58 EDT
Description of problem:
========================
On a tiered volume, After setting quotas I did some IOs . After quotas exceeded, I killed the quota deamon on local node. Now when I tried do new file creates, I am getting transport end point error on fuse mount.
I tried and  deleted some files and retried , new files got created till probably till disk quota got reached and again started to throw transport end point error


NOTE: This may not be a tiering problem alone. It has to be more to do with quotas.


[root@zod ~]# rpm -qa|grep gluster
glusterfs-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-api-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-client-xlators-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-server-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-cli-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-libs-3.7.4-0.43.gitf139283.el7.centos.x86_64
[root@zod ~]# gluster --version
glusterfs 3.7.4 built on Sep 19 2015 01:30:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod ~]# 


Steps to Reproduce:
==================
1.create a tier volume 
2.enable quota and set hardlimit of root volume / to say 10gb
3. Now enable ctr and set the demote freq to say 1500sec
3.Now mount from glustefs-fuse and now fill the hardlimit to full
4. Now kill the quota deamon on local node
5. Now do new file creates and you can see transport end point error.





Logs:
====
fuse mount log sample:
[root@localhost togo]# tail /var/log/glusterfs/mnt-togo.log 
[2015-09-28 09:10:12.329534] W [fuse-bridge.c:1978:fuse_create_cbk] 0-glusterfs-fuse: 168844: /file.205 => -1 (Transport endpoint is not connected)
[2015-09-28 09:10:12.371421] W [MSGID: 114031] [client-rpc-fops.c:2399:client3_3_create_cbk] 0-togo-client-7: remote operation failed. Path: /file.206 [Disk quota exceeded]




deleted some files and retried(file 131-151 got created and then started to error out)
[root@localhost togo]# for i in {131..230};do cp file.1 file.$i;done
	cp: error writing ‘file.152’: Disk quota exceeded
cp: failed to extend ‘file.152’: Disk quota exceeded
cp: failed to close ‘file.152’: Disk quota exceeded
cp: cannot create regular file ‘file.153’: Transport endpoint is not connected
cp: cannot create regular file ‘file.154’: Transport endpoint is not connected
cp: cannot create regular file ‘file.155’: Transport endpoint is not connected
cp: cannot create regular file ‘file.156’: Transport endpoint is not connected
Comment 1 nchilaka 2015-09-28 05:34:57 EDT
sosreports@rhsqe-repo.lab.eng.blr.redhat.com:/home/repo/sosreports/bug.1266863
Comment 2 Vijaikumar Mallikarjuna 2015-12-21 04:18:49 EST
Hi Nag Pavan,

When quotad is down, it is expected to get 'Transport endpoint is not connected' in the IO path. so this is not a bug

Note You need to log in before you can comment on or make changes to this bug.