Bug 1273354

Summary:	rm command fails with "Transport end point not connected" during add brick
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Sakshi <sabansal>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Anoop <annair>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	nbalacha, rgowdapp, rhs-bugs, shmohan, smohan, storage-qa-internal, tdesala, vbellur
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	dht-add-brick, dht-fixed
Fixed In Version:	3.7.9-10	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	994405	Environment:
Last Closed:	2016-08-22 13:03:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	994405
Bug Blocks:	1225330, 1229270, 1229271, 1233151, 1235202, 1265890

Description Sakshi 2015-10-20 09:23:44 UTC

+++ This bug was initially created as a clone of Bug #994405 +++

Description of problem:

while removing a directory from the mount point if we issue add-brick command then  rm fails with "Transport end point not connected"

Version-Release number of selected component (if applicable):

3.4.0.17rhs-1.el6rhs.x86_64
How reproducible:
always

Steps to Reproduce:
1. created a distributed volume
2. mount the volume and untarred the kernel
3. rm -rf linux-2.6.32.61

Actual results:

after sometime error pops on the mount point

[root@gqac024 mnt]# rm -rf linux-2.6.32.61
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-integrator/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-iop13xx/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-iop32x/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-iop33x/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ixp2000/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ixp23xx/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ixp4xx/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-kirkwood/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ks8695/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-l7200/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-lh7a40x/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-loki/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mmp/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-msm/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mv78xx0/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx1': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx2': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx25': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx3': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mxc91231': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-netx/include/mach': Transport endpoint is not connected


Expected results:


Additional info:
================

RHS nodes
=========
gqac022.sbu.lab.eng.bos.redhat.com
gqac023.sbu.lab.eng.bos.redhat.com

Mounted on 
============
gqac024.sbu.lab.eng.bos.redhat.com

mount point 
===========
/mnt

add-brick issued from gqac022.sbu.lab.eng.bos.redhat.com

[root@gqac022 rpm]# gluster v info anon
 
Volume Name: anon
Type: Distribute
Volume ID: 61e3c5b2-cb03-4ea8-9a69-a8762191d296
Status: Started
Number of Bricks: 15
Transport-type: tcp
Bricks:
Brick1: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon1
Brick2: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon2
Brick3: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon3
Brick4: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon4
Brick5: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon5
Brick6: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon6
Brick7: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon7
Brick8: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon8
Brick9: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon9
Brick10: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon10
Brick11: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon11
Brick12: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon12
Brick13: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon13
Brick14: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon14
Brick15: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon15


Rebalance was performed before adding the new bricks



mnt logs
=========
2013-08-07 08:06:00.201235] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-14: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221)
[2013-08-07 08:06:00.201274] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-13: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221)
[2013-08-07 08:06:00.215054] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 4-anon-client-13: remote operation failed: No such fil
e or directory
[2013-08-07 08:06:00.215480] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 4-anon-client-14: remote operation failed: No such fil
e or directory
[2013-08-07 08:06:00.228461] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-13: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221)
[2013-08-07 08:06:00.228536] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-14: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221)
[2013-08-07 08:06:00.229528] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-14: remote operation failed: No such file or
 directory
[2013-08-07 08:06:00.229607] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-13: remote operation failed: No such file or
 directory
[2013-08-07 08:06:00.230174] W [fuse-bridge.c:1688:fuse_unlink_cbk] 0-glusterfs-fuse: 1154112: RMDIR() <gfid:6ef0c711-d65c-4cec-90ba-
1ba87e1163e0>/virt/kvm => -1 (No such file or directory)
[2013-08-07 08:06:00.230932] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-14: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt (42ca8074-6470-48c0-9731-eb7e8a5d63ea)
[2013-08-07 08:06:00.231116] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-13: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt (42ca8074-6470-48c0-9731-eb7e8a5d63ea)
[2013-08-07 08:06:00.231941] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-14: remote operation failed: No such file or
 directory
[2013-08-07 08:06:00.232113] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-13: remote operation failed: No such file or
 directory
[2013-08-07 08:06:00.232618] W [fuse-bridge.c:1688:fuse_unlink_cbk] 0-glusterfs-fuse: 1154115: RMDIR() <gfid:6ef0c711-d65c-4cec-90ba-
1ba87e1163e0>/virt => -1 (No such file or directory)

Comment 2 Susant Kumar Palai 2015-11-27 10:39:45 UTC

*** Bug 994405 has been marked as a duplicate of this bug. ***

Comment 7 Prasad Desala 2016-08-22 12:41:51 UTC

This issue is not seen on glusterfs version: 3.7.9-10.el7rhgs.x86_64.

Here are the steps that were performed,

1. Created a distributed replica volume.
2. FUSE mounted the volume and untarred the kernel.
3. From mount issued "rm -rf linux-4.6.4.1", while "rm -rf" command execution is in-progress issued multiple add-brick commands one after the other till "rm -rf" executes. 
add-brick commands were executed successfully and "rm -rf" command did not throw any error.

Also, repeated the same steps with NFS mount and no error was seen.

Hence, marking the BZ as Verified.

Comment 8 Nithya Balachandran 2016-08-22 13:03:36 UTC

Thanks Prasad.

As per comment#7 I am closing this BZ.