Bug 1273354 - rm command fails with "Transport end point not connected" during add brick
rm command fails with "Transport end point not connected" during add brick
Status: CLOSED CURRENTRELEASE
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute (Show other bugs)
3.1
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Nithya Balachandran
Anoop
dht-add-brick, dht-fixed
: ZStream
: 994405 (view as bug list)
Depends On: 994405
Blocks: 1225330 1229270 1229271 1233151 1235202 1265890
  Show dependency treegraph
 
Reported: 2015-10-20 05:23 EDT by Sakshi
Modified: 2016-08-22 09:03 EDT (History)
8 users (show)

See Also:
Fixed In Version: 3.7.9-10
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 994405
Environment:
Last Closed: 2016-08-22 09:03:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sakshi 2015-10-20 05:23:44 EDT
+++ This bug was initially created as a clone of Bug #994405 +++

Description of problem:

while removing a directory from the mount point if we issue add-brick command then  rm fails with "Transport end point not connected"

Version-Release number of selected component (if applicable):

3.4.0.17rhs-1.el6rhs.x86_64
How reproducible:
always

Steps to Reproduce:
1. created a distributed volume
2. mount the volume and untarred the kernel
3. rm -rf linux-2.6.32.61

Actual results:

after sometime error pops on the mount point

[root@gqac024 mnt]# rm -rf linux-2.6.32.61
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-integrator/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-iop13xx/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-iop32x/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-iop33x/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ixp2000/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ixp23xx/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ixp4xx/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-kirkwood/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ks8695/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-l7200/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-lh7a40x/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-loki/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mmp/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-msm/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mv78xx0/include/mach': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx1': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx2': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx25': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx3': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mxc91231': Transport endpoint is not connected
rm: cannot remove `linux-2.6.32.61/arch/arm/mach-netx/include/mach': Transport endpoint is not connected


Expected results:


Additional info:
================

RHS nodes
=========
gqac022.sbu.lab.eng.bos.redhat.com
gqac023.sbu.lab.eng.bos.redhat.com

Mounted on 
============
gqac024.sbu.lab.eng.bos.redhat.com

mount point 
===========
/mnt

add-brick issued from gqac022.sbu.lab.eng.bos.redhat.com

[root@gqac022 rpm]# gluster v info anon
 
Volume Name: anon
Type: Distribute
Volume ID: 61e3c5b2-cb03-4ea8-9a69-a8762191d296
Status: Started
Number of Bricks: 15
Transport-type: tcp
Bricks:
Brick1: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon1
Brick2: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon2
Brick3: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon3
Brick4: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon4
Brick5: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon5
Brick6: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon6
Brick7: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon7
Brick8: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon8
Brick9: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon9
Brick10: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon10
Brick11: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon11
Brick12: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon12
Brick13: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon13
Brick14: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon14
Brick15: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon15


Rebalance was performed before adding the new bricks



mnt logs
=========
2013-08-07 08:06:00.201235] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-14: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221)
[2013-08-07 08:06:00.201274] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-13: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221)
[2013-08-07 08:06:00.215054] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 4-anon-client-13: remote operation failed: No such fil
e or directory
[2013-08-07 08:06:00.215480] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 4-anon-client-14: remote operation failed: No such fil
e or directory
[2013-08-07 08:06:00.228461] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-13: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221)
[2013-08-07 08:06:00.228536] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-14: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221)
[2013-08-07 08:06:00.229528] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-14: remote operation failed: No such file or
 directory
[2013-08-07 08:06:00.229607] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-13: remote operation failed: No such file or
 directory
[2013-08-07 08:06:00.230174] W [fuse-bridge.c:1688:fuse_unlink_cbk] 0-glusterfs-fuse: 1154112: RMDIR() <gfid:6ef0c711-d65c-4cec-90ba-
1ba87e1163e0>/virt/kvm => -1 (No such file or directory)
[2013-08-07 08:06:00.230932] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-14: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt (42ca8074-6470-48c0-9731-eb7e8a5d63ea)
[2013-08-07 08:06:00.231116] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-13: remote operation failed: No such file
 or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt (42ca8074-6470-48c0-9731-eb7e8a5d63ea)
[2013-08-07 08:06:00.231941] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-14: remote operation failed: No such file or
 directory
[2013-08-07 08:06:00.232113] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-13: remote operation failed: No such file or
 directory
[2013-08-07 08:06:00.232618] W [fuse-bridge.c:1688:fuse_unlink_cbk] 0-glusterfs-fuse: 1154115: RMDIR() <gfid:6ef0c711-d65c-4cec-90ba-
1ba87e1163e0>/virt => -1 (No such file or directory)
Comment 2 Susant Kumar Palai 2015-11-27 05:39:45 EST
*** Bug 994405 has been marked as a duplicate of this bug. ***
Comment 7 Prasad Desala 2016-08-22 08:41:51 EDT
This issue is not seen on glusterfs version: 3.7.9-10.el7rhgs.x86_64.

Here are the steps that were performed,

1. Created a distributed replica volume.
2. FUSE mounted the volume and untarred the kernel.
3. From mount issued "rm -rf linux-4.6.4.1", while "rm -rf" command execution is in-progress issued multiple add-brick commands one after the other till "rm -rf" executes. 
add-brick commands were executed successfully and "rm -rf" command did not throw any error.

Also, repeated the same steps with NFS mount and no error was seen.

Hence, marking the BZ as Verified.
Comment 8 Nithya Balachandran 2016-08-22 09:03:36 EDT
Thanks Prasad.

As per comment#7 I am closing this BZ.

Note You need to log in before you can comment on or make changes to this bug.