+++ This bug was initially created as a clone of Bug #994405 +++ Description of problem: while removing a directory from the mount point if we issue add-brick command then rm fails with "Transport end point not connected" Version-Release number of selected component (if applicable): 3.4.0.17rhs-1.el6rhs.x86_64 How reproducible: always Steps to Reproduce: 1. created a distributed volume 2. mount the volume and untarred the kernel 3. rm -rf linux-2.6.32.61 Actual results: after sometime error pops on the mount point [root@gqac024 mnt]# rm -rf linux-2.6.32.61 rm: cannot remove `linux-2.6.32.61/arch/arm/mach-integrator/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-iop13xx/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-iop32x/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-iop33x/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ixp2000/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ixp23xx/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ixp4xx/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-kirkwood/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-ks8695/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-l7200/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-lh7a40x/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-loki/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mmp/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-msm/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mv78xx0/include/mach': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx1': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx2': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx25': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mx3': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-mxc91231': Transport endpoint is not connected rm: cannot remove `linux-2.6.32.61/arch/arm/mach-netx/include/mach': Transport endpoint is not connected Expected results: Additional info: ================ RHS nodes ========= gqac022.sbu.lab.eng.bos.redhat.com gqac023.sbu.lab.eng.bos.redhat.com Mounted on ============ gqac024.sbu.lab.eng.bos.redhat.com mount point =========== /mnt add-brick issued from gqac022.sbu.lab.eng.bos.redhat.com [root@gqac022 rpm]# gluster v info anon Volume Name: anon Type: Distribute Volume ID: 61e3c5b2-cb03-4ea8-9a69-a8762191d296 Status: Started Number of Bricks: 15 Transport-type: tcp Bricks: Brick1: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon1 Brick2: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon2 Brick3: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon3 Brick4: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon4 Brick5: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon5 Brick6: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon6 Brick7: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon7 Brick8: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon8 Brick9: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon9 Brick10: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon10 Brick11: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon11 Brick12: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon12 Brick13: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon13 Brick14: gqac023.sbu.lab.eng.bos.redhat.com:/home/anon14 Brick15: gqac022.sbu.lab.eng.bos.redhat.com:/home/anon15 Rebalance was performed before adding the new bricks mnt logs ========= 2013-08-07 08:06:00.201235] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-14: remote operation failed: No such file or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221) [2013-08-07 08:06:00.201274] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-13: remote operation failed: No such file or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221) [2013-08-07 08:06:00.215054] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 4-anon-client-13: remote operation failed: No such fil e or directory [2013-08-07 08:06:00.215480] W [client-rpc-fops.c:2316:client3_3_readdirp_cbk] 4-anon-client-14: remote operation failed: No such fil e or directory [2013-08-07 08:06:00.228461] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-13: remote operation failed: No such file or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221) [2013-08-07 08:06:00.228536] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-14: remote operation failed: No such file or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt/kvm (28c128a7-87c1-493e-9aab-713bcbc73221) [2013-08-07 08:06:00.229528] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-14: remote operation failed: No such file or directory [2013-08-07 08:06:00.229607] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-13: remote operation failed: No such file or directory [2013-08-07 08:06:00.230174] W [fuse-bridge.c:1688:fuse_unlink_cbk] 0-glusterfs-fuse: 1154112: RMDIR() <gfid:6ef0c711-d65c-4cec-90ba- 1ba87e1163e0>/virt/kvm => -1 (No such file or directory) [2013-08-07 08:06:00.230932] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-14: remote operation failed: No such file or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt (42ca8074-6470-48c0-9731-eb7e8a5d63ea) [2013-08-07 08:06:00.231116] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 4-anon-client-13: remote operation failed: No such file or directory. Path: <gfid:6ef0c711-d65c-4cec-90ba-1ba87e1163e0>/virt (42ca8074-6470-48c0-9731-eb7e8a5d63ea) [2013-08-07 08:06:00.231941] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-14: remote operation failed: No such file or directory [2013-08-07 08:06:00.232113] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 4-anon-client-13: remote operation failed: No such file or directory [2013-08-07 08:06:00.232618] W [fuse-bridge.c:1688:fuse_unlink_cbk] 0-glusterfs-fuse: 1154115: RMDIR() <gfid:6ef0c711-d65c-4cec-90ba- 1ba87e1163e0>/virt => -1 (No such file or directory)
*** Bug 994405 has been marked as a duplicate of this bug. ***
This issue is not seen on glusterfs version: 3.7.9-10.el7rhgs.x86_64. Here are the steps that were performed, 1. Created a distributed replica volume. 2. FUSE mounted the volume and untarred the kernel. 3. From mount issued "rm -rf linux-4.6.4.1", while "rm -rf" command execution is in-progress issued multiple add-brick commands one after the other till "rm -rf" executes. add-brick commands were executed successfully and "rm -rf" command did not throw any error. Also, repeated the same steps with NFS mount and no error was seen. Hence, marking the BZ as Verified.
Thanks Prasad. As per comment#7 I am closing this BZ.