Bug 1455034

Summary: Brick Multiplexing: Getting "Directory not empty" error while doing rm -rf
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nag Pavan Chilakam <nchilaka>
Component: coreAssignee: Ravishankar N <ravishankar>
Status: CLOSED DUPLICATE QA Contact: Rahul Hinduja <rhinduja>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rhgs-3.3CC: amukherj, nbalacha, nchilaka, ravishankar, rhs-bugs, storage-qa-internal
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: brick-multiplexing
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-24 12:53:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nag Pavan Chilakam 2017-05-24 06:20:34 UTC 
Description of problem:
=========================
In my brick mux setup, I have mounted about 8 out of the 45 volumes all 1x3
I had populated many files.
I then issued rm -rf on all the clients

I saw the rm -rf taking a lot of time and then some clients failed to remove files completely due to "Directory not empty"



[root@dhcp37-197 smallfile]# rm -rf *

rm: cannot remove ‘file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_00’: Directory not empty
rm: cannot remove ‘file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_06/d_006’: Directory not empty
rm: cannot remove ‘file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_10/d_006’: Directory not empty

On the brick logs: I see below messages only on one of the bricks:
[2017-05-23 14:02:50.534466] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-45.lab.eng.blr.redhat.com-31274-2017/05/23-05:47:34:85630-cross3-27-client-2-0-0
[2017-05-23 14:02:50.536928] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-45.lab.eng.blr.redhat.com-31274-2017/05/23-05:47:34:85630-cross3-27-client-2-0-0
[2017-05-23 18:56:01.795869] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 133483267: RMDIR /smallfile/file_srcdir/dhcp37-151.lab.eng.blr.redhat.com/thrd_14/d_006/d_005/d_002 (f7204b9f-1531-4703-a213-080097293491/d_002) ==> (Directory not empty) [Directory not empty]
[2017-05-23 18:56:43.474309] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 133484528: RMDIR /smallfile/file_srcdir/dhcp37-151.lab.eng.blr.redhat.com/thrd_14/d_006/d_005 (9b65ccef-79a7-42cc-b3ae-7c33e72c787e/d_005) ==> (Directory not empty) [Directory not empty]
[2017-05-23 19:10:34.014515] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 133511066: RMDIR /smallfile/file_srcdir/dhcp37-151.lab.eng.blr.redhat.com/thrd_14/d_006 (979a6de8-b757-40a0-bbf8-f3eddea295a3/d_006) ==> (Directory not empty) [Directory not empty]
[2017-05-23 19:13:23.947394] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 133517220: RMDIR /smallfile/file_srcdir/dhcp37-151.lab.eng.blr.redhat.com/thrd_14 (d3c897f4-0511-4960-86c4-d1aa33535395/thrd_14) ==> (Directory not empty) [Directory not empty]
[2017-05-23 19:42:48.700328] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 133140981: RMDIR /smallfile/file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_00/d_006/d_005/d_003 (eb166d4c-d098-4fe8-aedc-f7d4bd4e9cc6/d_003) ==> (Directory not empty) [Directory not empty]
[2017-05-23 19:43:27.970248] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 133142259: RMDIR /smallfile/file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_00/d_006/d_005 (6e0adf17-0380-46bb-8734-8548b5d0d2e8/d_005) ==> (Directory not empty) [Directory not empty]
[2017-05-23 19:46:55.365321] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 133149593: RMDIR /smallfile/file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_00/d_006 (1d5ebce6-4e3c-4793-8bda-554f3c2e424b/d_006) ==> (Directory not empty) [Directory not empty]
[2017-05-24 01:12:52.738577] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 134012550: RMDIR /smallfile/file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_06/d_006/d_005/d_003 (921ee156-ab37-405a-bb85-2dae53bd3479/d_003) ==> (Directory not empty) [Directory not empty]
[2017-05-24 01:13:05.188788] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 134012985: RMDIR /smallfile/file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_06/d_006/d_005 (b1c93362-0b4f-4843-aa39-4643983a1542/d_005) ==> (Directory not empty) [Directory not empty]
[2017-05-24 02:40:38.092335] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 134227999: RMDIR /smallfile/file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_10/d_006/d_005/d_003 (a374dc24-dbab-47e6-b8f4-5614cb18b4e1/d_003) ==> (Directory not empty) [Directory not empty]
[2017-05-24 02:40:58.453841] I [MSGID: 115056] [server-rpc-fops.c:460:server_rmdir_cbk] 0-cross3-1-server: 134228845: RMDIR /smallfile/file_srcdir/dhcp37-197.lab.eng.blr.redhat.com/thrd_10/d_006/d_005 (b1d52e21-a122-4997-8189-e70aa47fec01/d_005) ==> (Directory not empty) [Directory not empty]
(END)



Version-Release number of selected component (if applicable):
=======
3.8.4-25

How reproducible:
==============
seen this quite a number of times in brick mux setup

Steps to Reproduce:
1.have brick mux enabled, and create say about 5 volumes and mount them on different fuse clients
2.populate files and dirs on these volumes
3. do a rm -rf * on these vols
Comment 2 Nag Pavan Chilakam 2017-05-24 06:21:28 UTC 
fuse logs:
2017-05-23 23:54:18.659255] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-cross3-50-replicate-0: Completed metadata selfheal on 788703ff-000d-4c4e-a70b-b033aeb06ec0. sources=[0] 1  sinks=2 
[2017-05-24 01:12:53.018340] W [MSGID: 114031] [client-rpc-fops.c:688:client3_3_rmdir_cbk] 0-cross3-50-client-2: remote operation failed [Directory not empty]
[2017-05-24 01:13:05.187161] W [MSGID: 114031] [client-rpc-fops.c:688:client3_3_rmdir_cbk] 0-cross3-50-client-2: remote operation failed [Directory not empty]
[2017-05-24 02:40:38.090196] W [MSGID: 114031] [client-rpc-fops.c:688:client3_3_rmdir_cbk] 0-cross3-50-client-2: remote operation failed [Directory not empty]
[2017-05-24 02:40:58.451769] W [MSGID: 114031] [client-rpc-fops.c:688:client3_3_rmdir_cbk] 0-cross3-50-client-2: remote operation failed [Directory not empty]
[2017-05-24 05:01:14.827159] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-cross3-50-replicate-0: performing metadata selfheal on 26b2407e-afe0-45ad-9c8d-84c34831bdad
[2017-05-24 05:01:15.923719] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-cross3-50-replicate-0: Completed metadata selfheal on 26b2407e-afe0-45ad-9c8d-84c34831bdad. sources=[0] 1  sinks=2 
(END)
Comment 4 Ravishankar N 2017-05-24 07:29:45 UTC 
Nag, Is this the bug filed for https://bugzilla.redhat.com/show_bug.cgi?id=1451720#c8 ? Can you attach the sos reports and provide volinfo details?
Comment 5 Nag Pavan Chilakam 2017-05-24 07:37:52 UTC 
(In reply to Ravishankar N from comment #4)
> Nag, Is this the bug filed for
> https://bugzilla.redhat.com/show_bug.cgi?id=1451720#c8 ? Can you attach the
> sos reports and provide volinfo details?

I filed it because I was seeing it frequently and hence need to track it seperately

sosreports http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1455034/
Comment 7 Nithya Balachandran 2017-05-24 07:52:59 UTC 
Are these dirs actually empty on the bricks?