Bug 1718155

Summary:	rm -rf fails for some directories with "directory not empty"
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Nag Pavan Chilakam <nchilaka>
Component:	replicate	Assignee:	Karthik U S <ksubrahm>
Status:	CLOSED DUPLICATE	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.5	CC:	rhs-bugs, sankarshan, saraut, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-07-23 05:03:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1593242
Bug Blocks:

Description Nag Pavan Chilakam 2019-06-07 06:10:18 UTC

Description of problem:
==================
rm -rf on some of the directories failed with "Directories not empty"

rm -rf was issued as reported in BZ#1718152

As part of the detailed steps mentioned in BZ#1718152, post rm -rf , a couple of directories failed to get deleted as below (in below example dir.41 failed to get deleted)




[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.41/linux-5.1.3’: Directory not empty

real    6562m49.361s
user    0m48.952s
sys     6m53.090s
[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]#
[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]#
[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]#
[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# time ls
dir.41  find.sh  linux-5.1.3.tar.xz  nag

real    0m0.354s
user    0m0.005s
sys     0m0.000s



[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# find dir.41/
dir.41/
dir.41/linux-5.1.3
dir.41/linux-5.1.3/drivers

It is also worth noting that rm -rf failed on other clients too at around similar times (as the directories failed was around dir37--dir42)


Version-Release number of selected component (if applicable):
=======================
6.0.3 on rhel7.7 beta

Steps to Reproduce:
.had a 4x3 volume on a 4 node cluster with quotas enabled
2. had run some tests as detailed out in steps in https://bugzilla.redhat.com/show_bug.cgi?id=1715438#c0

3. added a 5th node to the cluster
4. issued a replace brick to replace all bricks being hosted by n1 with the new n5
6. around same time, below IOs were going on from each client
 a. append to a file every 2 minutes with top o/p (one file each for each client
 b. had about 50 directories with linux untarred image in each in about 10 directories(1 for each client), for which I did rm -rf as above (including some log files, totalling to 100 addition files for each client), so each client would be trying to delete about 3.5 million files seperately(and not overlapping)


Actual results:
=================
rm -rf failed on some of the directories with dir not empty


for more details refer to BZ#1718152


c1:
[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.41/linux-5.1.3’: Directory not empty

real    6562m49.361s
user    0m48.952s
sys     6m53.090s
[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]#
[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]#
[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]#
[root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# time ls
dir.41  find.sh  linux-5.1.3.tar.xz  nag

real    0m0.354s
user    0m0.005s
sys     0m0.000s



c2:
[root@dhcp42-123 dhcp42-123.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.41/linux-5.1.3/arch/mips/include’: Directory not empty
rm: cannot remove ‘dir.42/linux-5.1.3/drivers/video’: Directory not empty

real    6615m34.006s
user    0m49.196s
sys     6m36.436s

c3:
[root@dhcp42-223 dhcp42-223.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.41/linux-5.1.3/drivers/net/ethernet’: Directory not empty

real    6583m56.382s
user    0m46.946s
sys     6m29.455s

c4:
[root@dhcp43-18 dhcp43-18.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.41/linux-5.1.3/drivers/net/wireless/marvell’: Directory not empty
rm: cannot remove ‘dir.42/linux-5.1.3’: Directory not empty

real    6589m5.156s
user    0m49.157s
sys     6m39.067s

c5:
sys     0m0.011s
[root@dhcp42-57 dhcp42-57.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.41/linux-5.1.3/drivers’: Directory not empty
rm: cannot remove ‘dir.42/linux-5.1.3’: Directory not empty

real    6592m29.748s
user    0m49.819s
sys     6m28.131s

c6:
[root@dhcp43-46 dhcp43-46.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.41/linux-5.1.3’: Directory not empty
rm: cannot remove ‘dir.42/linux-5.1.3/arch’: Directory not empty

real    6613m9.963s
user    0m47.070s
sys     6m31.563s

c7:
[root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.37/linux-5.1.3/tools/perf’: Directory not empty
rm: cannot remove ‘dir.37/linux-5.1.3/drivers’: Directory not empty

real    7249m34.626s
user    0m38.355s
sys     5m10.119s
[root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# 
[root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# 
[root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# 
[root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# time ls
dir.37  find.sh  linux-5.1.3.tar.xz  nag

c8:
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.37/linux-5.1.3/tools/testing/selftests’: Directory not empty
rm: cannot remove ‘dir.38/linux-5.1.3/arch’: Directory not empty

real    7250m50.768s
user    0m39.151s
sys     5m18.075s
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]#
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]#
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]#
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# time ls  
dir.37  dir.38  find.sh  linux-5.1.3.tar.xz  nag

real    0m0.611s
user    0m0.001s
sys     0m0.004s

c9:
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.37/linux-5.1.3/tools/testing/selftests’: Directory not empty
rm: cannot remove ‘dir.38/linux-5.1.3/arch’: Directory not empty

real    7250m50.768s
user    0m39.151s
sys     5m18.075s
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# 
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# 
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# 
[root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# time ls
dir.37  dir.38  find.sh  linux-5.1.3.tar.xz  nag

real    0m0.611s
user    0m0.001s
sys     0m0.004s


c10:
[root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar*
rm: cannot remove ‘dir.37/linux-5.1.3/arch/sh/kernel’: Directory not empty
rm: cannot remove ‘dir.37/linux-5.1.3/tools/testing’: Directory not empty
rm: cannot remove ‘dir.37/linux-5.1.3/drivers’: Directory not empty
rm: cannot remove ‘dir.38/linux-5.1.3/Documentation/devicetree’: Directory not empty
rm: cannot remove ‘dir.38/linux-5.1.3/arch/arm/mach-pxa/include’: Directory not empty

real    7247m26.930s
user    0m38.410s
sys     5m12.130s
[root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# 
[root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# 
[root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# 
[root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# time ls
dir.37  dir.38  find.sh  linux-5.1.3.tar.xz  nag

real    0m0.621s
user    0m0.002s
sys     0m0.001s

Comment 2 Nag Pavan Chilakam 2019-06-07 06:46:20 UTC

sosreports and logs same as BZ#1718152 and available @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1718152/

Comment 3 Nag Pavan Chilakam 2019-06-07 06:48:47 UTC

[root@rhs-gp-srv7 ~]# gluster v heal nftvol info 
Brick rhs-gp-srv5.lab.eng.blr.redhat.com:/gluster/brick1/nftvol-rb1
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.76/level5.87 
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.76 
/IOs/new-samedir/level1.1/level2.1/level3.45/level4.75/level5.12 - Is in split-brain
/IOs/new-samedir/level1.1/level2.1/level3.45/level4.75 
/IOs/new-samedir/level1.1/level2.2/level3.34/level4.80/level5.62 
<gfid:190f7dca-29d3-46e6-b798-aab6781a3c86> - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.64/level4.65/level5.35 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.64/level4.65 
/IOs/new-samedir/level1.1/level2.2/level3.79/level4.30/level5.87 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.79/level4.30 
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.17 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.23 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.28 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.41 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.96 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.54/level4.27/level5.56 
/IOs/new-samedir/level1.1/level2.3/level3.84/level4.55/level5.69 - Is in split-brain
Status: Connected
Number of entries: 17

Brick rhs-gp-srv8.lab.eng.blr.redhat.com:/gluster/brick1/nftvol-b1
/IOs/new-samedir/level1.1/level2.2/level3.34/level4.80/level5.62 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.34/level4.80 
/IOs/new-samedir/level1.1/level2.2/level3.64/level4.65/level5.35 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.64/level4.65 
/IOs/new-samedir/level1.1/level2.2/level3.79/level4.30/level5.87 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.79/level4.30 
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.17 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.23 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.28 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.41 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.96 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.54/level4.27 
/IOs/new-samedir/level1.1/level2.3/level3.54/level4.27/level5.56 
/IOs/new-samedir/level1.1/level2.3/level3.84/level4.55/level5.69 - Is in split-brain
/IOs/samedir-creates/level1.1/level2.3/level3.28/level4.13 
/IOs/samedir-creates/level1.1/level2.3/level3.28/level4.13/level5.9 
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.76/level5.87 
/IOs/kernel/dhcp43-21.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/scsi 
/IOs/new-samedir/level1.1/level2.1/level3.45/level4.75/level5.12 
<gfid:17e99f85-9526-4ab0-aa39-22eb932732be> - Is in split-brain
Status: Connected
Number of entries: 20

Brick rhs-gp-srv9.lab.eng.blr.redhat.com:/gluster/brick1/nftvol-b1
/IOs/samedir-creates/level1.1/level2.3/level3.28/level4.13/level5.9 
/IOs/new-samedir/level1.1/level2.2/level3.34/level4.80/level5.62 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.34/level4.80 
/IOs/new-samedir/level1.1/level2.2/level3.64/level4.65/level5.35 
<gfid:5405eaf8-2589-42dc-afde-1400e4da45ca> - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.79/level4.30/level5.87 
<gfid:8f772dfe-d7d8-4af4-bddd-2960881972ac> - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.17 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.23 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.28 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.41 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.96 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.54/level4.27 
/IOs/new-samedir/level1.1/level2.3/level3.54/level4.27/level5.56 
/IOs/new-samedir/level1.1/level2.3/level3.84/level4.55/level5.69 - Is in split-brain
/IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/char 
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.76/level5.87 
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.76 
/IOs/samedir-creates/level1.1/level2.3/level3.28/level4.13 
/IOs/new-samedir/level1.1/level2.1/level3.45/level4.75 
/IOs/new-samedir/level1.1/level2.1/level3.45/level4.75/level5.12 - Is in split-brain
Status: Connected
Number of entries: 21

Brick rhs-gp-srv8.lab.eng.blr.redhat.com:/gluster/brick2/nftvol-b2
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.7/level5.60 - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.35 - Is in split-brain
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.50 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.59 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.3 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.30/level4.29/level5.98 
<gfid:cb27b874-41b8-4909-b14b-6216c5f78b10> - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.60/level4.50/level5.30 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.60/level4.50 
/IOs/new-samedir/level1.1/level2.4/level3.33/level4.5/level5.77 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.33/level4.5 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.30 - Is in split-brain
Status: Connected
Number of entries: 14

Brick rhs-gp-srv9.lab.eng.blr.redhat.com:/gluster/brick2/nftvol-b2
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.50 
/IOs/new-samedir/level1.1/level2.2/level3.30/level4.29/level5.98 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.30/level4.29 
/IOs/new-samedir/level1.1/level2.3/level3.60/level4.50/level5.30 
<gfid:bf2c8e61-59c5-4c37-a0d1-6f67b1f4bae1> - Is in split-brain
<gfid:1e7c2e51-86b9-490f-acd5-519eb42cd4ff> - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.7/level5.60 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.33/level4.5/level5.77 
/IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/char 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.35 - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.3 - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.59 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.30 - Is in split-brain
Status: Connected
Number of entries: 15

Brick rhs-gp-srv10.lab.eng.blr.redhat.com:/gluster/brick1/nftvol-b2
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.7/level5.60 - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.35 - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.59 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.3 - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.30 - Is in split-brain
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.50 
/IOs/new-samedir/level1.1/level2.2/level3.30/level4.29/level5.98 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.30/level4.29 
/IOs/new-samedir/level1.1/level2.3/level3.60/level4.50/level5.30 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.60/level4.50 
/IOs/new-samedir/level1.1/level2.4/level3.33/level4.5/level5.77 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.33/level4.5 
Status: Connected
Number of entries: 12

Brick rhs-gp-srv9.lab.eng.blr.redhat.com:/gluster/brick3/nftvol-b3
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.17 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.83 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.97/level4.30/level5.76 
<gfid:8ba6523c-65fc-44ba-a2fd-cd983ba737ea> - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.38/level4.94/level5.5 
<gfid:f207a315-a5a1-4f17-8a34-bdeb67d5a184> - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.5/level4.67/level5.97 
<gfid:1ca8bc74-2b0e-4e88-a9b6-76b2b978e127> - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.23 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.44 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.91 - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.37 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.34 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.100 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.5 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.23 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.75 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66 
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.68/level5.62 
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.68 
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.44 
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.44/level5.9 
/IOs/new-samedir/level1.1/level2.1/level3.33/level4.34/level5.76 
/IOs/new-samedir/level1.1/level2.1/level3.33/level4.34 
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.46 
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.46/level5.53 
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.65/level5.23 
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.65 
/IOs/new-samedir/level1.1/level2.1/level3.27/level4.45 
/IOs/new-samedir/level1.1/level2.1/level3.27/level4.45/level5.85 
/IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/char 
/IOs/new-samedir/level1.1/level2.1/level3.35/level4.37/level5.36 
Status: Connected
Number of entries: 34

Brick rhs-gp-srv10.lab.eng.blr.redhat.com:/gluster/brick2/nftvol-b3
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.83 - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.100 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.37 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.75 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.5 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.17 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.23 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.34 
/IOs/new-samedir/level1.1/level2.2/level3.97/level4.30/level5.76 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.97/level4.30 
/IOs/new-samedir/level1.1/level2.3/level3.38/level4.94/level5.5 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.38/level4.94 
/IOs/new-samedir/level1.1/level2.4/level3.5/level4.67/level5.97 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.5/level4.67 
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.23 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.44 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.91 - Is in split-brain
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.8 
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.68/level5.62 
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.68 
/IOs/new-samedir/level1.1/level2.1/level3.33/level4.34/level5.76 
/IOs/new-samedir/level1.1/level2.1/level3.33/level4.34 
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.46 
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.46/level5.53 
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.65/level5.23 
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.65 
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.44/level5.9 
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.44 
/IOs/new-samedir/level1.1/level2.1/level3.27/level4.45 
/IOs/new-samedir/level1.1/level2.1/level3.27/level4.45/level5.85 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67 
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66 
/IOs/new-samedir/level1.1/level2.1/level3.35/level4.37 
/IOs/new-samedir/level1.1/level2.1/level3.35/level4.37/level5.36 
Status: Connected
Number of entries: 34

Brick rhs-gp-srv5.lab.eng.blr.redhat.com:/gluster/brick2/nftvol-rb3
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.46/level5.53 
/IOs/new-samedir/level1.1/level2.1/level3.3/level4.65/level5.23 
/IOs/new-samedir/level1.1/level2.1/level3.27/level4.45/level5.85 
/IOs/new-samedir/level1.1/level2.1/level3.30/level4.68/level5.62 
/IOs/new-samedir/level1.1/level2.1/level3.33/level4.34/level5.76 
/IOs/new-samedir/level1.1/level2.1/level3.35/level4.37/level5.36 
/IOs/new-samedir/level1.1/level2.1/level3.35/level4.37 
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.44/level5.9 
/IOs/new-samedir/level1.1/level2.2/level3.97/level4.30/level5.76 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.97/level4.30 
/IOs/new-samedir/level1.1/level2.3/level3.38/level4.94/level5.5 - Is in split-brain
/IOs/new-samedir/level1.1/level2.3/level3.38/level4.94 
/IOs/new-samedir/level1.1/level2.4/level3.5/level4.67/level5.97 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.5/level4.67 
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.23 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.44 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.91 - Is in split-brain
Status: Connected
Number of entries: 17

Brick rhs-gp-srv10.lab.eng.blr.redhat.com:/gluster/brick3/nftvol-b4
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.44 - Is in split-brain
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7 
/IOs/new-samedir/level1.1/level2.1/level3.1 
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.91/level5.15 - Is in split-brain
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.91 
/IOs/new-samedir/level1.1/level2.2/level3.57/level4.64/level5.88 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.57/level4.64 
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.37/level5.98 - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.37 
/IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.38/linux-5.1.3/Documentation/devicetree/bindings 
/IOs/kernel/dhcp43-18.lab.eng.blr.redhat.com/dir.42/linux-5.1.3/tools 
/IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.38/linux-5.1.3/arch/arm/mach-pxa/include/mach 
Status: Connected
Number of entries: 12

Brick rhs-gp-srv5.lab.eng.blr.redhat.com:/gluster/brick3/nftvol-rb4
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.91/level5.15 - Is in split-brain
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.91 
/IOs/new-samedir/level1.1/level2.2/level3.57/level4.64/level5.88 - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.57/level4.64 
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.37 
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.37/level5.98 - Is in split-brain
Status: Connected
Number of entries: 6

Brick rhs-gp-srv8.lab.eng.blr.redhat.com:/gluster/brick3/nftvol-b4
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.4 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.1 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.2 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.3 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.5 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.46 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.47 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.48 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.50 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.51 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.49 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.52 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.53 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.54 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.55 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.56 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.57 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.58 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.59 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.60 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.61 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.62 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.63 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.64 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.65 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.69 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.71 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.73 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.74 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.76 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.77 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.80 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.81 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.84 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.86 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.88 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.91 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.93 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.94 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.97 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.98 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.66 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.67 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.68 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.70 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.72 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.75 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.78 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.79 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.82 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.83 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.85 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.87 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.89 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.90 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.92 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.95 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.96 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.99 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.100 
/IOs/new-samedir/level1.1/level2.1/level3.61/level4.91/level5.15 
<gfid:41023c2e-c5c3-452a-8796-3f8629561aaf> - Is in split-brain
/IOs/new-samedir/level1.1/level2.2/level3.57/level4.64/level5.88 
<gfid:cde95c92-fa52-4671-86dc-51b7c78583d9> - Is in split-brain
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.37 
/IOs/new-samedir/level1.1/level2.4/level3.9/level4.37/level5.98 
<gfid:b9db1153-996b-4a5a-8530-181e8892b9c6> - Is in split-brain
/IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.44 - Is in split-brain
/IOs/new-samedir/level1.1/level2.1/level3.1 
/IOs/kernel/dhcp42-166.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/tools/testing/selftests/powerpc 
/IOs/kernel/dhcp43-21.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/scsi 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.6 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.7 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.8 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.9 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.10 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.11 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.12 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.13 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.14 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.15 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.16 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.17 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.18 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.19 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.20 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.21 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.22 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.23 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.24 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.25 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.26 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.27 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.28 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.29 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.30 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.31 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.32 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.33 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.34 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.35 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.36 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.37 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.39 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.40 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.38 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.41 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.42 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.43 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.44 
/IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.45 
Status: Connected
Number of entries: 112