Description of problem: ================== rm -rf on some of the directories failed with "Directories not empty" rm -rf was issued as reported in BZ#1718152 As part of the detailed steps mentioned in BZ#1718152, post rm -rf , a couple of directories failed to get deleted as below (in below example dir.41 failed to get deleted) [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.41/linux-5.1.3’: Directory not empty real 6562m49.361s user 0m48.952s sys 6m53.090s [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# time ls dir.41 find.sh linux-5.1.3.tar.xz nag real 0m0.354s user 0m0.005s sys 0m0.000s [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# find dir.41/ dir.41/ dir.41/linux-5.1.3 dir.41/linux-5.1.3/drivers It is also worth noting that rm -rf failed on other clients too at around similar times (as the directories failed was around dir37--dir42) Version-Release number of selected component (if applicable): ======================= 6.0.3 on rhel7.7 beta Steps to Reproduce: .had a 4x3 volume on a 4 node cluster with quotas enabled 2. had run some tests as detailed out in steps in https://bugzilla.redhat.com/show_bug.cgi?id=1715438#c0 3. added a 5th node to the cluster 4. issued a replace brick to replace all bricks being hosted by n1 with the new n5 6. around same time, below IOs were going on from each client a. append to a file every 2 minutes with top o/p (one file each for each client b. had about 50 directories with linux untarred image in each in about 10 directories(1 for each client), for which I did rm -rf as above (including some log files, totalling to 100 addition files for each client), so each client would be trying to delete about 3.5 million files seperately(and not overlapping) Actual results: ================= rm -rf failed on some of the directories with dir not empty for more details refer to BZ#1718152 c1: [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.41/linux-5.1.3’: Directory not empty real 6562m49.361s user 0m48.952s sys 6m53.090s [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# [root@dhcp42-60 dhcp42-60.lab.eng.blr.redhat.com]# time ls dir.41 find.sh linux-5.1.3.tar.xz nag real 0m0.354s user 0m0.005s sys 0m0.000s c2: [root@dhcp42-123 dhcp42-123.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.41/linux-5.1.3/arch/mips/include’: Directory not empty rm: cannot remove ‘dir.42/linux-5.1.3/drivers/video’: Directory not empty real 6615m34.006s user 0m49.196s sys 6m36.436s c3: [root@dhcp42-223 dhcp42-223.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.41/linux-5.1.3/drivers/net/ethernet’: Directory not empty real 6583m56.382s user 0m46.946s sys 6m29.455s c4: [root@dhcp43-18 dhcp43-18.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.41/linux-5.1.3/drivers/net/wireless/marvell’: Directory not empty rm: cannot remove ‘dir.42/linux-5.1.3’: Directory not empty real 6589m5.156s user 0m49.157s sys 6m39.067s c5: sys 0m0.011s [root@dhcp42-57 dhcp42-57.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.41/linux-5.1.3/drivers’: Directory not empty rm: cannot remove ‘dir.42/linux-5.1.3’: Directory not empty real 6592m29.748s user 0m49.819s sys 6m28.131s c6: [root@dhcp43-46 dhcp43-46.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.41/linux-5.1.3’: Directory not empty rm: cannot remove ‘dir.42/linux-5.1.3/arch’: Directory not empty real 6613m9.963s user 0m47.070s sys 6m31.563s c7: [root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.37/linux-5.1.3/tools/perf’: Directory not empty rm: cannot remove ‘dir.37/linux-5.1.3/drivers’: Directory not empty real 7249m34.626s user 0m38.355s sys 5m10.119s [root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# [root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# [root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# [root@dhcp43-21 dhcp43-21.lab.eng.blr.redhat.com]# time ls dir.37 find.sh linux-5.1.3.tar.xz nag c8: [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.37/linux-5.1.3/tools/testing/selftests’: Directory not empty rm: cannot remove ‘dir.38/linux-5.1.3/arch’: Directory not empty real 7250m50.768s user 0m39.151s sys 5m18.075s [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# time ls dir.37 dir.38 find.sh linux-5.1.3.tar.xz nag real 0m0.611s user 0m0.001s sys 0m0.004s c9: [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.37/linux-5.1.3/tools/testing/selftests’: Directory not empty rm: cannot remove ‘dir.38/linux-5.1.3/arch’: Directory not empty real 7250m50.768s user 0m39.151s sys 5m18.075s [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# [root@dhcp42-166 dhcp42-166.lab.eng.blr.redhat.com]# time ls dir.37 dir.38 find.sh linux-5.1.3.tar.xz nag real 0m0.611s user 0m0.001s sys 0m0.004s c10: [root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# time rm -rf dir* tar* untar* rm: cannot remove ‘dir.37/linux-5.1.3/arch/sh/kernel’: Directory not empty rm: cannot remove ‘dir.37/linux-5.1.3/tools/testing’: Directory not empty rm: cannot remove ‘dir.37/linux-5.1.3/drivers’: Directory not empty rm: cannot remove ‘dir.38/linux-5.1.3/Documentation/devicetree’: Directory not empty rm: cannot remove ‘dir.38/linux-5.1.3/arch/arm/mach-pxa/include’: Directory not empty real 7247m26.930s user 0m38.410s sys 5m12.130s [root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# [root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# [root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# [root@dhcp43-234 dhcp43-234.lab.eng.blr.redhat.com]# time ls dir.37 dir.38 find.sh linux-5.1.3.tar.xz nag real 0m0.621s user 0m0.002s sys 0m0.001s
sosreports and logs same as BZ#1718152 and available @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1718152/
[root@rhs-gp-srv7 ~]# gluster v heal nftvol info Brick rhs-gp-srv5.lab.eng.blr.redhat.com:/gluster/brick1/nftvol-rb1 /IOs/new-samedir/level1.1/level2.1/level3.30/level4.76/level5.87 /IOs/new-samedir/level1.1/level2.1/level3.30/level4.76 /IOs/new-samedir/level1.1/level2.1/level3.45/level4.75/level5.12 - Is in split-brain /IOs/new-samedir/level1.1/level2.1/level3.45/level4.75 /IOs/new-samedir/level1.1/level2.2/level3.34/level4.80/level5.62 <gfid:190f7dca-29d3-46e6-b798-aab6781a3c86> - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.64/level4.65/level5.35 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.64/level4.65 /IOs/new-samedir/level1.1/level2.2/level3.79/level4.30/level5.87 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.79/level4.30 /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.17 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.23 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.28 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.41 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.96 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.54/level4.27/level5.56 /IOs/new-samedir/level1.1/level2.3/level3.84/level4.55/level5.69 - Is in split-brain Status: Connected Number of entries: 17 Brick rhs-gp-srv8.lab.eng.blr.redhat.com:/gluster/brick1/nftvol-b1 /IOs/new-samedir/level1.1/level2.2/level3.34/level4.80/level5.62 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.34/level4.80 /IOs/new-samedir/level1.1/level2.2/level3.64/level4.65/level5.35 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.64/level4.65 /IOs/new-samedir/level1.1/level2.2/level3.79/level4.30/level5.87 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.79/level4.30 /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.17 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.23 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.28 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.41 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.96 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.54/level4.27 /IOs/new-samedir/level1.1/level2.3/level3.54/level4.27/level5.56 /IOs/new-samedir/level1.1/level2.3/level3.84/level4.55/level5.69 - Is in split-brain /IOs/samedir-creates/level1.1/level2.3/level3.28/level4.13 /IOs/samedir-creates/level1.1/level2.3/level3.28/level4.13/level5.9 /IOs/new-samedir/level1.1/level2.1/level3.30/level4.76/level5.87 /IOs/kernel/dhcp43-21.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/scsi /IOs/new-samedir/level1.1/level2.1/level3.45/level4.75/level5.12 <gfid:17e99f85-9526-4ab0-aa39-22eb932732be> - Is in split-brain Status: Connected Number of entries: 20 Brick rhs-gp-srv9.lab.eng.blr.redhat.com:/gluster/brick1/nftvol-b1 /IOs/samedir-creates/level1.1/level2.3/level3.28/level4.13/level5.9 /IOs/new-samedir/level1.1/level2.2/level3.34/level4.80/level5.62 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.34/level4.80 /IOs/new-samedir/level1.1/level2.2/level3.64/level4.65/level5.35 <gfid:5405eaf8-2589-42dc-afde-1400e4da45ca> - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.79/level4.30/level5.87 <gfid:8f772dfe-d7d8-4af4-bddd-2960881972ac> - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.17 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.23 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.28 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.41 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.20/level4.30/level5.96 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.54/level4.27 /IOs/new-samedir/level1.1/level2.3/level3.54/level4.27/level5.56 /IOs/new-samedir/level1.1/level2.3/level3.84/level4.55/level5.69 - Is in split-brain /IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/char /IOs/new-samedir/level1.1/level2.1/level3.30/level4.76/level5.87 /IOs/new-samedir/level1.1/level2.1/level3.30/level4.76 /IOs/samedir-creates/level1.1/level2.3/level3.28/level4.13 /IOs/new-samedir/level1.1/level2.1/level3.45/level4.75 /IOs/new-samedir/level1.1/level2.1/level3.45/level4.75/level5.12 - Is in split-brain Status: Connected Number of entries: 21 Brick rhs-gp-srv8.lab.eng.blr.redhat.com:/gluster/brick2/nftvol-b2 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.7/level5.60 - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.35 - Is in split-brain /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.50 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.59 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.3 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.30/level4.29/level5.98 <gfid:cb27b874-41b8-4909-b14b-6216c5f78b10> - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.60/level4.50/level5.30 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.60/level4.50 /IOs/new-samedir/level1.1/level2.4/level3.33/level4.5/level5.77 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.33/level4.5 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.30 - Is in split-brain Status: Connected Number of entries: 14 Brick rhs-gp-srv9.lab.eng.blr.redhat.com:/gluster/brick2/nftvol-b2 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.50 /IOs/new-samedir/level1.1/level2.2/level3.30/level4.29/level5.98 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.30/level4.29 /IOs/new-samedir/level1.1/level2.3/level3.60/level4.50/level5.30 <gfid:bf2c8e61-59c5-4c37-a0d1-6f67b1f4bae1> - Is in split-brain <gfid:1e7c2e51-86b9-490f-acd5-519eb42cd4ff> - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.7/level5.60 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.33/level4.5/level5.77 /IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/char /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.35 - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.3 - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.59 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.30 - Is in split-brain Status: Connected Number of entries: 15 Brick rhs-gp-srv10.lab.eng.blr.redhat.com:/gluster/brick1/nftvol-b2 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.7/level5.60 - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.35 - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.59 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.3 - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.68/level5.30 - Is in split-brain /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.50 /IOs/new-samedir/level1.1/level2.2/level3.30/level4.29/level5.98 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.30/level4.29 /IOs/new-samedir/level1.1/level2.3/level3.60/level4.50/level5.30 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.60/level4.50 /IOs/new-samedir/level1.1/level2.4/level3.33/level4.5/level5.77 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.33/level4.5 Status: Connected Number of entries: 12 Brick rhs-gp-srv9.lab.eng.blr.redhat.com:/gluster/brick3/nftvol-b3 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.17 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.83 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.97/level4.30/level5.76 <gfid:8ba6523c-65fc-44ba-a2fd-cd983ba737ea> - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.38/level4.94/level5.5 <gfid:f207a315-a5a1-4f17-8a34-bdeb67d5a184> - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.5/level4.67/level5.97 <gfid:1ca8bc74-2b0e-4e88-a9b6-76b2b978e127> - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.23 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.44 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.91 - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.37 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.34 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.100 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.5 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.23 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.75 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66 /IOs/new-samedir/level1.1/level2.1/level3.30/level4.68/level5.62 /IOs/new-samedir/level1.1/level2.1/level3.30/level4.68 /IOs/new-samedir/level1.1/level2.1/level3.61/level4.44 /IOs/new-samedir/level1.1/level2.1/level3.61/level4.44/level5.9 /IOs/new-samedir/level1.1/level2.1/level3.33/level4.34/level5.76 /IOs/new-samedir/level1.1/level2.1/level3.33/level4.34 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.46 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.46/level5.53 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.65/level5.23 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.65 /IOs/new-samedir/level1.1/level2.1/level3.27/level4.45 /IOs/new-samedir/level1.1/level2.1/level3.27/level4.45/level5.85 /IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/char /IOs/new-samedir/level1.1/level2.1/level3.35/level4.37/level5.36 Status: Connected Number of entries: 34 Brick rhs-gp-srv10.lab.eng.blr.redhat.com:/gluster/brick2/nftvol-b3 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.83 - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66/level5.100 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.37 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.75 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.5 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.17 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.23 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8/level5.34 /IOs/new-samedir/level1.1/level2.2/level3.97/level4.30/level5.76 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.97/level4.30 /IOs/new-samedir/level1.1/level2.3/level3.38/level4.94/level5.5 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.38/level4.94 /IOs/new-samedir/level1.1/level2.4/level3.5/level4.67/level5.97 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.5/level4.67 /IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.23 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.44 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.91 - Is in split-brain /IOs/new-samedir/level1.1/level2.1/level3.1/level4.8 /IOs/new-samedir/level1.1/level2.1/level3.30/level4.68/level5.62 /IOs/new-samedir/level1.1/level2.1/level3.30/level4.68 /IOs/new-samedir/level1.1/level2.1/level3.33/level4.34/level5.76 /IOs/new-samedir/level1.1/level2.1/level3.33/level4.34 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.46 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.46/level5.53 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.65/level5.23 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.65 /IOs/new-samedir/level1.1/level2.1/level3.61/level4.44/level5.9 /IOs/new-samedir/level1.1/level2.1/level3.61/level4.44 /IOs/new-samedir/level1.1/level2.1/level3.27/level4.45 /IOs/new-samedir/level1.1/level2.1/level3.27/level4.45/level5.85 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.66 /IOs/new-samedir/level1.1/level2.1/level3.35/level4.37 /IOs/new-samedir/level1.1/level2.1/level3.35/level4.37/level5.36 Status: Connected Number of entries: 34 Brick rhs-gp-srv5.lab.eng.blr.redhat.com:/gluster/brick2/nftvol-rb3 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.46/level5.53 /IOs/new-samedir/level1.1/level2.1/level3.3/level4.65/level5.23 /IOs/new-samedir/level1.1/level2.1/level3.27/level4.45/level5.85 /IOs/new-samedir/level1.1/level2.1/level3.30/level4.68/level5.62 /IOs/new-samedir/level1.1/level2.1/level3.33/level4.34/level5.76 /IOs/new-samedir/level1.1/level2.1/level3.35/level4.37/level5.36 /IOs/new-samedir/level1.1/level2.1/level3.35/level4.37 /IOs/new-samedir/level1.1/level2.1/level3.61/level4.44/level5.9 /IOs/new-samedir/level1.1/level2.2/level3.97/level4.30/level5.76 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.97/level4.30 /IOs/new-samedir/level1.1/level2.3/level3.38/level4.94/level5.5 - Is in split-brain /IOs/new-samedir/level1.1/level2.3/level3.38/level4.94 /IOs/new-samedir/level1.1/level2.4/level3.5/level4.67/level5.97 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.5/level4.67 /IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.23 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.44 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.9/level4.40/level5.91 - Is in split-brain Status: Connected Number of entries: 17 Brick rhs-gp-srv10.lab.eng.blr.redhat.com:/gluster/brick3/nftvol-b4 /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.44 - Is in split-brain /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7 /IOs/new-samedir/level1.1/level2.1/level3.1 /IOs/new-samedir/level1.1/level2.1/level3.61/level4.91/level5.15 - Is in split-brain /IOs/new-samedir/level1.1/level2.1/level3.61/level4.91 /IOs/new-samedir/level1.1/level2.2/level3.57/level4.64/level5.88 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.57/level4.64 /IOs/new-samedir/level1.1/level2.4/level3.9/level4.37/level5.98 - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.9/level4.37 /IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.38/linux-5.1.3/Documentation/devicetree/bindings /IOs/kernel/dhcp43-18.lab.eng.blr.redhat.com/dir.42/linux-5.1.3/tools /IOs/kernel/dhcp43-234.lab.eng.blr.redhat.com/dir.38/linux-5.1.3/arch/arm/mach-pxa/include/mach Status: Connected Number of entries: 12 Brick rhs-gp-srv5.lab.eng.blr.redhat.com:/gluster/brick3/nftvol-rb4 /IOs/new-samedir/level1.1/level2.1/level3.61/level4.91/level5.15 - Is in split-brain /IOs/new-samedir/level1.1/level2.1/level3.61/level4.91 /IOs/new-samedir/level1.1/level2.2/level3.57/level4.64/level5.88 - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.57/level4.64 /IOs/new-samedir/level1.1/level2.4/level3.9/level4.37 /IOs/new-samedir/level1.1/level2.4/level3.9/level4.37/level5.98 - Is in split-brain Status: Connected Number of entries: 6 Brick rhs-gp-srv8.lab.eng.blr.redhat.com:/gluster/brick3/nftvol-b4 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.4 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.1 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.2 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.3 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.5 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.46 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.47 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.48 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.50 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.51 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.49 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.52 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.53 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.54 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.55 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.56 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.57 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.58 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.59 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.60 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.61 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.62 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.63 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.64 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.65 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.69 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.71 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.73 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.74 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.76 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.77 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.80 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.81 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.84 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.86 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.88 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.91 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.93 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.94 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.97 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.98 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.66 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.67 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.68 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.70 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.72 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.75 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.78 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.79 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.82 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.83 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.85 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.87 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.89 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.90 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.92 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.95 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.96 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.99 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.100 /IOs/new-samedir/level1.1/level2.1/level3.61/level4.91/level5.15 <gfid:41023c2e-c5c3-452a-8796-3f8629561aaf> - Is in split-brain /IOs/new-samedir/level1.1/level2.2/level3.57/level4.64/level5.88 <gfid:cde95c92-fa52-4671-86dc-51b7c78583d9> - Is in split-brain /IOs/new-samedir/level1.1/level2.4/level3.9/level4.37 /IOs/new-samedir/level1.1/level2.4/level3.9/level4.37/level5.98 <gfid:b9db1153-996b-4a5a-8530-181e8892b9c6> - Is in split-brain /IOs/samedir-creates/newiter/level1.1/level2.1/level3.1/level4.67/level5.44 - Is in split-brain /IOs/new-samedir/level1.1/level2.1/level3.1 /IOs/kernel/dhcp42-166.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/tools/testing/selftests/powerpc /IOs/kernel/dhcp43-21.lab.eng.blr.redhat.com/dir.37/linux-5.1.3/drivers/scsi /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.6 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.7 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.8 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.9 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.10 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.11 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.12 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.13 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.14 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.15 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.16 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.17 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.18 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.19 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.20 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.21 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.22 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.23 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.24 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.25 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.26 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.27 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.28 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.29 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.30 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.31 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.32 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.33 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.34 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.35 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.36 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.37 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.39 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.40 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.38 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.41 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.42 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.43 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.44 /IOs/new-samedir/level1.1/level2.1/level3.1/level4.7/level5.45 Status: Connected Number of entries: 112