Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: ====================== on my test cluster , I see that the fuse mount logs say "insufficient available children for this request" even though all bricks are up. My cluster test bed is as below: =========================== This setup was trying to mimic commvault(cvlt) testbed 3 node cluster, brickmux enabled 1 1x3 volume 1 12x(4+2) ec volume--->each node hosts 2 bricks of same disperse set --->other-eager-lock disabled due to existing issue BZ#1703455 both volumes mounted on localhost I see some operation failures too in the logs as below: [2019-06-02 16:17:33.995107] E [fuse-bridge.c:220:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7febbc03197b] (--> /usr/lib64/glusterfs/6.0/xlator/mount/fuse.so(+0x81d1)[0x7febb33dc1d1] (--> /usr/lib64/glusterfs/6.0/xlator/mount/fuse.so(+0x9908)[0x7febb33dd908] (--> /lib64/libpthread.so.0(+0x7ea5)[0x7febbae6eea5] (--> /lib64/libc.so.6(clone+0x6d)[0x7febba7348cd] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory [2019-06-02 19:37:04.237597] E [MSGID: 122034] [ec-common.c:701:ec_child_select] 0-cvlt-ecv-disperse-5: Insufficient available children for this request (have 0, need 4). FOP : 'READ' failed on gfid 40558ba9-406c-42e3-b8b7-5379febaa547 [2019-06-02 19:37:04.244809] E [MSGID: 122034] [ec-common.c:701:ec_child_select] 0-cvlt-ecv-disperse-5: Insufficient available children for this request (have 0, need 4). FOP : 'FXATTROP' failed on gfid 40558ba9-406c-42e3-b8b7-5379febaa547 [2019-06-02 19:37:04.244832] E [MSGID: 122037] [ec-common.c:2268:ec_update_size_version_done] 0-cvlt-ecv-disperse-5: Failed to update version and size. FOP : 'FXATTROP' failed on gfid 40558ba9-406c-42e3-b8b7-5379febaa547 [Input/output error] [2019-06-02 19:44:55.818256] E [MSGID: 122034] [ec-common.c:701:ec_child_select] 0-cvlt-ecv-disperse-9: Insufficient available children for this request (have 0, need 4). FOP : 'READ' failed on gfid 2dc5e95f-a440-4c9b-8bbc-b768154fefd6 [2019-06-02 19:44:55.818396] E [MSGID: 122034] [ec-common.c:701:ec_child_select] 0-cvlt-ecv-disperse-9: Insufficient available children for this request (have 0, need 4). FOP : 'FXATTROP' failed on gfid 2dc5e95f-a440-4c9b-8bbc-b768154fefd6 [2019-06-02 19:44:55.818422] E [MSGID: 122037] [ec-common.c:2268:ec_update_size_version_done] 0-cvlt-ecv-disperse-9: Failed to update version and size. FOP : 'FXATTROP' failed on gfid 2dc5e95f-a440-4c9b-8bbc-b768154fefd6 [Input/output error] [2019-06-03 01:16:55.409333] E [MSGID: 122034] [ec-common.c:701:ec_child_select] 0-cvlt-ecv-disperse-9: Insufficient available children for this request (have 0, need 4). FOP : 'READ' failed on gfid 23b6b28c-fddb-45cc-b315-c7eb22af9c14 [2019-06-03 01:16:55.416443] E [MSGID: 122034] [ec-common.c:701:ec_child_select] 0-cvlt-ecv-disperse-9: Insufficient available children for this request (have 0, need 4). FOP : 'FXATTROP' failed on gfid 23b6b28c-fddb-45cc-b315-c7eb22af9c14 [2019-06-03 01:16:55.416466] E [MSGID: 122037] [ec-common.c:2268:ec_update_size_version_done] 0-cvlt-ecv-disperse-9: Failed to update version and size. FOP : 'FXATTROP' failed on gfid 23b6b28c-fddb-45cc-b315-c7eb22af9c14 [Input/output error] [2019-06-03 01:50:50.063952] E [fuse-bridge.c:220:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7febbc03197b] (--> /usr/lib64/glusterfs/6.0/xlator/mount/fuse.so(+0x81d1)[0x7febb33dc1d1] (--> /usr/lib64/glusterfs/6.0/xlator/mount/fuse.so(+0x9908)[0x7febb33dd908] (--> /lib64/libpthread.so.0(+0x7ea5)[0x7febbae6eea5] (--> /lib64/libc.so.6(clone+0x6d)[0x7febba7348cd] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory Version-Release number of selected component (if applicable): ====================== 6.0.3 on rhel7.7 beta 23-may compose How reproducible: ================ consistent every few hours on my cluster Steps to Reproduce: 1. created a 3 node cluster, enabled brickmux 2. created a 1x3 volume 3. created a 12x(4+2) ec volume-->each node hosts 2 bricks of same disperse set, other-eager-lock disabled 4. fuse mounted both the volumes, on each localhost IOs as below =========== 1) started capturing top o/p every 2 minutes of each node -->gets appended to file being hosted on the ec volume, one file each for every node 2) started sequential IO, using dd in below pattern: for 10 iterations--> {create 10000 files of size 50MB on ec volume, then sleep for 5minutes, followed by deletion of all the 10k files created in this iteration, again sleep 5min } Also, log the details in a file which is hosted by the 1x3 volume(including any errors during dd or rm , time, etc) for j in {1..10};do for i in {1..10000};do echo "########## LOOP $i in $j #################" |& tee -a /mnt/logvol/$HOSTNAME-dd-err.log;date |& tee -a /mnt/logvol/$HOSTNAME-dd-err.log;dd if=/dev/urandom of=file.$i.$j bs=1024 count=50000 2>>/mnt/logvol/$HOSTNAME-dd-err.log;date |& tee -a /mnt/logvol/$HOSTNAME-dd-err.log;done;sleep 300; echo "slept for 5min before rm -rf " |& tee -a /mnt/logvol/$HOSTNAME-dd-err.log;rm -rf file.*.$j 2>>/mnt/logvol/$HOSTNAME-dd-err.log;sleep 300; echo "slept for 5min POST rm -rf " |& tee -a /mnt/logvol/$HOSTNAME-dd-err.log;done Actual results: =============== seeing errors in fuse mount related to downtime of bricks and fxattr fails, even though no brick was brought down Volume info: =========== [root@rhs-gp-srv1 glusterfs]# gluster v info Volume Name: cvlt-ecv Type: Distributed-Disperse Volume ID: c500e86b-f505-48c8-8141-cb3de5956c24 Status: Started Snapshot Count: 0 Number of Bricks: 12 x (4 + 2) = 72 Transport-type: tcp Bricks: Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-ecv-sv1 Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-ecv-sv1 Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-ecv-sv1 Brick4: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-ecv-sv1 Brick5: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-ecv-sv1 Brick6: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-ecv-sv1 Brick7: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-ecv-sv2 Brick8: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-ecv-sv2 Brick9: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-ecv-sv2 Brick10: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-ecv-sv2 Brick11: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-ecv-sv2 Brick12: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-ecv-sv2 Brick13: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-ecv-sv3 Brick14: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-ecv-sv3 Brick15: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-ecv-sv3 Brick16: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-ecv-sv3 Brick17: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-ecv-sv3 Brick18: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-ecv-sv3 Brick19: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-ecv-sv4 Brick20: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-ecv-sv4 Brick21: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-ecv-sv4 Brick22: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-ecv-sv4 Brick23: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-ecv-sv4 Brick24: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-ecv-sv4 Brick25: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-ecv-sv5 Brick26: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-ecv-sv5 Brick27: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-ecv-sv5 Brick28: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-ecv-sv5 Brick29: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-ecv-sv5 Brick30: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-ecv-sv5 Brick31: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-ecv-sv6 Brick32: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-ecv-sv6 Brick33: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-ecv-sv6 Brick34: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-ecv-sv6 Brick35: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-ecv-sv6 Brick36: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-ecv-sv6 Brick37: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-ecv-sv7 Brick38: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-ecv-sv7 Brick39: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-ecv-sv7 Brick40: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-ecv-sv7 Brick41: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-ecv-sv7 Brick42: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-ecv-sv7 Brick43: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-ecv-sv8 Brick44: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-ecv-sv8 Brick45: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-ecv-sv8 Brick46: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-ecv-sv8 Brick47: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-ecv-sv8 Brick48: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-ecv-sv8 Brick49: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-ecv-sv9 Brick50: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-ecv-sv9 Brick51: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-ecv-sv9 Brick52: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-ecv-sv9 Brick53: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-ecv-sv9 Brick54: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-ecv-sv9 Brick55: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-ecv-sv10 Brick56: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-ecv-sv10 Brick57: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-ecv-sv10 Brick58: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-ecv-sv10 Brick59: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-ecv-sv10 Brick60: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-ecv-sv10 Brick61: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-ecv-sv11 Brick62: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-ecv-sv11 Brick63: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-ecv-sv11 Brick64: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-ecv-sv11 Brick65: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-ecv-sv11 Brick66: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-ecv-sv11 Brick67: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-ecv-sv12 Brick68: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-ecv-sv12 Brick69: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-ecv-sv12 Brick70: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-ecv-sv12 Brick71: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-ecv-sv12 Brick72: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-ecv-sv12 Options Reconfigured: disperse.other-eager-lock: off transport.address-family: inet nfs.disable: on cluster.brick-multiplex: enable Volume Name: logvol Type: Replicate Volume ID: dd1a76f5-6d16-40bd-90c1-215cf031c2ae Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick1/logvol Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick1/logvol Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick1/logvol Options Reconfigured: transport.address-family: inet nfs.disable: on performance.client-io-threads: off cluster.brick-multiplex: enable
sosreports and logs @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1716279/
I do see files going into healpending status as below [root@rhs-gp-srv1 ~]# gluster v heal cvlt-ecv info Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-ecv-sv1 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-ecv-sv1 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-ecv-sv1 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-ecv-sv1 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-ecv-sv1 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-ecv-sv1 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-ecv-sv2 /logs/file.8065.7 Status: Connected Number of entries: 1 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-ecv-sv2 /logs/file.8065.7 Status: Connected Number of entries: 1 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-ecv-sv2 /logs/file.8065.7 Status: Connected Number of entries: 1 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-ecv-sv2 /logs/file.8065.7 Status: Connected Number of entries: 1 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-ecv-sv2 /logs/file.8065.7 Status: Connected Number of entries: 1 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-ecv-sv2 /logs/file.8065.7 Status: Connected Number of entries: 1 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-ecv-sv3 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-ecv-sv3 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-ecv-sv3 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-ecv-sv3 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-ecv-sv3 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-ecv-sv3 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-ecv-sv4 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-ecv-sv4 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-ecv-sv4 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-ecv-sv4 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-ecv-sv4 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-ecv-sv4 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-ecv-sv5 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-ecv-sv5 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-ecv-sv5 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-ecv-sv5 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-ecv-sv5 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-ecv-sv5 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-ecv-sv6 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-ecv-sv6 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-ecv-sv6 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-ecv-sv6 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-ecv-sv6 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-ecv-sv6 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-ecv-sv7 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-ecv-sv7 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-ecv-sv7 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-ecv-sv7 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-ecv-sv7 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-ecv-sv7 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-ecv-sv8 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-ecv-sv8 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-ecv-sv8 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-ecv-sv8 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-ecv-sv8 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-ecv-sv8 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-ecv-sv9 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-ecv-sv9 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-ecv-sv9 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-ecv-sv9 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-ecv-sv9 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-ecv-sv9 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-ecv-sv10 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-ecv-sv10 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-ecv-sv10 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-ecv-sv10 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-ecv-sv10 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-ecv-sv10 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-ecv-sv11 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-ecv-sv11 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-ecv-sv11 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-ecv-sv11 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-ecv-sv11 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-ecv-sv11 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-ecv-sv12 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-ecv-sv12 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-ecv-sv12 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-ecv-sv12 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-ecv-sv12 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-ecv-sv12 Status: Connected Number of entries: 0 getfattr details of the above file [root@rhs-gp-srv1 cvlt-ecv]# getfattr -d -m . -e hex /gluster/brick*/cvlt-ecv*/logs/file.8065.7 getfattr: Removing leading '/' from absolute path names # file: gluster/brick4/cvlt-ecv-sv2/logs/file.8065.7 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.config=0x0000080602000200 trusted.ec.dirty=0x00000000000000010000000000000000 trusted.ec.size=0x0000000000000000 trusted.ec.version=0x00000000000000000000000000000000 trusted.gfid=0x26fb9dbd5d1143bbbccb4b1d2bd57f53 trusted.gfid2path.ffca17f547497637=0x33613439336337392d303261332d343666392d393636302d6335353832356230613936622f66696c652e383036352e37 trusted.glusterfs.mdata=0x010000000000000000000000005cf4d6ee000000002f37f4ee000000005cf4d6ee000000002f37f4ee000000005cf4d6eb000000001ed3a73a # file: gluster/brick5/cvlt-ecv-sv2/logs/file.8065.7 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.config=0x0000080602000200 trusted.ec.dirty=0x00000000000000010000000000000000 trusted.ec.size=0x0000000000000000 trusted.ec.version=0x00000000000000000000000000000000 trusted.gfid=0x26fb9dbd5d1143bbbccb4b1d2bd57f53 trusted.gfid2path.ffca17f547497637=0x33613439336337392d303261332d343666392d393636302d6335353832356230613936622f66696c652e383036352e37 trusted.glusterfs.mdata=0x010000000000000000000000005cf4d6ee000000002f37f4ee000000005cf4d6ee000000002f37f4ee000000005cf4d6eb000000001ed3a73a root@rhs-gp-srv2 cvlt-ecv]# getfattr -d -m . -e hex /gluster/brick*/cvlt-ecv*/logs/file.8065.7 getfattr: Removing leading '/' from absolute path names # file: gluster/brick4/cvlt-ecv-sv2/logs/file.8065.7 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.config=0x0000080602000200 trusted.ec.dirty=0x00000000000000010000000000000000 trusted.ec.size=0x0000000000000000 trusted.ec.version=0x00000000000000000000000000000000 trusted.gfid=0x26fb9dbd5d1143bbbccb4b1d2bd57f53 trusted.gfid2path.ffca17f547497637=0x33613439336337392d303261332d343666392d393636302d6335353832356230613936622f66696c652e383036352e37 trusted.glusterfs.mdata=0x010000000000000000000000005cf4d6ee000000002f37f4ee000000005cf4d6ee000000002f37f4ee000000005cf4d6eb000000001ed3a73a # file: gluster/brick5/cvlt-ecv-sv2/logs/file.8065.7 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.config=0x0000080602000200 trusted.ec.dirty=0x00000000000000010000000000000000 trusted.ec.size=0x0000000000000000 trusted.ec.version=0x00000000000000000000000000000000 trusted.gfid=0x26fb9dbd5d1143bbbccb4b1d2bd57f53 trusted.gfid2path.ffca17f547497637=0x33613439336337392d303261332d343666392d393636302d6335353832356230613936622f66696c652e383036352e37 trusted.glusterfs.mdata=0x010000000000000000000000005cf4d6ee000000002f37f4ee000000005cf4d6ee000000002f37f4ee000000005cf4d6eb000000001ed3a73a [root@rhs-gp-srv2 cvlt-ecv]# root@rhs-gp-srv4 cvlt-ecv]# getfattr -d -m . -e hex /gluster/brick*/cvlt-ecv*/logs/file.8065.7 getfattr: Removing leading '/' from absolute path names # file: gluster/brick4/cvlt-ecv-sv2/logs/file.8065.7 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.config=0x0000080602000200 trusted.ec.dirty=0x00000000000000010000000000000000 trusted.ec.size=0x0000000000000000 trusted.ec.version=0x00000000000000000000000000000000 trusted.gfid=0x26fb9dbd5d1143bbbccb4b1d2bd57f53 trusted.gfid2path.ffca17f547497637=0x33613439336337392d303261332d343666392d393636302d6335353832356230613936622f66696c652e383036352e37 trusted.glusterfs.mdata=0x010000000000000000000000005cf4d6ee000000002f37f4ee000000005cf4d6ee000000002f37f4ee000000005cf4d6eb000000001ed3a73a # file: gluster/brick5/cvlt-ecv-sv2/logs/file.8065.7 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.config=0x0000080602000200 trusted.ec.dirty=0x00000000000000010000000000000000 trusted.ec.size=0x0000000000000000 trusted.ec.version=0x00000000000000000000000000000000 trusted.gfid=0x26fb9dbd5d1143bbbccb4b1d2bd57f53 trusted.gfid2path.ffca17f547497637=0x33613439336337392d303261332d343666392d393636302d6335353832356230613936622f66696c652e383036352e37 trusted.glusterfs.mdata=0x010000000000000000000000005cf4d6ee000000002f37f4ee000000005cf4d6ee000000002f37f4ee000000005cf4d6eb000000001ed3a73a
logs and sosreports @ rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1716279/failqa-c14/ failure logs http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1716279/failqa-c14/rhs-gp-srv4.lab.eng.blr.redhat.com/log/
blocks verification of BZ#1716792
for now marking this bug as blocking verification of blocks verification of BZ#1716792 i did trigger heal, and didn't see the issue mentioned in BZ#1716792 But for now keeping BZ#1716792 in same state till this bug is fixed