Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: Seen 5 failures in the rebalance status, after performing add-brick and rebalance start. [root@dhcp37-200 ~]# gluster vol rebalance ec-vol status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- dhcp37-157.lab.eng.blr.redhat.com 695355 14.1GB 3639986 2 162187 completed 158:54:11 localhost 696197 15.8GB 3640039 3 161265 completed 158:54:11 volume rebalance: ec-vol: success ---------------------8<------------------ REBALANCE LOG SNIPPET [2020-03-23 11:35:13.348285] W [MSGID: 122040] [ec-common.c:1232:ec_prepare_update_cbk] 0-ec-vol-disperse-3: Failed to get size and version : FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-23 11:35:13.348464] E [MSGID: 109006] [dht-selfheal.c:655:dht_selfheal_dir_xattr_cbk] 0-ec-vol-dht: layout setxattr failed on ec-vol-disperse-3, path:/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms gfid:a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-23 11:35:13.348583] E [MSGID: 122034] [ec-common.c:706:ec_child_select] 0-ec-vol-disperse-3: Insufficient available children for this request (have 0, need 4). FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.348638] E [MSGID: 122037] [ec-common.c:2320:ec_update_size_version_done] 0-ec-vol-disperse-3: Failed to update version and size. FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-23 11:35:13.360383] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-0: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.360806] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-1: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.361159] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-2: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.361511] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-3: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.361935] I [dht-rebalance.c:3297:gf_defrag_process_dir] 0-ec-vol-dht: migrate data called on /IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms --------------------------8<-------------------------- set the rebal-throttle to aggressive and started the rebalance again. gluster vol rebala ec-vol status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- dhcp37-200.lab.eng.blr.redhat.com 648867 14.9GB 4710033 2 320869 completed 57:31:30 localhost 14570 169.8MB 4724607 2 147617 completed 22:31:53 volume rebalance: ec-vol: success [root@dhcp37-157 ~]# [2020-03-28 12:06:50.873631] W [MSGID: 122040] [ec-common.c:1232:ec_prepare_update_cbk] 0-ec-vol-disperse-3: Failed to get size and version : FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-28 12:06:50.873733] E [MSGID: 109006] [dht-selfheal.c:655:dht_selfheal_dir_xattr_cbk] 0-ec-vol-dht: layout setxattr failed on ec-vol-disperse-3, path:/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms gfid:a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-28 12:06:50.873803] E [MSGID: 122034] [ec-common.c:706:ec_child_select] 0-ec-vol-disperse-3: Insufficient available children for this request (have 0, need 4). FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-28 12:06:50.873841] E [MSGID: 122037] [ec-common.c:2320:ec_update_size_version_done] 0-ec-vol-disperse-3: Failed to update version and size. FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-28 12:06:50.910924] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-0: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-28 12:06:50.911362] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-1: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-28 12:06:50.911761] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-2: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-28 12:06:50.912117] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-3: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 --------------------------8<--------------------------- Version-Release number of selected component (if applicable): How reproducible: 2/2 Steps to Reproduce: 1. On a three node cluster, enabled brick mux. 2. Created a distributed-disperse volume(3X(4+2)) and two replicate volumes(1X3) 3. Mounted on 10 clients and ran untar, crefi, performed du -sh, ll -ltrh 4. After soaking the cluster for 5 days. 5. Performed add-brick and started rebalance 6. After a day of rebalance, a node went down. due to disk issue can't bring back the node online. 7. Rebalance got completed after 4 days with 5 failures. 8. Set the "rebal-throttle to aggressive" and with no IO from the clients 9. started rebalance again and rebalance got completed and seen 4 failures with similar errors in the rebalance logs Actual results: Seen failures in the rebalance status and Input/Output errors in the rebalance logs. Expected results: Should not see any failures rebalance should be successful Additional info: gluster vol info ec-vol Volume Name: ec-vol Type: Distributed-Disperse Volume ID: 0f40b39e-466e-497b-919c-354d4e78fad6 Status: Started Snapshot Count: 0 Number of Bricks: 4 x (4 + 2) = 24 Transport-type: tcp Bricks: Brick1: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick2/ec-vol2 Brick2: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick2/ec-vol2 Brick3: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick2/ec-vol2 Brick4: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick3/ec-vol3 Brick5: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick3/ec-vol3 Brick6: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick3/ec-vol3 Brick7: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick4/ec-vol4 Brick8: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick4/ec-vol4 Brick9: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick4/ec-vol4 Brick10: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick5/ec-vol5 Brick11: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick5/ec-vol5 Brick12: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick5/ec-vol5 Brick13: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick6/ec-vol6 Brick14: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick6/ec-vol6 Brick15: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick6/ec-vol6 Brick16: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick7/ec-vol7 Brick17: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick7/ec-vol7 Brick18: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick7/ec-vol7 Brick19: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick8/ec-vol8 Brick20: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick8/ec-vol8 Brick21: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick8/ec-vol8 Brick22: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick9/ec-vol9 Brick23: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick9/ec-vol9 Brick24: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick9/ec-vol9 Options Reconfigured: cluster.rebal-throttle: aggressive disperse.shd-max-threads: 24 client.event-threads: 8 server.event-threads: 8 transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on cluster.brick-multiplex: enable -----------------------8<-------------------------
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days