Bug 1818836
| Summary: | [RHEL8.1] After performing add-brick and rebalance start, Seeing Input/Output errors in the rebalance logs | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Bala Konda Reddy M <bmekala> |
| Component: | rpc | Assignee: | Mohit Agrawal <moagrawa> |
| Status: | CLOSED NOTABUG | QA Contact: | Rahul Hinduja <rhinduja> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rhgs-3.5 | CC: | jahernan, moagrawa, rhs-bugs, spalai, storage-qa-internal |
| Target Milestone: | --- | Keywords: | Reopened, ZStream |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-04-21 14:01:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: Seen 5 failures in the rebalance status, after performing add-brick and rebalance start. [root@dhcp37-200 ~]# gluster vol rebalance ec-vol status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- dhcp37-157.lab.eng.blr.redhat.com 695355 14.1GB 3639986 2 162187 completed 158:54:11 localhost 696197 15.8GB 3640039 3 161265 completed 158:54:11 volume rebalance: ec-vol: success ---------------------8<------------------ REBALANCE LOG SNIPPET [2020-03-23 11:35:13.348285] W [MSGID: 122040] [ec-common.c:1232:ec_prepare_update_cbk] 0-ec-vol-disperse-3: Failed to get size and version : FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-23 11:35:13.348464] E [MSGID: 109006] [dht-selfheal.c:655:dht_selfheal_dir_xattr_cbk] 0-ec-vol-dht: layout setxattr failed on ec-vol-disperse-3, path:/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms gfid:a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-23 11:35:13.348583] E [MSGID: 122034] [ec-common.c:706:ec_child_select] 0-ec-vol-disperse-3: Insufficient available children for this request (have 0, need 4). FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.348638] E [MSGID: 122037] [ec-common.c:2320:ec_update_size_version_done] 0-ec-vol-disperse-3: Failed to update version and size. FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-23 11:35:13.360383] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-0: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.360806] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-1: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.361159] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-2: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.361511] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-3: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-23 11:35:13.361935] I [dht-rebalance.c:3297:gf_defrag_process_dir] 0-ec-vol-dht: migrate data called on /IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms --------------------------8<-------------------------- set the rebal-throttle to aggressive and started the rebalance again. gluster vol rebala ec-vol status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- dhcp37-200.lab.eng.blr.redhat.com 648867 14.9GB 4710033 2 320869 completed 57:31:30 localhost 14570 169.8MB 4724607 2 147617 completed 22:31:53 volume rebalance: ec-vol: success [root@dhcp37-157 ~]# [2020-03-28 12:06:50.873631] W [MSGID: 122040] [ec-common.c:1232:ec_prepare_update_cbk] 0-ec-vol-disperse-3: Failed to get size and version : FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-28 12:06:50.873733] E [MSGID: 109006] [dht-selfheal.c:655:dht_selfheal_dir_xattr_cbk] 0-ec-vol-dht: layout setxattr failed on ec-vol-disperse-3, path:/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms gfid:a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-28 12:06:50.873803] E [MSGID: 122034] [ec-common.c:706:ec_child_select] 0-ec-vol-disperse-3: Insufficient available children for this request (have 0, need 4). FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-28 12:06:50.873841] E [MSGID: 122037] [ec-common.c:2320:ec_update_size_version_done] 0-ec-vol-disperse-3: Failed to update version and size. FOP : 'XATTROP' failed on '/IOs/kernel/dhcp42-76.lab.eng.blr.redhat.com/dir.3/linux-5.3.2/arch/powerpc/platforms' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [Input/output error] [2020-03-28 12:06:50.910924] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-0: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-28 12:06:50.911362] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-1: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-28 12:06:50.911761] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-2: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 [2020-03-28 12:06:50.912117] W [MSGID: 122035] [ec-common.c:668:ec_child_select] 0-ec-vol-disperse-3: Executing operation with some subvolumes unavailable. (24). FOP : 'INODELK' failed on '(null)' with gfid a06abdf4-0531-4202-b718-c1a3fe42da96 --------------------------8<--------------------------- Version-Release number of selected component (if applicable): How reproducible: 2/2 Steps to Reproduce: 1. On a three node cluster, enabled brick mux. 2. Created a distributed-disperse volume(3X(4+2)) and two replicate volumes(1X3) 3. Mounted on 10 clients and ran untar, crefi, performed du -sh, ll -ltrh 4. After soaking the cluster for 5 days. 5. Performed add-brick and started rebalance 6. After a day of rebalance, a node went down. due to disk issue can't bring back the node online. 7. Rebalance got completed after 4 days with 5 failures. 8. Set the "rebal-throttle to aggressive" and with no IO from the clients 9. started rebalance again and rebalance got completed and seen 4 failures with similar errors in the rebalance logs Actual results: Seen failures in the rebalance status and Input/Output errors in the rebalance logs. Expected results: Should not see any failures rebalance should be successful Additional info: gluster vol info ec-vol Volume Name: ec-vol Type: Distributed-Disperse Volume ID: 0f40b39e-466e-497b-919c-354d4e78fad6 Status: Started Snapshot Count: 0 Number of Bricks: 4 x (4 + 2) = 24 Transport-type: tcp Bricks: Brick1: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick2/ec-vol2 Brick2: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick2/ec-vol2 Brick3: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick2/ec-vol2 Brick4: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick3/ec-vol3 Brick5: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick3/ec-vol3 Brick6: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick3/ec-vol3 Brick7: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick4/ec-vol4 Brick8: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick4/ec-vol4 Brick9: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick4/ec-vol4 Brick10: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick5/ec-vol5 Brick11: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick5/ec-vol5 Brick12: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick5/ec-vol5 Brick13: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick6/ec-vol6 Brick14: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick6/ec-vol6 Brick15: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick6/ec-vol6 Brick16: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick7/ec-vol7 Brick17: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick7/ec-vol7 Brick18: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick7/ec-vol7 Brick19: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick8/ec-vol8 Brick20: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick8/ec-vol8 Brick21: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick8/ec-vol8 Brick22: dhcp37-200.lab.eng.blr.redhat.com:/bricks/brick9/ec-vol9 Brick23: dhcp37-157.lab.eng.blr.redhat.com:/bricks/brick9/ec-vol9 Brick24: dhcp37-114.lab.eng.blr.redhat.com:/bricks/brick9/ec-vol9 Options Reconfigured: cluster.rebal-throttle: aggressive disperse.shd-max-threads: 24 client.event-threads: 8 server.event-threads: 8 transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on cluster.brick-multiplex: enable -----------------------8<-------------------------