Bug 1757307
| Summary: | heal not completing possibly due to a an extra entry on sink(probably due to a rename) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> | ||||||
| Component: | disperse | Assignee: | Sheetal Pamecha <spamecha> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Nag Pavan Chilakam <nchilaka> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | rhgs-3.5 | CC: | aspandey, puebele, rhs-bugs, spamecha, storage-qa-internal | ||||||
| Target Milestone: | --- | Keywords: | Reopened | ||||||
| Target Release: | --- | Flags: | aspandey:
needinfo-
|
||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2020-09-02 05:16:10 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | 1596165, 1765114, 1787538 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1621358 [details]
screenshot shows one file is extra in sink bricks
Created attachment 1621359 [details]
the extra file on sink is a t file
upstream patch - https://review.gluster.org/#/c/glusterfs/+/23937/ *** This bug has been marked as a duplicate of bug 1596165 *** |
Description of problem: ======================= I see that even after a day heal is not completing for one particular entry. Hit the issue when post upgrading first of the three nodes, from 6.0.14 to 6.0.15. Version-Release number of selected component (if applicable): ============ 6.0.14 to 6.0.15 upgrade How reproducible: =============== hit it once Steps to Reproduce: ================== 1) 3 node cluster with bmux enabled 2) one 1x3 volume and one 20x(4+2) ecvol(see vol info details) 3) triggered below IOs, on total of 5 clients: a) linux untar from 4 clients on ec-volume b) top o/p of clients appending to files on afrvolume c) crefi on 2 clients on ec-volume: for j in {1..100};do for i in {create,chmod,chown,chgrp,symlink,truncate,rename,hardlink}; do python crefi.py --multi -n 5 -b 20 -d 10 --max=1K --min=50 --random -T 2 -t text --fop=$i /mnt/cvlt-async/IOs/crefi/$HOSTNAME/ ; sleep 10 ; done;done d) lookups while true;find *|xargs stat from all clients on ec-vol 4) after about 2 days, triggered inservice upgrade from 6.0.14 to 6.0.15 on first node. 5) heal info is showing below entry as pending for heal even after 1 day and hence unable to upgrade the whole cluster completely note: bricks were down as part of upgrade for hardly less than 1hr at max Actual results: ================ heal info is showing below entry as pending for heal even after 1 day and hence unable to upgrade the whole cluster completely [root@rhs-gp-srv1 ~]# gluster v heal cvlt-async info Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16 /IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 Status: Connected Number of entries: 1 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16 /IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 Status: Connected Number of entries: 1 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16 /IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 Status: Connected Number of entries: 1 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16 /IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 Status: Connected Number of entries: 1 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20 Status: Connected Number of entries: 0 Brick rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20 Status: Connected Number of entries: 0 Brick rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20 Status: Connected Number of entries: 0 Brick rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20 Status: Connected Number of entries: 0 Expected results: ================ heal should complete Additional info: =============== volinfo Volume Name: cvlt-async Type: Distributed-Disperse Volume ID: 7cba899d-5642-44b9-a05b-57d78247d5ea Status: Started Snapshot Count: 0 Number of Bricks: 20 x (4 + 2) = 120 Transport-type: tcp Bricks: Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1 Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1 Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick1/cvlt-async-sv-1 Brick4: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1 Brick5: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1 Brick6: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick2/cvlt-async-sv-1 Brick7: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2 Brick8: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2 Brick9: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick3/cvlt-async-sv-2 Brick10: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2 Brick11: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2 Brick12: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick4/cvlt-async-sv-2 Brick13: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3 Brick14: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3 Brick15: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick5/cvlt-async-sv-3 Brick16: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3 Brick17: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3 Brick18: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick6/cvlt-async-sv-3 Brick19: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4 Brick20: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4 Brick21: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick7/cvlt-async-sv-4 Brick22: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4 Brick23: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4 Brick24: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick8/cvlt-async-sv-4 Brick25: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5 Brick26: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5 Brick27: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick9/cvlt-async-sv-5 Brick28: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5 Brick29: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5 Brick30: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick10/cvlt-async-sv-5 Brick31: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6 Brick32: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6 Brick33: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick11/cvlt-async-sv-6 Brick34: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6 Brick35: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6 Brick36: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick12/cvlt-async-sv-6 Brick37: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7 Brick38: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7 Brick39: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick13/cvlt-async-sv-7 Brick40: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7 Brick41: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7 Brick42: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick14/cvlt-async-sv-7 Brick43: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8 Brick44: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8 Brick45: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick15/cvlt-async-sv-8 Brick46: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8 Brick47: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8 Brick48: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick16/cvlt-async-sv-8 Brick49: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9 Brick50: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9 Brick51: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick17/cvlt-async-sv-9 Brick52: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9 Brick53: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9 Brick54: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick18/cvlt-async-sv-9 Brick55: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10 Brick56: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10 Brick57: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick19/cvlt-async-sv-10 Brick58: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10 Brick59: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10 Brick60: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick20/cvlt-async-sv-10 Brick61: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11 Brick62: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11 Brick63: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick21/cvlt-async-sv-11 Brick64: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11 Brick65: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11 Brick66: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick22/cvlt-async-sv-11 Brick67: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12 Brick68: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12 Brick69: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick23/cvlt-async-sv-12 Brick70: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12 Brick71: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12 Brick72: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick24/cvlt-async-sv-12 Brick73: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13 Brick74: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13 Brick75: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick25/cvlt-async-sv-13 Brick76: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13 Brick77: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13 Brick78: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick26/cvlt-async-sv-13 Brick79: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14 Brick80: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14 Brick81: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick27/cvlt-async-sv-14 Brick82: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14 Brick83: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14 Brick84: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick28/cvlt-async-sv-14 Brick85: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15 Brick86: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15 Brick87: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick29/cvlt-async-sv-15 Brick88: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15 Brick89: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15 Brick90: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick30/cvlt-async-sv-15 Brick91: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16 Brick92: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16 Brick93: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick31/cvlt-async-sv-16 Brick94: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16 Brick95: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16 Brick96: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick32/cvlt-async-sv-16 Brick97: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17 Brick98: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17 Brick99: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick33/cvlt-async-sv-17 Brick100: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17 Brick101: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17 Brick102: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick34/cvlt-async-sv-17 Brick103: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18 Brick104: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18 Brick105: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick35/cvlt-async-sv-18 Brick106: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18 Brick107: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18 Brick108: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick36/cvlt-async-sv-18 Brick109: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19 Brick110: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19 Brick111: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick37/cvlt-async-sv-19 Brick112: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19 Brick113: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19 Brick114: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick38/cvlt-async-sv-19 Brick115: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20 Brick116: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20 Brick117: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick39/cvlt-async-sv-20 Brick118: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20 Brick119: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20 Brick120: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick40/cvlt-async-sv-20 Options Reconfigured: features.ctime: enable cluster.disperse-self-heal-daemon: enable disperse.shd-max-threads: 24 client.event-threads: 8 server.event-threads: 8 transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on cluster.brick-multiplex: enable Volume Name: repvol Type: Replicate Volume ID: 5566101e-5321-4cdf-b560-7804b7addd58 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: rhs-gp-srv1.lab.eng.blr.redhat.com:/gluster/brick44/repvol Brick2: rhs-gp-srv2.lab.eng.blr.redhat.com:/gluster/brick44/repvol Brick3: rhs-gp-srv4.lab.eng.blr.redhat.com:/gluster/brick44/repvol Options Reconfigured: cluster.self-heal-daemon: enable transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: off cluster.brick-multiplex: enable getfattr details of pending heal: ================================ good bricks of n1: ------------------ [root@rhs-gp-srv1 glusterfs]# getfattr -d -m . -e hex /gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 getfattr: Removing leading '/' from absolute path names # file: gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.dirty=0x00000000000000050000000000000000 trusted.ec.version=0x00000000000000a100000000000000a8 trusted.gfid=0x991669123a0d486ba59284fe9e6784d5 trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48 [root@rhs-gp-srv1 glusterfs]# getfattr -d -m . -e hex /gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 getfattr: Removing leading '/' from absolute path names # file: gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.dirty=0x00000000000000050000000000000000 trusted.ec.version=0x00000000000000a100000000000000a8 trusted.gfid=0x991669123a0d486ba59284fe9e6784d5 trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48 good bricks of n2: ================ [root@rhs-gp-srv2 glusterfs]# getfattr -d -m . -e hex /gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 getfattr: Removing leading '/' from absolute path names # file: gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.dirty=0x00000000000000050000000000000000 trusted.ec.version=0x00000000000000a100000000000000a8 trusted.gfid=0x991669123a0d486ba59284fe9e6784d5 trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48 [root@rhs-gp-srv2 glusterfs]# getfattr -d -m . -e hex /gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 getfattr: Removing leading '/' from absolute path names # file: gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.dirty=0x00000000000000050000000000000000 trusted.ec.version=0x00000000000000a100000000000000a8 trusted.gfid=0x991669123a0d486ba59284fe9e6784d5 trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48 bricks of node n3(which was upgraded): ------------------------------ [root@rhs-gp-srv4 glusterfs]# getfattr -d -m . -e hex /gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 getfattr: Removing leading '/' from absolute path names # file: gluster/brick32/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.version=0x000000000000002500000000000000a8 trusted.gfid=0x991669123a0d486ba59284fe9e6784d5 trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48 [root@rhs-gp-srv4 glusterfs]# getfattr -d -m . -e hex /gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 getfattr: Removing leading '/' from absolute path names # file: gluster/brick31/cvlt-async-sv-16/IOs/crefi/rhs-client45.lab.eng.blr.redhat.com/thread0/level018/level118/level218/level318/level418/level518 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.ec.version=0x000000000000002500000000000000a8 trusted.gfid=0x991669123a0d486ba59284fe9e6784d5 trusted.glusterfs.dht=0x0000000000000000f3333324ffffffff trusted.glusterfs.mdata=0x010000000000000000000000005d92a15300000000298a60a1000000005d9241a1000000002cb0ea11000000005d91b286000000002eb3db48