Bug 1638883
| Summary: | gluster heal problem | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | jszep |
| Component: | replicate | Assignee: | Ravishankar N <ravishankar> |
| Status: | CLOSED WORKSFORME | QA Contact: | Nag Pavan Chilakam <nchilaka> |
| Severity: | urgent | Docs Contact: | |
| Priority: | high | ||
| Version: | mainline | CC: | bugs, jszep, sankarshan, vbellur |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-11-14 04:13:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
jszep
2018-10-12 17:22:36 UTC
1. Can you provide the getfattr output of dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup from all 3 bricks? getfattr -d -m. -e hex /path-to-brick-mount/path-to-dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup 2. Are the files inside dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup identical in all bricks of the replica? jszep, I am changing the 'Product' to glusterfs. I'm assuming you are using the upstream gluster version. If you are a RHGS customer, please reach out to the Red Hat support team to assist you. 1. Can you provide the getfattr output of dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup from all 3 bricks? getfattr -d -m. -e hex /path-to-brick-mount/path-to-dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup We have 3 glutser servers in the cluster: cs-fs1, cs-fs2, and cs-fs3. The directory dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup only exists on cs-fs2. (Note: cs-fs1 was the system that was rebooted that started all this.) [root@cs-fs2 tasks]# getfattr -d -m. -e hex +/mnt/data/vm/brick/1f48f887-dd49-4363-9e5c-603c007a9baf/master/tasks/dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup getfattr: Removing leading '/' from absolute path names # file: mnt/data/vm/brick/1f48f887-dd49-4363-9e5c-603c007a9baf/master/tasks/dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.vm-client-0=0x000000000000000100000001 trusted.gfid=0x1a550e7627b3448cad5818e13fbb8671 trusted.glusterfs.dht=0x000000010000000000000000ffffffff trusted.glusterfs.dht.mds=0x00000000 In addition, there is a directory: /mnt/data/vm/brick/1f48f887-dd49-4363-9e5c-603c007a9baf/master/tasks/dc8b1e1e-f7d3-4199-aa84-2e809cc78a33/ (no .backup) that DOES exist on all three servers with identical contents: [root@cs-fs2 tasks]# ls /mnt/data/vm/brick/1f48f887-dd49-4363-9e5c-603c007a9baf/master/tasks/dc8b1e1e-f7d3-4199-aa84-2e809cc78a33/ dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.job.0 dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.result dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.recover.0 dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.task 2. Are the files inside dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup identical in all bricks of the replica? No. dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup only exits on cs-fs2. It's contents are: [root@cs-fs2 tasks]# ls dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.recover.0 The contents of the file dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.recover.0 are: cs-fs2: cat dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.recover.0 function = create_image_rollback moduleName = sd params = /rhev/data-center/mnt/glusterSD/cs-fs1.bu.edu:_vm/1f48f887-dd49-4363-9e5c-603c007a9baf/images/34fc74a4-2665-44e8-b66d-455da248e209 name = create image rollback: 34fc74a4-2665-44e8-b66d-455da248e209 object = StorageDomain Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions. Hi jszep,
> No. dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup only exits on cs-fs2.
1.Is the setup still in the same state now? Can you also provide the getfattr output of the parent directory (mnt/data/vm/brick/1f48f887-dd49-4363-9e5c-603c007a9baf/master/tasks) on all 3 bricks?
If you explicitly do a stat from the fuse mount point to the path (/1f48f887-dd49-4363-9e5c-603c007a9baf/master/tasks/dc8b1e1e-f7d3-4199-aa84-2e809cc78a33.backup), the directory should get created on the other 2 bricks as well.
2. Could you provide the gluster volume info output?
If you have some sort of a reproducer, that would help in identifying the issue.
Huh - I answered this request over a week ago but it did not show up here. Anyway, the problem is solved. I have updated and rebooted the other file servers and everything is running as expected. Thank you for your assustance. You can close this case. |