Bug 1335156
| Summary: | Ext4 FS corruption in VM hosted on sharded volume | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Bhaskarakiran <byarlaga> | ||||
| Component: | sharding | Assignee: | Krutika Dhananjay <kdhananj> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Bhaskarakiran <byarlaga> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | rhgs-3.1 | CC: | amukherj, byarlaga, mzywusko, pkarampu, rcyriac, rhinduja, sabose | ||||
| Target Milestone: | --- | Keywords: | ZStream | ||||
| Target Release: | RHGS 3.1.3 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | glusterfs-3.7.9-6 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-06-29 14:09:45 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1258386, 1311817 | ||||||
| Attachments: |
|
||||||
|
Description
Bhaskarakiran
2016-05-11 12:57:20 UTC
Input/output error would most likely be due to a split-brain. Did you check for that? If the issue is split-brain, then the bug is not in sharding. -Krutika My bad for delaying this. I have not seen any split-brain during this corruption. Could you please attach the sosreports? The same VM is not there now due to setup issues. I will have to re-create it. Will provide the sosreports once it is done. Created attachment 1160452 [details]
sosreport
Hi Bhaskar, I checked the attachment. There are no directories '/var/log/glusterfs' or '/var/lib/glusterd' in the sosreport. Could you attach the correct sosreports? -Krutika all the sosreports are copied to http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1335156/ I checked your setup. For the disks with uuid badff993-fe76-4147-97b7-dfd53293907f and 7ff05ee6-b76d-46cd-a98a-17569e2bd318, there are corresponding directories under images/ in the volume 'vmstore' each of which contain another uuid file, a .lease and a .meta for this uuid file. For 7ff05ee6-b76d-46cd-a98a-17569e2bd318, turns out there were two such files, with two .meta and .lease files. Is this normal? Also, for one of those files - namely 8bd7bc1d-8ed3-49e4-92ff-72cd009cd6a6 - the file was empty and all its shards were also 0 bytes. In essence, the file contains no data at all. Not sure if this is normal either. Still needs investigation. -Krutika One more thing: I found the files associataed with th uuid for the corrupt data disks given by Bhaskarakiran on the volume vmstore as opposed to data. Is this okay? Or is that incorrect? The primary disk i.e. OS disk is on data while the IO is run on the disk that is attached from vmstore. The corruption is seen on the disk that comes from vmstore. https://code.engineering.redhat.com/gerrit/74760 which fixes 1330044 is the patch which fixes the I/O errors even when there were no files in split-brain Pranith Tested on 3.7.9-6 build and didn' see the corruption. Marking this as fixed for now. Will re-open if its seen again. |