Bug 1440051
Summary: | Application VMs with their disk images on sharded-replica 3 volume are unable to boot after performing rebalance | |||
---|---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Krutika Dhananjay <kdhananj> | |
Component: | distribute | Assignee: | Krutika Dhananjay <kdhananj> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
Severity: | high | Docs Contact: | ||
Priority: | unspecified | |||
Version: | mainline | CC: | amukherj, bugs, kdhananj, knarra, rcyriac, rgowdapp, rhinduja, rhs-bugs, sasundar, storage-qa-internal, vnosov | |
Target Milestone: | --- | Keywords: | Triaged | |
Target Release: | --- | |||
Hardware: | x86_64 | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.11.0 | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | 1439753 | |||
: | 1440635 1440637 (view as bug list) | Environment: | ||
Last Closed: | 2017-05-30 18:49:22 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1440635, 1440637 |
Description
Krutika Dhananjay
2017-04-07 08:36:08 UTC
REVIEW: https://review.gluster.org/17010 (features/shard: Fix vm corruption upon fix-layout) posted (#1) for review on master by Krutika Dhananjay (kdhananj) COMMIT: https://review.gluster.org/17010 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 99c8c0b03a3368d81756440ab48091e1f2430a5f Author: Krutika Dhananjay <kdhananj> Date: Thu Apr 6 18:10:41 2017 +0530 features/shard: Fix vm corruption upon fix-layout shard's writev implementation, as part of identifying presence of participant shards that aren't in memory, first sends an MKNOD on these shards, and upon EEXIST error, looks up the shards before proceeding with the writes. The VM corruption was caused when the following happened: 1. DHT had n subvolumes initially. 2. Upon add-brick + fix-layout, the layout of .shard changed although the existing shards under it were yet to be migrated to their new hashed subvolumes. 3. During this time, there were writes on the VM falling in regions of the file whose corresponding shards were already existing under .shard. 4. Sharding xl sent MKNOD on these shards, now creating them in their new hashed subvolumes although there already exist shard blocks for this region with valid data. 5. All subsequent writes were wound on these newly created copies. The net outcome is that both copies of the shard didn't have the correct data. This caused the affected VMs to be unbootable. FIX: For want of better alternatives in DHT, the fix changes shard fops to do a LOOKUP before the MKNOD and upon EEXIST error, perform another lookup. Change-Id: I8a2e97d91ba3275fbc7174a008c7234fa5295d36 BUG: 1440051 RCA'd-by: Raghavendra Gowdappa <rgowdapp> Reported-by: Mahdi Adnan <mahdi.adnan> Signed-off-by: Krutika Dhananjay <kdhananj> Reviewed-on: https://review.gluster.org/17010 Smoke: Gluster Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> REVIEW: https://review.gluster.org/17014 (features/shard: Initialize local->fop in readv) posted (#1) for review on master by Krutika Dhananjay (kdhananj) REVIEW: https://review.gluster.org/17014 (features/shard: Initialize local->fop in readv) posted (#2) for review on master by Krutika Dhananjay (kdhananj) COMMIT: https://review.gluster.org/17014 committed in master by Vijay Bellur (vbellur) ------ commit 594a7c6a187cf780bd666e7343c39a2d92fc67ef Author: Krutika Dhananjay <kdhananj> Date: Mon Apr 10 11:04:31 2017 +0530 features/shard: Initialize local->fop in readv Change-Id: I9008ca9960df4821636501ae84f93a68f370c67f BUG: 1440051 Signed-off-by: Krutika Dhananjay <kdhananj> Reviewed-on: https://review.gluster.org/17014 NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu> CentOS-regression: Gluster Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Vijay Bellur <vbellur> REVIEW: https://review.gluster.org/17085 (cluster/dht: Pass the req dict instead of NULL in dht_attr2()) posted (#1) for review on master by Krutika Dhananjay (kdhananj) REVIEW: https://review.gluster.org/17086 (mount/fuse: Replace GF_LOG_OCCASIONALLY with gf_log() to report fop failure at all times) posted (#1) for review on master by Krutika Dhananjay (kdhananj) REVIEW: https://review.gluster.org/17087 (cluster/dht: Do not sync xattrs between src and dst twice during rebalance) posted (#1) for review on master by Krutika Dhananjay (kdhananj) REVIEW: https://review.gluster.org/17086 (mount/fuse: Replace GF_LOG_OCCASIONALLY with gf_log() to report fop failure at all times) posted (#2) for review on master by Krutika Dhananjay (kdhananj) REVIEW: https://review.gluster.org/17085 (cluster/dht: Pass the req dict instead of NULL in dht_attr2()) posted (#2) for review on master by Krutika Dhananjay (kdhananj) COMMIT: https://review.gluster.org/17085 committed in master by Raghavendra G (rgowdapp) ------ commit d60ca8e96bbc16b13f8f3456f30ebeb16d0d1e47 Author: Krutika Dhananjay <kdhananj> Date: Thu Apr 20 10:08:02 2017 +0530 cluster/dht: Pass the req dict instead of NULL in dht_attr2() This bug was causing VMs to pause during rebalance. When qemu winds down a STAT, shard fills the trusted.glusterfs.shard.file-size attribute in the req dict which DHT doesn't wind its STAT fop with upon detecting the file has undergone migration. As a result shard doesn't find the value to this key in the unwind path, causing it to fail the STAT with EINVAL. Also, the same bug exists in other fops too, which is also fixed in this patch. Change-Id: Id7823fd932b4e5a9b8779ebb2b612a399c0ef5f0 BUG: 1440051 Signed-off-by: Krutika Dhananjay <kdhananj> Reviewed-on: https://review.gluster.org/17085 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp> REVIEW: https://review.gluster.org/17126 (cluster/dht: Pass the correct xdata in fremovexattr fop) posted (#1) for review on master by Krutika Dhananjay (kdhananj) COMMIT: https://review.gluster.org/17126 committed in master by Raghavendra G (rgowdapp) ------ commit ab88f655e6423f51e2f2fac9265ff4d4f5c3e579 Author: Krutika Dhananjay <kdhananj> Date: Thu Apr 27 11:53:24 2017 +0530 cluster/dht: Pass the correct xdata in fremovexattr fop Change-Id: Id84bc87e48f435573eba3b24d3fb3c411fd2445d BUG: 1440051 Signed-off-by: Krutika Dhananjay <kdhananj> Reviewed-on: https://review.gluster.org/17126 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp> COMMIT: https://review.gluster.org/17086 committed in master by Raghavendra G (rgowdapp) ------ commit ef60a29703f520c5bd06467efc4a0d0a33552a06 Author: Krutika Dhananjay <kdhananj> Date: Thu Apr 20 10:17:07 2017 +0530 mount/fuse: Replace GF_LOG_OCCASIONALLY with gf_log() to report fop failure at all times Change-Id: Ibd8e1c6172812951092ff6097ba4bed943051b7c BUG: 1440051 Signed-off-by: Krutika Dhananjay <kdhananj> Reviewed-on: https://review.gluster.org/17086 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra Bhat <raghavendra> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.0, please open a new bug report. glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html [2] https://www.gluster.org/pipermail/gluster-users/ |