If a guest which has disk on gluster volume , is paused due to storage error & does not resume when the storage comes up .These guests cannot be resumed manually also. The disks in the storage domain are readable & writable (touch). Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: vm is paused & does not resume automatically or manually once the storage is up. Expected results: VM auto unpauses once the storage domain comes up.
REVIEW: http://review.gluster.org/12593 (performance/write-behind: retry "failed syncs to backend") posted (#1) for review on release-3.7 by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#1) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#2) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#3) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#4) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#5) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#6) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#7) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#8) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#9) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#10) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#11) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#12) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#13) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#14) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#15) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#16) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#17) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#18) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#19) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#20) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#21) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#22) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#23) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#24) for review on master by Vijay Bellur (vbellur)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#25) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#26) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#27) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#28) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#29) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#30) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#31) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/12594 (performance/write-behind: retry "failed syncs to backend") posted (#32) for review on master by Raghavendra G (rgowdapp)
COMMIT: http://review.gluster.org/12594 committed in master by Raghavendra G (rgowdapp) ------ commit 3fcead2de7bcdb4e1312f37e7e750abd8d9d9770 Author: Raghavendra G <rgowdapp> Date: Tue Nov 17 12:57:54 2015 +0530 performance/write-behind: retry "failed syncs to backend" 1. When sync fails, the cached-write is still preserved unless there is a flush/fsync waiting on it. 2. When a sync fails and there is a flush/fsync waiting on the cached-write, the cache is thrown away and no further retries will be made. In other words flush/fsync act as barriers for all the previous writes. The behaviour of fsync acting as a barrier is controlled by an option (see below for details). All previous writes are either successfully synced to backend or forgotten in case of an error. Without such barrier fop (especially flush which is issued prior to a close), we end up retrying for ever even after fd is closed. 3. If a fop is waiting on cached-write and syncing to backend fails, the waiting fop is failed. 4. sync failures when no fop is waiting are ignored and are not propagated to application. For eg., a. first attempt of sync of a cached-write w1 fails b. second attempt of sync of w1 succeeds If there are no fops dependent on w1 are issued b/w a and b, application won't know about failure encountered in a. 5. The effect of repeated sync failures is that, there will be no cache for future writes and they cannot be written behind. fsync as a barrier and resync of cached writes post fsync failure: ================================================================== Whether to keep retrying failed syncs post fsync is controlled by an option "resync-failed-syncs-after-fsync". By default, this option is set to "off". If sync of "cached-writes issued before fsync" (to backend) fails, this option configures whether to retry syncing them after fsync or forget them. If set to on, cached-writes are retried till a "flush" fop (or a successful sync) on sync failures. fsync itself is failed irrespective of the value of this option, when there is a sync failure of any cached-writes issued before fsync. Change-Id: I6097c9257bfb9ee5b15616fbe6a0576ae9af369a Signed-off-by: Raghavendra G <rgowdapp> BUG: 1279730 Reviewed-on: http://review.gluster.org/12594
REVIEW: http://review.gluster.org/13113 (wb: Remove inline keyword) posted (#1) for review on master by Raghavendra Talur (rtalur)
REVIEW: http://review.gluster.org/13117 (glusterd: GD_OP_VERSION should not be a released one) posted (#1) for review on master by Raghavendra Talur (rtalur)
REVIEW: http://review.gluster.org/13117 (glusterd: use yet to be released GD_OP_VERSION) posted (#2) for review on master by Raghavendra Talur (rtalur)
REVIEW: http://review.gluster.org/13113 (wb: remove inline keyword) posted (#2) for review on master by Raghavendra Talur (rtalur)
COMMIT: http://review.gluster.org/13113 committed in master by Raghavendra Talur (rtalur) ------ commit 96f4ec28a80c013b71aa723efaa5810d2eacdd7f Author: Raghavendra Talur <rtalur> Date: Wed Dec 30 13:23:33 2015 +0530 wb: remove inline keyword When compiled with -Werror flag gcc throws the following error: ‘iov_length’ is static but used in inline function ‘__wb_modify_write_request’ which is not static. Let gcc decide what functions to inline and remove the inline keyword. Change-Id: I6d832596eefcf08306634936e11d2c8d4b8f9ccd BUG: 1279730 Signed-off-by: Raghavendra Talur <rtalur> Reviewed-on: http://review.gluster.org/13113
REVIEW: http://review.gluster.org/13117 (glusterd: use yet to be released GD_OP_VERSION) posted (#3) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/13117 (glusterd: GD_OP_VERSION should not be a released one) posted (#4) for review on master by Raghavendra Talur (rtalur)
REVIEW: http://review.gluster.org/13117 (glusterd: GD_OP_VERSION should not be a released one) posted (#5) for review on master by Raghavendra Talur (rtalur)
REVIEW: http://review.gluster.org/13177 (performance/write-behind: maintain correct transit size in case of short writes.) posted (#1) for review on master by Raghavendra G (rgowdapp)
REVIEW: http://review.gluster.org/13117 (glusterd: GD_OP_VERSION should not be a released one) posted (#6) for review on master by Raghavendra Talur (rtalur)
COMMIT: http://review.gluster.org/13177 committed in master by Raghavendra G (rgowdapp) ------ commit ea42ffa13c00263a574226626d30749b6b0f3776 Author: Raghavendra G <rgowdapp> Date: Tue Jan 5 22:16:31 2016 +0530 performance/write-behind: maintain correct transit size in case of short writes. 1. Imagine a write when cache is filled with failed syncs. 2. This write won't be unwound since cache size has exceeded configured limit. 3. With trickling-writes on by default, the last write request wont be considered for winding when there is non zero in-transit size. 4. There was a bug in accounting of in-transit size when winds resulted in short writes. Due to this bug, in-transit size used to be non-zero even when there are no syncs in progress. 5. Due to 3 and 4, current write request won't be wound till there is another write or fsync or flush from application. But application can't do any other fop till current write request is unwound. This resulted in deadlock and hence application would be hung in 'D' state. This patch fixes bug in accounting of in-transit size during short writes. Change-Id: I04ce8bb510efaaed7623cac38d69b32dbc3730ce Signed-off-by: Raghavendra G <rgowdapp> BUG: 1279730 Reviewed-on: http://review.gluster.org/13177 Tested-by: Gluster Build System <jenkins.com>
REVIEW: http://review.gluster.org/13117 (glusterd: GD_OP_VERSION should not be a released one) posted (#7) for review on master by Raghavendra Talur (rtalur)
REVIEW: http://review.gluster.org/13117 (glusterd: GD_OP_VERSION should not be a released one) posted (#8) for review on master by Raghavendra Talur (rtalur)
COMMIT: http://review.gluster.org/13117 committed in master by Raghavendra G (rgowdapp) ------ commit 7d4f708b18c1e6c965ebe8c84e14dd69ae4b7859 Author: Raghavendra Talur <rtalur> Date: Wed Dec 30 16:19:44 2015 +0530 glusterd: GD_OP_VERSION should not be a released one performance.resync-failed-syncs-after-fsync was introduced after 3.7.6 was released. Hence it should use 3_7_7 as op version not 3_7_6. Change-Id: If4def1bf0fdc9fa4938ccb78308bec77eeaa2284 BUG: 1279730 Signed-off-by: Raghavendra Talur <rtalur> Reviewed-on: http://review.gluster.org/13117 Reviewed-by: Atin Mukherjee <amukherj> Tested-by: Gluster Build System <jenkins.com>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user