Bug 1457732
| Summary: | "split-brain observed [Input/output error]" error messages in samba logs during parallel rm -rf | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Ravishankar N <ravishankar> |
| Component: | replicate | Assignee: | Ravishankar N <ravishankar> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.10 | CC: | bugs, ksubrahm |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | glusterfs-3.10.4 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1456582 | Environment: | |
| Last Closed: | 2017-08-23 10:21:18 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1454689, 1456582, 1460661 | ||
| Bug Blocks: | 1457558, 1457616 | ||
|
Description
Ravishankar N
2017-06-01 08:12:21 UTC
REVIEW: https://review.gluster.org/17515 (posix: use the correct op_errno) posted (#1) for review on release-3.10 by Ravishankar N (ravishankar) REVIEW: https://review.gluster.org/17516 (afr: add errno to afr_inode_refresh_done()) posted (#1) for review on release-3.10 by Ravishankar N (ravishankar) COMMIT: https://review.gluster.org/17515 committed in release-3.10 by Raghavendra Talur (rtalur) ------ commit e1ce2c6616f0466352a92fa400afe2af360139ad Author: Ravishankar N <ravishankar> Date: Mon May 29 21:38:14 2017 +0530 posix: use the correct op_errno Problem: If readdir/fstat was performed on a directory that was removed, posix_fd_ctx_get() fails with ENOENT but we incorrectly use the ret value (-1 in this case) as op_errno, logging "Operation not permitted" messages in the brick logs. Also in case of fstat, the -1 op_errno was also propagated to the client via stack unwind, causing the message to appear in protocol/client logs as well. Fix: Use the right op_errno in readdir, fstat and writev. Also, if posix_fd_ctx_get() failed with ENOENT, convert it into EBADF because ENOENT is not a valid error for an fd operation. > Reviewed-on: https://review.gluster.org/17414 > Smoke: Gluster Build System <jenkins.org> > Reviewed-by: Pranith Kumar Karampuri <pkarampu> > Tested-by: Pranith Kumar Karampuri <pkarampu> > NetBSD-regression: NetBSD Build System <jenkins.org> > Reviewed-by: Amar Tumballi <amarts> > CentOS-regression: Gluster Build System <jenkins.org> (cherry picked from commit de92c363c95d16966dbcc9d8763fd4448dd84d13) Change-Id: Ie43c0789d5040ec73b7cf885d015a183b8c64d70 BUG: 1457732 Signed-off-by: Ravishankar N <ravishankar> Reviewed-on: https://review.gluster.org/17515 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu> COMMIT: https://review.gluster.org/17516 committed in release-3.10 by Raghavendra Talur (rtalur) ------ commit 5bd806f1cdb3a975fe8f2272d47b741c05155ca1 Author: Ravishankar N <ravishankar> Date: Mon Jun 5 09:40:51 2017 +0530 afr: add errno to afr_inode_refresh_done() Backport of https://review.gluster.org/17413 and https://review.gluster.org/17436 Problem: When parellel `rm -rf`s were being done from cifs clients, opendir might fail on some replicas with ENOENT. DHT ignores partial opendir failures in dht_fd_cbk() and winds readdirs on those replicas. Afr inode refresh (as a part of readdirp read_txn) sees in its fd context that the state of the fds is *not* AFR_FD_OPENED and bails out to afr_inode_refresh_done() without doing a refresh. When this happens, the errno is set as EIO due to lack of readable subvols, logging split-brain messages in the logs. Fix: Introduce an errno argument to afr_inode_refresh_do() to bail out with the right error value when inode refresh is not performed. Change-Id: Id88e604278abb8df47750d45258d9e6dde710600 BUG: 1457732 Signed-off-by: Ravishankar N <ravishankar> Reviewed-on: https://review.gluster.org/17516 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu> One more fix to the patch in comment #3 REVIEW: https://review.gluster.org/17571 (posix: Revert modifying op_errno in __posix_fd_ctx_get) posted (#1) for review on release-3.10 by Ravishankar N (ravishankar) COMMIT: https://review.gluster.org/17571 committed in release-3.10 by Raghavendra Talur (rtalur) ------ commit 34b136a8b55a2518529e2bdc43de99e43868dde8 Author: Ravishankar N <ravishankar> Date: Mon Jun 19 13:45:55 2017 +0530 posix: Revert modifying op_errno in __posix_fd_ctx_get https://review.gluster.org/#/c/17414/ converted ENOENT to EBADFD because ENOENT is not a valid error for fd based operations, but this apparently breaks dht rebalance behaviour (see comments in the backport 17517. So reverting that part of the change. > Reviewed-on: https://review.gluster.org/17565 > Reviewed-by: Pranith Kumar Karampuri <pkarampu> > Tested-by: Pranith Kumar Karampuri <pkarampu> > Smoke: Gluster Build System <jenkins.org> > NetBSD-regression: NetBSD Build System <jenkins.org> > CentOS-regression: Gluster Build System <jenkins.org> (cherry picked from commit 61924b98a61108a7ec453fb7f1fc5487d1386e56) Change-Id: Idcf5c65a47b096a3766cf7f20ca938d988572052 BUG: 1457732 Signed-off-by: Ravishankar N <ravishankar> Reviewed-on: https://review.gluster.org/17571 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu> CentOS-regression: Gluster Build System <jenkins.org> |