Bug 1220347
Summary: | Read operation on a file which is in split-brain condition is successful | |||
---|---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Shruti Sampat <ssampat> | |
Component: | replicate | Assignee: | bugs <bugs> | |
Status: | CLOSED DUPLICATE | QA Contact: | ||
Severity: | high | Docs Contact: | ||
Priority: | unspecified | |||
Version: | 3.7.0 | CC: | bugs, gluster-bugs, jdarcy, ravishankar, rtalur | |
Target Milestone: | --- | Keywords: | Triaged | |
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | Bug Fix | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1223758 1224709 (view as bug list) | Environment: | ||
Last Closed: | 2015-06-09 16:29:05 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1223758, 1224709 |
Description
Shruti Sampat
2015-05-11 11:54:03 UTC
Observations from debugging the setup. When debugging the mount process with gdb, it was observed that in afr_lookup_done, we do afr_inode_read_subvol_reset() and consequently when afr_read_txn(), afr_read_txn_refresh_done() is called, we bail out because there are no read subvols and the client gets EIO. When no gdb was attached, the client again began reading stale data. On further examination, it was observed that fuse sends the following FOPS when 'cat' was performed on the mount: 1)fuse_fop_resume-->fuse_lookup_resume 2)fuse_fop_resume-->fuse_open_resume 3)fuse_fop_resume-->fuse_getattr_resume--->afr_fstat-->afr_read_txn-->bail out with EIO. 4)fuse_fop_resume-->fuse_flush_resume However when 'cat' was done in rapid succession, (3) was not being called. i.e only fuse_lookup_resume, fuse_open_resume and fuse_flush_resume were being called. Since the getattr was not sent by fuse, it did not get the EIO and was serving data from kernel cache. It was noted that this data returned was always the one written to the latest brick, "World" in this case. I don't think we should hit the issue if we perform a 1) drop_caches on the existing mount, or 2) do a remount or 3)mount with the options attribute-timeout and entry-timeout set to zero to begin with.
>
> I don't think we should hit the issue if we perform a 1) drop_caches on the
> existing mount, or 2) do a remount or 3)mount with the options
> attribute-timeout and entry-timeout set to zero to begin with.
Tried each of the above 3 and did not hit the issue.
Can be closed now that it is proved it kernel cache in action? or can be this taken as a feature? Ravi, I guess you can decide. Raghavendra G has suggested a fix where we can set attribute-timeout to zero for the files that are in split-brain forcing fuse to send a fuse_getattr_resume(). I'll send a patch for it, let us see if it is acceptable. Keeping the bug open until then. Closing this as a duplicate of 1229226 (instead of the other way around) because there's more discussion there. *** This bug has been marked as a duplicate of bug 1229226 *** |