Bug 1542475 - Random failures in tests/bugs/nfs/bug-974972.t
Summary: Random failures in tests/bugs/nfs/bug-974972.t
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: nfs
Version: 3.12
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Karthik U S
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-06 12:21 UTC by Karthik U S
Modified: 2018-03-05 07:14 UTC (History)
1 user (show)

Fixed In Version: glusterfs-3.12.6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-05 07:14:45 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Karthik U S 2018-02-06 12:21:57 UTC
Description of problem:
Test case bug-974972.t is randomly failing with the below logs.

[2018-02-05 05:54:53.488934]:++++++++++ G_LOG:./bugs/nfs/bug-974972.t: TEST: 30 1 afr_child_up_status_in_nfs patchy 0 ++++++++++
[2018-02-05 05:54:54.541079]:++++++++++ G_LOG:./bugs/nfs/bug-974972.t: TEST: 31 1 afr_child_up_status_in_nfs patchy 1 ++++++++++
[2018-02-05 05:54:55.607606]:++++++++++ G_LOG:./bugs/nfs/bug-974972.t: TEST: 34 ls /mnt/nfs/0/1 ++++++++++
[2018-02-05 05:54:55.612783] W [MSGID: 108027] [afr-common.c:2737:afr_discover_done] 0-patchy-replicate-0: no read subvols for <gfid:ba721bb9-8a3c-44d6-8f26-4e591cafb246>
[2018-02-05 05:54:55.620246]:++++++++++ G_LOG:./bugs/nfs/bug-974972.t: TEST: 35 ! cat /mnt/nfs/0/1 ++++++++++
[2018-02-05 05:54:55.623576] E [MSGID: 108008] [afr-read-txn.c:90:afr_read_txn_refresh_done] 0-patchy-replicate-0: Failing STAT on gfid ba721bb9-8a3c-44d6-8f26-4e591cafb246: split-brain observed. [Input/output error]
[2018-02-05 05:54:55.624789] E [MSGID: 108008] [afr-read-txn.c:90:afr_read_txn_refresh_done] 0-patchy-replicate-0: Failing GETXATTR on gfid ba721bb9-8a3c-44d6-8f26-4e591cafb246: split-brain observed. [Input/output error]
[2018-02-05 05:54:55.627529] W [MSGID: 108027] [afr-common.c:2737:afr_discover_done] 0-patchy-replicate-0: no read subvols for <gfid:ba721bb9-8a3c-44d6-8f26-4e591cafb246>
[2018-02-05 05:54:55.629329] E [MSGID: 108008] [afr-read-txn.c:90:afr_read_txn_refresh_done] 0-patchy-replicate-0: Failing STAT on gfid ba721bb9-8a3c-44d6-8f26-4e591cafb246: split-brain observed. [Input/output error]
[2018-02-05 05:54:55.629382] W [MSGID: 112199] [nfs3-helpers.c:3414:nfs3_log_common_res] 0-nfs-nfsv3: <gfid:ba721bb9-8a3c-44d6-8f26-4e591cafb246> => (XID: 445d2a50, GETATTR: NFS: 5(I/O error), POSIX: 5(Input/output error))
[2018-02-05 05:54:55.635471] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-patchy-replicate-0: performing metadata selfheal on ba721bb9-8a3c-44d6-8f26-4e591cafb246
[2018-02-05 05:54:55.637240] E [MSGID: 108008] [afr-read-txn.c:90:afr_read_txn_refresh_done] 0-patchy-replicate-0: Failing ACCESS on gfid ba721bb9-8a3c-44d6-8f26-4e591cafb246: split-brain observed. [Input/output error]
[2018-02-05 05:54:55.638991] E [MSGID: 108008] [afr-read-txn.c:90:afr_read_txn_refresh_done] 0-patchy-replicate-0: Failing GETXATTR on gfid ba721bb9-8a3c-44d6-8f26-4e591cafb246: split-brain observed. [Input/output error]
[2018-02-05 05:54:55.648613] I [MSGID: 108026] [afr-self-heal-common.c:1656:afr_log_selfheal] 0-patchy-replicate-0: Completed metadata selfheal on ba721bb9-8a3c-44d6-8f26-4e591cafb246. sources=[0]  sinks=1 
[2018-02-05 05:54:55.665468] I [MSGID: 108026] [afr-self-heal-common.c:1656:afr_log_selfheal] 0-patchy-replicate-0: Completed data selfheal on ba721bb9-8a3c-44d6-8f26-4e591cafb246. sources=[1]  sinks=0 

Version-Release number of selected component (if applicable):


How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Read on a split-brain file is allowed.

Expected results:
Read on a split-brain file should fail.

Additional info:

Comment 1 Worker Ant 2018-02-06 12:34:43 UTC
REVIEW: https://review.gluster.org/19510 (nfs: Adding check to make sure write is wound on the file) posted (#1) for review on master by Karthik U S

Comment 2 Worker Ant 2018-02-08 09:26:55 UTC
REVIEW: https://review.gluster.org/19526 (nfs: Fixing the failure in bug-974972.t test case) posted (#1) for review on release-3.12 by Karthik U S

Comment 3 Worker Ant 2018-02-12 10:13:57 UTC
COMMIT: https://review.gluster.org/19526 committed in release-3.12 by "jiffin tony Thottan" <jthottan@redhat.com> with a commit message- nfs: Fixing the failure in bug-974972.t test case

Problem:
gNFS servers is restarted even before the pending marker is set
becuase of eager lock being on, leading to a file in split-brain
to get healed.

Fix:
Switch off the eager lock.
This is fixed in the later versions by the patch:
https://review.gluster.org/#/c/13075/

Change-Id: If59e5ede2de1cbfcdeac01cca38dc0d046f1993c
BUG: 1542475
Signed-off-by: karthik-us <ksubrahm@redhat.com>

Comment 4 Jiffin 2018-03-05 07:14:45 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.12.6, please open a new bug report.

glusterfs-3.12.6 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2018-February/033552.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.