Bug 1420635 - Modified volume options not synced once offline nodes comes up.
Summary: Modified volume options not synced once offline nodes comes up.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: RHGS 3.2.0
Assignee: Atin Mukherjee
QA Contact: Prasad Desala
URL:
Whiteboard:
Depends On: 1420637 1420991 1420993
Blocks: 1351528
TreeView+ depends on / blocked
 
Reported: 2017-02-09 06:49 UTC by Byreddy
Modified: 2017-03-23 06:05 UTC (History)
5 users (show)

Fixed In Version: glusterfs-3.8.4-15
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1420637 (view as bug list)
Environment:
Last Closed: 2017-03-23 06:05:02 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description Byreddy 2017-02-09 06:49:11 UTC
Description of problem:
=======================
modification done to the volume when some cluster are down are not synced once offline nodes comes up.



Version-Release number of selected component (if applicable):
==============================================================
glusterfs-3.8.4-14

How reproducible:
=================
Always


Steps to Reproduce:
====================
1. Have 3 nodes cluster
2. Create and start a Distributed volume using 3 bricks (pick one from each node)
3. stop glusterd on two nodes (say n2 and n3 )
4. change these volume options from default to 
performance.readdir-ahead from on to off
cluster.server-quorum-ratio from default value to 30
5. Now start glusterd on n2 and n3 nodes
6. check the volume info on both nodes and check modified volume options are synced.

Actual results:
===============
Modified volume options not synced once offline nodes comes up.


Expected results:
=================
sync should happen once nodes comes up.


Additional info:

Comment 2 Byreddy 2017-02-09 06:52:24 UTC
errors in glusterd log:
=======================
[2017-02-09 06:40:29.199737] E [MSGID: 106422] [glusterd-utils.c:4357:glusterd_compare_friend_data] 0-management: Importing global options failed
[2017-02-09 06:40:29.199775] E [MSGID: 106376] [glusterd-sm.c:1397:glusterd_friend_sm] 0-glusterd: handler returned: 2
[2017-02-09 06:40:29.199926] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 273c5136-66a9-4b3e-8f1d-fb45509a4a18, host: dhcp41-198.lab.eng.blr.redhat.com, port: 0
[2017-02-09 06:40:29.238089] I [MSGID: 106492] [glusterd-handler.c:2788:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 273c5136-66a9-4b3e-8f1d-fb45509a4a18
[2017-02-09 06:40:29.238127] I [MSGID: 106502] [glusterd-handler.c:2833:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-02-09 06:40:29.270561] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 273c5136-66a9-4b3e-8f1d-fb45509a4a18
[2017-02-09 06:40:29.270981] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-02-09 06:40:29.271042] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-02-09 06:40:29.271475] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-02-09 06:40:29.271522] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-02-09 06:40:29.271591] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-02-09 06:40:29.271715] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-02-09 06:40:29.271807] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-02-09 06:40:29.271841] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-02-09 06:40:29.271901] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-02-09 06:40:29.271947] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-02-09 06:40:29.272106] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-02-09 06:40:30.976089] I [MSGID: 106488] [glusterd-handler.c:1539:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2017-02-09 06:40:30.977864] I [MSGID: 106488] [glusterd-handler.c:1539:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2017-02-09 06:40:44.641763] I [MSGID: 106163] [glusterd-handshake.c:1274:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30901
[2017-02-09 06:40:44.723849] I [MSGID: 106490] [glusterd-handler.c:2610:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c744d8ef-71ba-4429-9243-0456d2654824
[2017-02-09 06:40:44.764377] I [MSGID: 106493] [glusterd-handler.c:3865:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.43.71 (0), ret: 0, op_ret: 0
[2017-02-09 06:40:44.916543] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-02-09 06:40:44.916586] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-02-09 06:40:44.916926] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-02-09 06:40:44.916951] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-02-09 06:40:44.916985] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-02-09 06:40:44.917006] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-02-09 06:40:44.917041] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-02-09 06:40:44.917067] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-02-09 06:40:44.917133] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-02-09 06:40:44.917161] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-02-09 06:40:44.924636] I [MSGID: 106492] [glusterd-handler.c:2788:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c744d8ef-71ba-4429-9243-0456d2654824
[2017-02-09 06:40:44.941841] I [MSGID: 106502] [glusterd-handler.c:2833:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-02-09 06:50:54.497245] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x4 sent = 2017-02-09 06:40:44.860661. timeout = 600 for 10.70.43.71:24007
(END)

Comment 4 Atin Mukherjee 2017-02-09 07:33:31 UTC
upstream patch : https://review.gluster.org/16574

Comment 5 Byreddy 2017-02-09 08:24:23 UTC
This issue is not there in 3.1.3 GA build.
Added Regression keyword.

Comment 7 Atin Mukherjee 2017-02-09 08:41:34 UTC
http://review.gluster.org/14876 (came as part of rebase to 3.8.4) has caused this regression.

Comment 8 Atin Mukherjee 2017-02-12 15:51:47 UTC
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/97503/

Comment 10 Prasad Desala 2017-02-21 05:45:42 UTC
Verified this BZ on glusterfs version 3.8.4-15.el7rhgs.x86_64.

Followed the below steps:
1) Created and started a Distributed volume using 3 bricks (picked one from each node)
2) stopped glusterd on two nodes (n2 and n3 )
3) From node1, changed the below volume options from default
performance.readdir-ahead from on to off
cluster.server-quorum-ratio from default value to 30
4) Now, started glusterd on n2 and n3 nodes
5) checked the volume info on all the 3 nodes and the above modified volume options are in sync across all the 3 nodes.

Hence, moving this BZ to verified.

Comment 12 errata-xmlrpc 2017-03-23 06:05:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.