Bug 1247947 - [upgrade] After in-service software upgrade from RHGS 2.1 to RHGS 3.1, bumping up op-version failed
[upgrade] After in-service software upgrade from RHGS 2.1 to RHGS 3.1, bumpin...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
3.1
x86_64 Linux
unspecified Severity high
: ---
: RHGS 3.1.2
Assigned To: Satish Mohan
Byreddy
glusterd
: ZStream
Depends On:
Blocks: 1260783
  Show dependency treegraph
 
Reported: 2015-07-29 06:15 EDT by SATHEESARAN
Modified: 2016-03-10 02:20 EST (History)
9 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-0.3
Doc Type: Bug Fix
Doc Text:
Previously, bump-up op-version command failed after upgrading to Red Hat Gluster Storage 3.1. With this release, this issue is fixed.
Story Points: ---
Clone Of:
: 1248298 (view as bug list)
Environment:
Last Closed: 2016-03-01 00:33:08 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description SATHEESARAN 2015-07-29 06:15:28 EDT
Description of problem:
------------------------
Upgraded RHGS 2.1 nodes to RHGS 3.1, using In-service Software Upgrade.
After upgrade, bumping up op-version to 30703 failed

Version-Release number of selected component (if applicable):
--------------------------------------------------------------
RHGS 2.1 ( glusterfs-3.4.0.72-1.el6rhs )
RHGS 3.1 ( glusterfs-3.7.1-11.el6rhs )

How reproducible:
------------------
Always

Steps to Reproduce:
--------------------
1. Upgrade RHGS 2.1 Nodes to RHGS 3.1, in In-service Software Upgrade
2. After upgrade bump up op-version to 30703

Actual results:
---------------
Bumping up op-version failed

Expected results:
-----------------
Bumping up op-version should succeed

Additional info:
----------------
[2015-07-29 11:50:31.860731]  : volume set all cluster.op-version 30703 : FAILED :

[root@ ~]# gluster volume get drvol op-version
Option Value
------ -----
cluster.op-version                      30703

Following are the logs from 2 nodes.

NODE-1
----------
[2015-07-29 11:50:31.860355] E [MSGID: 106116] [glusterd-mgmt.c:134:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on dhcp37-126.lab.eng.blr.redhat.com. Please check log file for details.
[2015-07-29 11:50:31.860493] E [MSGID: 106152] [glusterd-syncop.c:1562:gd_unlock_op_phase] 0-management: Failed to unlock on some peer(s)
[2015-07-29 11:50:31.860587] E [MSGID: 106025] [glusterd-locks.c:641:glusterd_mgmt_v3_unlock] 0-management: name is null. [Invalid argument]
[2015-07-29 11:50:31.860666] E [MSGID: 106118] [glusterd-syncop.c:1588:gd_unlock_op_phase] 0-management: Unable to release lock for (null)
[2015-07-29 11:50:31.875251] I [run.c:190:runner_log] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fdcd220c5e0] (--> /usr/lib64/libglusterfs.so.0(runner_log+0x105)[0x7fdcd225ff95] (--> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x4cc)[0x7fdcc6cac10c] (--> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(+0xed422)[0x7fdcc6cac422] (--> /lib64/libpthread.so.0(+0x3429c07a51)[0x7fdcd12f3a51] ))))) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=all -o cluster.op-version=30703 --gd-workdir=/var/lib/glusterd
[2015-07-29 11:50:31.893561] I [run.c:190:runner_log] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fdcd220c5e0] (--> /usr/lib64/libglusterfs.so.0(runner_log+0x105)[0x7fdcd225ff95] (--> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x4cc)[0x7fdcc6cac10c] (--> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(+0xed422)[0x7fdcc6cac422] (--> /lib64/libpthread.so.0(+0x3429c07a51)[0x7fdcd12f3a51] ))))) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=all -o cluster.op-version=30703 --gd-workdir=/var/lib/glusterd


NODE-2
-----------

[2015-07-29 11:50:31.622533] E [MSGID: 106118] [glusterd-op-sm.c:3619:glusterd_op_ac_unlock] 0-management: Unable to release lock for all
[2015-07-29 11:50:31.622788] E [MSGID: 106376] [glusterd-op-sm.c:7286:glusterd_op_sm] 0-management: handler returned: -1
Comment 1 SATHEESARAN 2015-07-29 07:32:24 EDT
The volume set fails, but the op-version actually got bumped up.
There are no problems functionally
Comment 2 Anand Nekkunti 2015-10-05 01:29:15 EDT
upstream patch merged: http://review.gluster.org/#/c/11798/
Comment 5 Anand Nekkunti 2015-10-15 13:26:31 EDT
Due to rebase with 3.7.5 , patch is pulled automatically to 3.1.2 branch , so moving to modified.
Comment 6 Byreddy 2015-10-20 05:53:23 EDT
This bug is verified with the  rhgs version = glusterfs-3.7.5-0.2

Steps Done:
----------
1. Created two node cluster with rhgs 2.1 update 6.
2. Created Distributed and replica volume
3. Done in-service update from 2.1u6 to 3.1.2, one node at a time.
4. Bumped up the Op-version to 30705 ( which is for 3.1.2) and it worked successfully. 
    [root@ ~]# gluster volume set all cluster.op-version 30706
    volume set: success


5. verified the op-version  by querying it.
[root@ ~]# gluster volume get replica  cluster.op-version
Option                                  Value                                   
------                                  -----                                   
cluster.op-version                      30706                                   

[root@ ~]# gluster volume get Dis  cluster.op-version
Option                                  Value                                   
------                                  -----                                   
cluster.op-version                      30706                                   
[root@ ~]# 

Moving to verified state based on above info
Comment 10 errata-xmlrpc 2016-03-01 00:33:08 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html

Note You need to log in before you can comment on or make changes to this bug.