Bug 1355628

Summary: Upgrade from 3.7.8 to 3.8.1 doesn't regenerate the volfiles
Product: [Community] GlusterFS Reporter: Kotresh HR <khiremat>
Component: glusterdAssignee: Kotresh HR <khiremat>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: unspecified    
Version: mainlineCC: amukherj, bsrirama, bugs, kramdoss, rcyriac, sanandpa, sankarshan
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: v3.10.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1353470
: 1356426 (view as bug list) Environment:
Last Closed: 2017-03-08 09:31:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1353470    
Bug Blocks: 1356426, 1356439    

Comment 1 Kotresh HR 2016-07-12 06:26:47 UTC
Description of problem:

On CentOS6 base OS, post upgrade of gluster from 3.7.8 to 3.8.1, volfiles are not recreated. During upgrade, glusterd is brought up with "--xlator-option *.upgrade=on -N" parameters to ensure the volfiles are regenerated w.r.t latest bits but that fails as glusterd init () fails from glusterd_check_gsync_present () with a error log saying "0-glusterd: geo-replication module not working as desired".

Version-Release number of selected component (if applicable):


How reproducible:
Always

Additional info:

As a workaround, post 'yum update' following needs to be done.
1. grep -irns "geo-replication module not working as desired" /var/log/glusterfs/etc-glusterfs-glusterd.vol.log | wc -l

If the output is non-zero, then go to step 2

2. Check if glusterd instance is running or not by 'ps aux | grep glusterd', if it is, then stop the glusterd service.

3. glusterd --xlator-option *.upgrade=on -N

and then proceed ahead with the rest of the steps as per the upgrade section from installation guide.

Comment 2 Vijay Bellur 2016-07-12 06:31:07 UTC
REVIEW: http://review.gluster.org/14898 (glusterd: Fix gsyncd upgrade issue) posted (#1) for review on master by Kotresh HR (khiremat)

Comment 3 Vijay Bellur 2016-07-12 06:51:01 UTC
REVIEW: http://review.gluster.org/14898 (glusterd: Fix gsyncd upgrade issue) posted (#2) for review on master by Kotresh HR (khiremat)

Comment 4 Vijay Bellur 2016-07-13 15:02:58 UTC
COMMIT: http://review.gluster.org/14898 committed in master by Atin Mukherjee (amukherj) 
------
commit 1b998788ece8c8b52657e8b9aae65d3279690c5b
Author: Kotresh HR <khiremat>
Date:   Tue Jul 12 00:39:31 2016 +0530

    glusterd: Fix gsyncd upgrade issue
    
    Problem:
        gluster upgrade is not generating new volfiles
    
    Cause:
    During upgrade, "glusterd --xlator-option *.upgrade=on -N"
    is run to generate new volfiles. It is run post 'glusterfs'
    rpm installation. The above command fails during upgrade
    if geo-replication is installed. This is because on
    glusterd start 'gsyncd' binary is called to configure
    geo-replication related stuff. Since 'glusterfs' rpm is
    installed prior to 'geo-rep' rpm, the 'gsyncd' binary
    used to glusterd upgrade command is of old version and
    hence it fails before generating new volfiles.
    
    Solution:
    Don't call geo-replication configure during upgrade/downgrade.
    Geo-replication configuration happens during start of glusterd
    after upgrade.
    
    Change-Id: Id58ea44ead9f69982f86fb68dc5b9ee3f6cd11a1
    BUG: 1355628
    Signed-off-by: Kotresh HR <khiremat>
    Reviewed-on: http://review.gluster.org/14898
    Reviewed-by: Atin Mukherjee <amukherj>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>