Bug 1220051 - Data Tiering: Volume inconsistency errors getting logged when attaching uneven(odd) number of hot bricks in hot tier(pure distribute tier layer) to a dist-rep volume
Summary: Data Tiering: Volume inconsistency errors getting logged when attaching uneve...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: tiering
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact: bugs@gluster.org
URL:
Whiteboard:
Depends On: 1216898
Blocks: qe_tracker_everglades glusterfs-tiering-supportability 1229269 1260923 1273726 1274411
TreeView+ depends on / blocked
 
Reported: 2015-05-09 13:01 UTC by Mohammed Rafi KC
Modified: 2015-10-30 17:32 UTC (History)
4 users (show)

Fixed In Version: glusterfs-3.7.0
Doc Type: Bug Fix
Doc Text:
Clone Of: 1216898
Environment:
Last Closed: 2015-05-15 17:10:10 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Mohammed Rafi KC 2015-05-09 13:01:09 UTC
+++ This bug was initially created as a clone of Bug #1216898 +++

Description of problem:
=======================
When we try to attach a hot tier with only distribute(no replica set in hot layer) hot bricks, with the distribute being odd number of bricks(say 1 hot brick only), then there are errors displayed for volume inconsistency as below:


Also it can be seen that the number of bricks formula is wrong(though i guess this may be fixed as part of displaying hot and cold tier  seperately)
Number of Bricks: 2 x 2 = 5


[2015-04-29 07:07:42.214122] I [glusterd-brick-ops.c:388:__glusterd_handle_add_brick] 0-management: Received add brick req
[2015-04-29 07:07:42.218608] E [glusterd-brick-ops.c:1343:glusterd_op_stage_add_brick] 0-management: Unable to get replica count
[2015-04-29 07:07:42.231549] E [run.c:190:runner_log] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f9f6dda7116] (--> /lib64/libglusterfs.so.0(runner_log+0xfc)[0x7f9f6ddf419c] (--> /usr/lib64/glusterfs/3.7.0alpha0/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x47a)[0x7f9f62c181ba] (--> /usr/lib64/glusterfs/3.7.0alpha0/xlator/mgmt/glusterd.so(+0x51e15)[0x7f9f62b99e15] (--> /usr/lib64/glusterfs/3.7.0alpha0/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x46)[0x7f9f62ba0176] ))))) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=vol200 --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd
[2015-04-29 07:07:42.231625] I [glusterd-brick-ops.c:1108:glusterd_op_perform_add_bricks] 0-management: type is set 5, need to change it
[2015-04-29 07:07:42.237710] E [glusterd-volgen.c:2713:volgen_graph_build_clients] 0-: volume inconsistency: total number of bricks (5) is not divisible with number of bricks per cluster (2) in a multi-cluster setup
[2015-04-29 07:07:42.237741] E [glusterd-volgen.c:4976:glusterd_create_volfiles] 0-management: Could not generate trusted client volfiles
[2015-04-29 07:07:42.237752] E [glusterd-brick-ops.c:1919:glusterd_op_add_brick] 0-: Unable to add bricks
[2015-04-29 07:07:42.237770] E [glusterd-syncop.c:1330:gd_commit_op_phase] 0-management: Commit of operation 'Volume Add brick' failed on localhost    
[2015-04-29 07:07:47.096802] I [glusterd-handler.c:1317:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req



Version-Release number of selected component (if applicable):
============================================================
glusterfs 3.7.0alpha0 built on Apr 28 2015 01:37:11
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@yarrow ~]# rpm -qa|grep gluster
glusterfs-libs-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-server-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-fuse-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-api-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-cli-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
[root@yarrow ~]# 


Steps to Reproduce:
==================
1.create a dist-rep volume and start the volume
[root@yarrow ~]# gluster v create vol200 replica 2 moonshine:/brick_100G_9/vol200 yarrow:/brick_100G_9/vol200 moonshine:/brick_100G_10/vol200 yarrow:/brick_100G_10/vol200 force
volume create: vol200: success: please start the volume to access data
[root@yarrow ~]# gluster v info vol200
 
Volume Name: vol200
Type: Distributed-Replicate
Volume ID: bb0c179a-c466-4137-a8bd-110788481aa1
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: moonshine:/brick_100G_9/vol200
Brick2: yarrow:/brick_100G_9/vol200
Brick3: moonshine:/brick_100G_10/vol200
Brick4: yarrow:/brick_100G_10/vol200
[root@yarrow ~]# gluster v start vol200
volume start: vol200: success

2.attach a tier with only one hot brick and view the logs

[root@yarrow ~]# gluster v attach-tier vol200 yarrow:/ssdbricks_50G_1/vol200
volume add-brick: failed: Commit failed on localhost. Please check the log file for more details.
[root@yarrow ~]# gluster v info vol200
 
Volume Name: vol200
Type: Tier
Volume ID: bb0c179a-c466-4137-a8bd-110788481aa1
Status: Started
Number of Bricks: 2 x 2 = 5
Transport-type: tcp
Bricks:
Brick1: yarrow:/ssdbricks_50G_1/vol200
Brick2: moonshine:/brick_100G_9/vol200
Brick3: yarrow:/brick_100G_9/vol200
Brick4: moonshine:/brick_100G_10/vol200
Brick5: yarrow:/brick_100G_10/vol200


Actual results:


Expected results:


Additional info:

--- Additional comment from nchilaka on 2015-04-29 03:28:03 EDT ---

logs available at [root@rhsqe-repo sosreports]# pwd
/home/repo/sosreports/1216898

--- Additional comment from Anand Avati on 2015-05-02 09:51:28 EDT ---

REVIEW: http://review.gluster.org/10494 (tiering: Remove unwanted check for tier type volume) posted (#1) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Anand Avati on 2015-05-04 11:53:09 EDT ---

REVIEW: http://review.gluster.org/10494 (tiering: Remove unwanted check for tier type volume) posted (#2) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Anand Avati on 2015-05-05 01:25:01 EDT ---

REVIEW: http://review.gluster.org/10494 (tiering: Remove unwanted check for tier type volume) posted (#3) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Anand Avati on 2015-05-08 17:56:53 EDT ---

REVIEW: http://review.gluster.org/10494 (tiering: Remove unwanted check for tier type volume) posted (#4) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Anand Avati on 2015-05-08 23:14:27 EDT ---

COMMIT: http://review.gluster.org/10494 committed in master by Vijay Bellur (vbellur) 
------
commit 5fda96d954dd06dbacc6b2851d4ba058ec1295a2
Author: Mohammed Rafi KC <rkavunga>
Date:   Sat May 2 19:19:27 2015 +0530

    tiering: Remove unwanted check for tier type volume
    
    Change-Id: I2def61ebf348558e5f6a138265e3329d9a5407a3
    BUG: 1216898
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: http://review.gluster.org/10494
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Gluster Build System <jenkins.com>
    Tested-by: NetBSD Build System

Comment 1 Anand Avati 2015-05-09 21:20:58 UTC
REVIEW: http://review.gluster.org/10712 (tiering: Remove unwanted check for tier type volume) posted (#2) for review on release-3.7 by Niels de Vos (ndevos)

Comment 2 Anand Avati 2015-05-10 03:27:26 UTC
COMMIT: http://review.gluster.org/10712 committed in release-3.7 by Vijay Bellur (vbellur) 
------
commit 9ff65ba5ff96b92ee39b48b06a9b08873c248e87
Author: Mohammed Rafi KC <rkavunga>
Date:   Sat May 2 19:19:27 2015 +0530

    tiering: Remove unwanted check for tier type volume
    
            Back port of http://review.gluster.org/10494
    
    >Change-Id: I2def61ebf348558e5f6a138265e3329d9a5407a3
    >BUG: 1216898
    >Signed-off-by: Mohammed Rafi KC <rkavunga>
    >Reviewed-on: http://review.gluster.org/10494
    >Reviewed-by: Dan Lambright <dlambrig>
    >Tested-by: Gluster Build System <jenkins.com>
    >Tested-by: NetBSD Build System
    
    Change-Id: I35f460297dfd6b6883c62a6826c99e4f1f3aece2
    BUG: 1220051
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: http://review.gluster.org/10712
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Gluster Build System <jenkins.com>
    Tested-by: NetBSD Build System

Comment 3 Niels de Vos 2015-05-15 17:10:10 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.