Bug 1005862 - GlusterFS: Can't add a new peer to the cluster - "Number of Bricks" is messed up
GlusterFS: Can't add a new peer to the cluster - "Number of Bricks" is messed up
Status: CLOSED DEFERRED
Product: GlusterFS
Classification: Community
Component: glusterd (Show other bugs)
3.3.2
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: bugs@gluster.org
SATHEESARAN
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-09 10:16 EDT by Tobias Pottgüter
Modified: 2014-12-14 14:40 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-12-14 14:40:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 1005860 None None None Never

  None (edit)
Description Tobias Pottgüter 2013-09-09 10:16:24 EDT
Description of problem:
I can't add another peer to a cluster after a restart, when the "Number of Bricks" shown by "gluster volume info" is messed up.

-------------------------------------------------------

Version-Release number of selected component (if applicable):
3.3.2-2

-------------------------------------------------------

How reproducible:
every time

-------------------------------------------------------

Steps to Reproduce:
I don't know if there is an easier way to encounter this error. That's just what i've done.

1. Get a clean installation of five equal configured servers.
My servers (N = 1-5):
----
serverN> cat /etc/system-release
CentOS release 6.4 (Final)
----
serverN> uname -r
2.6.32-358.14.1.el6.x86_64
----
serverN> yum list installed | grep gluster
glusterfs.x86_64 3.3.2-2.el6 @glusterfs-3-3-epel
glusterfs-fuse.x86_64 3.3.2-2.el6 @glusterfs-3-3-epel
glusterfs-server.x86_64 3.3.2-2.el6 @glusterfs-3-3-epel
----
Gluster isn't configured yet, so let's probe all server.
----
server3> gluster peer probe server1
Probe successful
server3> gluster peer probe server2
Probe successful
server3> gluster peer probe server4
Probe successful
server1> gluster peer probe server3
Probe successful

2. Setup a replica two set on two of them.
Setup a volume with a replica two set, using server3 as the main configuration server.
----
server3> gluster volume create glusterFS replica 2 server3:/mnt/gluster/glusterFS server4:/mnt/gluster/glusterFS
Creation of volume glusterFS has been successful. Please start the volume to access data.
----
Start the volume
----
server3> gluster volume start glusterFS
Starting volume glusterFS has been successful

3. Expand the now existing volume with the replica two set to a replica four set.
Let's expand the volume to four servers.
We do this on our new main configuration server - server1.
----
server1> gluster volume add-brick glusterFS replica 4 server1:/mnt/gluster/glusterFS server2:/mnt/gluster/glusterFS
Add Brick successful

4. Now let's shrink this volume back to a replica two set with only two servers, server1 and server2 remaining.
Let's start the shrinking process.
----
server1> gluster volume remove-brick glusterFS replica 2 server3:/mnt/gluster/glusterFS server4:/mnt/gluster/glusterFS start
Remove Brick start successful
----
Commit if ready.
----
server1> gluster volume remove-brick glusterFS replica 2 server3:/mnt/gluster/glusterFS server4:/mnt/gluster/glusterFS commit
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
Remove Brick commit successful
----

5. Let's see what "gluster volume info" shows
----
server1> gluster volume info

Volume Name: glusterFS
Type: Replicate
Volume ID: 71b83e52-b5ee-41ee-9012-141d35da76b1
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1:/mnt/gluster/glusterFS
Brick2: server2:/mnt/gluster/glusterFS
----
This seems fine. Until we restart gluster.
Let's do that.
----
server1> /etc/init.d/glusterd restart
server2> /etc/init.d/glusterd restart
server3> /etc/init.d/glusterd restart
server4> /etc/init.d/glusterd restart
server5> /etc/init.d/glusterd restart
----
server1> gluster volume info

Volume Name: glusterFS
Type: Replicate
Volume ID: 71b83e52-b5ee-41ee-9012-141d35da76b1
Status: Started
Number of Bricks: 0 x 4 = 2
Transport-type: tcp
Bricks:
Brick1: server1:/mnt/gluster/glusterFS
Brick2: server2:/mnt/gluster/glusterFS
----
And there it goes.
"Number of Bricks: 0 x 4 = 2"

6. Let's try to probe server5 as a new server to the cluster.
----
server1> gluster peer probe server5
----
This takes a minute or so and doesn't show any result in the console.
----
server1> gluster peer status
Number of Peers: 4

Hostname: server2
Uuid: 0e697f01-96c8-4933-803b-1b65fb7180aa
State: Peer in Cluster (Connected)

Hostname: server3
Uuid: b282aad7-348c-47b6-922b-8bb255802a7a
State: Peer in Cluster (Connected)

Hostname: server4
Uuid: 6bf4e23c-24b5-4cb5-bc85-6bdfd9d6a390
State: Peer in Cluster (Connected)

Hostname: server5
Uuid: 9bc0593f-8214-46c9-b913-107b8303ee45
State: Probe Sent to Peer (Connected)
----
"State: Probe Sent to Peer (Connected)" on server5. That's all and it doesn't change.
----
server5 > gluster peer status
Number of Peers: 1

Hostname: server1
Uuid: 0ad8d07e-f8d5-4ce0-992c-56a6c3e0b13d
State: Connected to Peer (Connected)
----
Even a restart doesn't change a thing.
-------------------------------------------------------

Actual results:
"Number of Bricks: 0 x 4 = 2"
Unable to add a server to the cluster.

-------------------------------------------------------

Expected results:
"Number of Bricks: 1 x 2 = 2"
Being able to add a server to the cluster.

-------------------------------------------------------

Additional info:
none

-------------------------------------------------------
Comment 1 Tobias Pottgüter 2013-09-24 02:20:46 EDT
Any progress on this?
Comment 2 Richard 2013-09-24 03:42:10 EDT
Try the latest QA release if your playing with gluster as its much better than the 3.2 or 3.3 release.

http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.4.1qa1/
Comment 3 Niels de Vos 2014-11-27 09:54:34 EST
The version that this bug has been reported against, does not get any updates from the Gluster Community anymore. Please verify if this report is still valid against a current (3.4, 3.5 or 3.6) release and update the version, or close this bug.

If there has been no update before 9 December 2014, this bug will get automatocally closed.

Note You need to log in before you can comment on or make changes to this bug.