Bug 1005860

Description of problem:
I can't add another (a third) brick to a volume with a replica two set, because the "Number of Bricks" shown by "gluster volume info" is messed up.

-------------------------------------------------------

Version-Release number of selected component (if applicable):
3.3.2-2

-------------------------------------------------------

How reproducible:
every time

-------------------------------------------------------

Steps to Reproduce:
I don't know if there is an easier way to encounter this error. That's just what i've done.

1. Get a clean installation of five equal configured servers.
My servers (N = 1-5):
----
serverN> cat /etc/system-release
CentOS release 6.4 (Final)
----
serverN> uname -r
2.6.32-358.14.1.el6.x86_64
----
serverN> yum list installed | grep gluster
glusterfs.x86_64 3.3.2-2.el6 @glusterfs-3-3-epel
glusterfs-fuse.x86_64 3.3.2-2.el6 @glusterfs-3-3-epel
glusterfs-server.x86_64 3.3.2-2.el6 @glusterfs-3-3-epel
----
Gluster isn't configured yet, so let's probe all server.
----
server3> gluster peer probe server1
Probe successful
server3> gluster peer probe server2
Probe successful
server3> gluster peer probe server4
Probe successful
server3> gluster peer probe server5
Probe successful
server1> gluster peer probe server3
Probe successful

2. Setup a replica two set on two of them.
Setup a volume with a replica two set, using server3 as the main configuration server.
----
server3> gluster volume create glusterFS replica 2 server3:/mnt/gluster/glusterFS server4:/mnt/gluster/glusterFS
Creation of volume glusterFS has been successful. Please start the volume to access data.
----
Start the volume
----
server3> gluster volume start glusterFS
Starting volume glusterFS has been successful

3. Expand the now existing volume with the replica two set to a replica four set.
Let's expand the volume to four servers.
We do this on our new main configuration server - server1.
----
server1> gluster volume add-brick glusterFS replica 4 server1:/mnt/gluster/glusterFS server2:/mnt/gluster/glusterFS
Add Brick successful

4. Now let's shrink this volume back to a replica two set with only two servers, server1 and server2 remaining.
Let's start the shrinking process.
----
server1> gluster volume remove-brick glusterFS replica 2 server3:/mnt/gluster/glusterFS server4:/mnt/gluster/glusterFS start
Remove Brick start successful
----
Commit if ready.
----
server1> gluster volume remove-brick glusterFS replica 2 server3:/mnt/gluster/glusterFS server4:/mnt/gluster/glusterFS commit
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
Remove Brick commit successful
----

5. Let's see what "gluster volume info" shows
----
server1> gluster volume info

Volume Name: glusterFS
Type: Replicate
Volume ID: 71b83e52-b5ee-41ee-9012-141d35da76b1
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1:/mnt/gluster/glusterFS
Brick2: server2:/mnt/gluster/glusterFS
----
This seems fine. Until we restart gluster.
Let's do that.
----
server1> /etc/init.d/glusterd restart
server2> /etc/init.d/glusterd restart
server3> /etc/init.d/glusterd restart
server4> /etc/init.d/glusterd restart
server5> /etc/init.d/glusterd restart
----
server1> gluster volume info

Volume Name: glusterFS
Type: Replicate
Volume ID: 71b83e52-b5ee-41ee-9012-141d35da76b1
Status: Started
Number of Bricks: 0 x 4 = 2
Transport-type: tcp
Bricks:
Brick1: server1:/mnt/gluster/glusterFS
Brick2: server2:/mnt/gluster/glusterFS
----
And there it goes.
"Number of Bricks: 0 x 4 = 2"

6. Let's try to add server5 as a new brick to the volume, expanding this volume to a replica 3 set.
----
server1> gluster volume add-brick glusterFS replica 3 server5:/mnt/gluster/glusterFS
Incorrect replica count (3) supplied. Volume already has (4)
----
Something went horribly wrong. Volume should have two bricks, as it is shown under "Bricks:" in "gluster volume info", and not four.
-------------------------------------------------------

Actual results:
"Number of Bricks: 0 x 4 = 2"
Unable to add a brick to a volume.

-------------------------------------------------------

Expected results:
"Number of Bricks: 1 x 2 = 2"
Being able to add a brick to a volume.

-------------------------------------------------------

Additional info:
none

-------------------------------------------------------

Possible solution:
On each server, set "sub_count=4" in "/var/lib/glusterd/vols/glusterFS/info" to "sub_count=2".
Restart each server.
Now "gluster volume info" shows "Number of Bricks: 1 x 2 = 2".
----
server1> gluster volume info

Volume Name: glusterFS
Type: Replicate
Volume ID: 71b83e52-b5ee-41ee-9012-141d35da76b1
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1:/mnt/gluster/glusterFS
Brick2: server2:/mnt/gluster/glusterFS
-------------------------------------------------------