Description of problem: I can't add another (a third) brick to a volume with a replica two set, because the "Number of Bricks" shown by "gluster volume info" is messed up. ------------------------------------------------------- Version-Release number of selected component (if applicable): 3.3.2-2 ------------------------------------------------------- How reproducible: every time ------------------------------------------------------- Steps to Reproduce: I don't know if there is an easier way to encounter this error. That's just what i've done. 1. Get a clean installation of five equal configured servers. My servers (N = 1-5): ---- serverN> cat /etc/system-release CentOS release 6.4 (Final) ---- serverN> uname -r 2.6.32-358.14.1.el6.x86_64 ---- serverN> yum list installed | grep gluster glusterfs.x86_64 3.3.2-2.el6 @glusterfs-3-3-epel glusterfs-fuse.x86_64 3.3.2-2.el6 @glusterfs-3-3-epel glusterfs-server.x86_64 3.3.2-2.el6 @glusterfs-3-3-epel ---- Gluster isn't configured yet, so let's probe all server. ---- server3> gluster peer probe server1 Probe successful server3> gluster peer probe server2 Probe successful server3> gluster peer probe server4 Probe successful server3> gluster peer probe server5 Probe successful server1> gluster peer probe server3 Probe successful 2. Setup a replica two set on two of them. Setup a volume with a replica two set, using server3 as the main configuration server. ---- server3> gluster volume create glusterFS replica 2 server3:/mnt/gluster/glusterFS server4:/mnt/gluster/glusterFS Creation of volume glusterFS has been successful. Please start the volume to access data. ---- Start the volume ---- server3> gluster volume start glusterFS Starting volume glusterFS has been successful 3. Expand the now existing volume with the replica two set to a replica four set. Let's expand the volume to four servers. We do this on our new main configuration server - server1. ---- server1> gluster volume add-brick glusterFS replica 4 server1:/mnt/gluster/glusterFS server2:/mnt/gluster/glusterFS Add Brick successful 4. Now let's shrink this volume back to a replica two set with only two servers, server1 and server2 remaining. Let's start the shrinking process. ---- server1> gluster volume remove-brick glusterFS replica 2 server3:/mnt/gluster/glusterFS server4:/mnt/gluster/glusterFS start Remove Brick start successful ---- Commit if ready. ---- server1> gluster volume remove-brick glusterFS replica 2 server3:/mnt/gluster/glusterFS server4:/mnt/gluster/glusterFS commit Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y Remove Brick commit successful ---- 5. Let's see what "gluster volume info" shows ---- server1> gluster volume info Volume Name: glusterFS Type: Replicate Volume ID: 71b83e52-b5ee-41ee-9012-141d35da76b1 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: server1:/mnt/gluster/glusterFS Brick2: server2:/mnt/gluster/glusterFS ---- This seems fine. Until we restart gluster. Let's do that. ---- server1> /etc/init.d/glusterd restart server2> /etc/init.d/glusterd restart server3> /etc/init.d/glusterd restart server4> /etc/init.d/glusterd restart server5> /etc/init.d/glusterd restart ---- server1> gluster volume info Volume Name: glusterFS Type: Replicate Volume ID: 71b83e52-b5ee-41ee-9012-141d35da76b1 Status: Started Number of Bricks: 0 x 4 = 2 Transport-type: tcp Bricks: Brick1: server1:/mnt/gluster/glusterFS Brick2: server2:/mnt/gluster/glusterFS ---- And there it goes. "Number of Bricks: 0 x 4 = 2" 6. Let's try to add server5 as a new brick to the volume, expanding this volume to a replica 3 set. ---- server1> gluster volume add-brick glusterFS replica 3 server5:/mnt/gluster/glusterFS Incorrect replica count (3) supplied. Volume already has (4) ---- Something went horribly wrong. Volume should have two bricks, as it is shown under "Bricks:" in "gluster volume info", and not four. ------------------------------------------------------- Actual results: "Number of Bricks: 0 x 4 = 2" Unable to add a brick to a volume. ------------------------------------------------------- Expected results: "Number of Bricks: 1 x 2 = 2" Being able to add a brick to a volume. ------------------------------------------------------- Additional info: none ------------------------------------------------------- Possible solution: On each server, set "sub_count=4" in "/var/lib/glusterd/vols/glusterFS/info" to "sub_count=2". Restart each server. Now "gluster volume info" shows "Number of Bricks: 1 x 2 = 2". ---- server1> gluster volume info Volume Name: glusterFS Type: Replicate Volume ID: 71b83e52-b5ee-41ee-9012-141d35da76b1 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: server1:/mnt/gluster/glusterFS Brick2: server2:/mnt/gluster/glusterFS -------------------------------------------------------
Any progress on this?
As far as I understand it you have to add new servers in the same number as the volumes in the current replica. so in your example, as you original replica had 2 servers mirroring each other in it, you have to add you new servers in pairs. If you add two new servers to this it should work. Also, try the latest QA release if your playing with gluster as its much better than the 3.2 or 3.3 release. http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.4.1qa1/ It may or may not be production ready, but I'm using it on my development servers without issue. Rich
Tobias, Please refer BZ 1002556 for the fix, which has been merged upstream in master and 3.4. The same patch http://review.gluster.org/#/c/5902/ can be applied in 3.3 also.
The version that this bug has been reported against, does not get any updates from the Gluster Community anymore. Please verify if this report is still valid against a current (3.4, 3.5 or 3.6) release and update the version, or close this bug. If there has been no update before 9 December 2014, this bug will get automatocally closed.