Description of problem: Rebalance will not be started by glusterd if we have any active mounts of the volume Version-Release number of selected component (if applicable): vdsm-gluster-4.13.0-24.el6rhs.noarch glusterfs-fuse-3.4.0.69rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.69rhs-1.el6rhs.x86_64 gluster-swift-account-1.10.0-2.el6rhs.noarch glusterfs-3.4.0.69rhs-1.el6rhs.x86_64 glusterfs-geo-replication-3.4.0.69rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.69rhs-1.el6rhs.x86_64 gluster-swift-plugin-1.10.0-5.el6rhs.noarch gluster-swift-object-1.10.0-2.el6rhs.noarch gluster-swift-container-1.10.0-2.el6rhs.noarch samba-glusterfs-3.6.9-167.10.el6rhs.x86_64 glusterfs-libs-3.4.0.69rhs-1.el6rhs.x86_64 glusterfs-api-3.4.0.69rhs-1.el6rhs.x86_64 gluster-swift-1.10.0-2.el6rhs.noarch How reproducible: always Steps to Reproduce: 1.create a volume and mount it 2. add brick and start rebalance Actual results: rebalance will not be started fails with the error [root@rhs-client4 ~]# gluster v rebalance distrep start volume rebalance: distrep: failed: Volume distrep has one or more connected clients of a version lower than RHS-2.1 update 5. Starting rebalance in this state could lead to data loss. Please disconnect those clients before attempting this command again.
This happens only with fuse mount not NFS
This bug also affects quota. Since quota does a temporary mount whenever enabled, so user will not be able to do "rebalance start" however user can use "start force" to trigger rebalance.
Verified this bug on the latest build of *.70.* as shown below : [root@dht-rhs-20 ~]# rpm -qa | grep -i gluster gluster-swift-container-1.10.0-2.el6rhs.noarch glusterfs-fuse-3.4.0.70rhs-1.el6rhs.x86_64 gluster-swift-1.10.0-2.el6rhs.noarch glusterfs-libs-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-geo-replication-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-api-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.70rhs-1.el6rhs.x86_64 gluster-swift-proxy-1.10.0-2.el6rhs.noarch gluster-swift-account-1.10.0-2.el6rhs.noarch gluster-swift-plugin-1.10.0-5.el6rhs.noarch vdsm-gluster-4.13.0-24.el6rhs.noarch gluster-swift-object-1.10.0-2.el6rhs.noarch samba-glusterfs-3.6.9-167.10.el6rhs.x86_64 glusterfs-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.70rhs-1.el6rhs.x86_64 [root@dht-rhs-20 ~]# Steps followed to verify: 1. Created a volume, mounted it both with Fuse and NFS. 2. Added bricks to it. 3. Started rebalance after adding the brick. 4. Rebalance started without any error message. 5. Further, enabled the quota on the volume and specified a limit to a folder and initiated rebalance again. 6. Rebalance finished successfully without any errors. The screenshots of the verification steps: --------------------------------------------------------------------- [root@dht-rhs-20 ~]# rpm -qa | grep -i gluster gluster-swift-container-1.10.0-2.el6rhs.noarch glusterfs-fuse-3.4.0.70rhs-1.el6rhs.x86_64 gluster-swift-1.10.0-2.el6rhs.noarch glusterfs-libs-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-geo-replication-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-api-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.70rhs-1.el6rhs.x86_64 gluster-swift-proxy-1.10.0-2.el6rhs.noarch gluster-swift-account-1.10.0-2.el6rhs.noarch gluster-swift-plugin-1.10.0-5.el6rhs.noarch vdsm-gluster-4.13.0-24.el6rhs.noarch gluster-swift-object-1.10.0-2.el6rhs.noarch samba-glusterfs-3.6.9-167.10.el6rhs.x86_64 glusterfs-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.70rhs-1.el6rhs.x86_64 [root@dht-rhs-20 ~]# mount /dev/vda2 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/vda1 on /boot type ext4 (rw) /dev/vdb1 on /data/newbrick1 type xfs (rw) /dev/vdb2 on /data/newbrick2 type xfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) localhost:gv0 on /var/run/gluster/gv0 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) 10.70.47.101:/gv0 on /mnt/gvmnt type nfs (rw,addr=10.70.47.101) [root@dht-rhs-20 ~]# gluster volume info Volume Name: gv0 Type: Distribute Volume ID: e9a0d5b5-c043-47e6-9803-20c78d3b79a0 Status: Started Number of Bricks: 11 Transport-type: tcp Bricks: Brick1: 10.70.47.99:/data/newbrick1/gv0 Brick2: 10.70.47.101:/data/newbrick1/gv0 Brick3: 10.70.47.99:/data/newbrick2/br1 Brick4: 10.70.47.99:/data/newbrick2/br2 Brick5: 10.70.47.99:/data/newbrick2/br3 Brick6: 10.70.47.101:/data/newbrick2/br1 Brick7: 10.70.47.101:/data/newbrick2/br2 Brick8: 10.70.47.101:/data/newbrick2/br3 Brick9: 10.70.47.99:/data/newbrick2/br4 Brick10: 10.70.47.101:/data/newbrick2/br4 Brick11: 10.70.47.99:/data/newbrick2/brick-100 Options Reconfigured: features.quota: on [root@dht-rhs-20 ~]# gluster volume add-brick gv0 10.70.47.101:/data/newbrick2/brick-100 force volume add-brick: success [root@dht-rhs-20 ~]# [root@dht-rhs-20 ~]# gluster volume info Volume Name: gv0 Type: Distribute Volume ID: e9a0d5b5-c043-47e6-9803-20c78d3b79a0 Status: Started Number of Bricks: 12 Transport-type: tcp Bricks: Brick1: 10.70.47.99:/data/newbrick1/gv0 Brick2: 10.70.47.101:/data/newbrick1/gv0 Brick3: 10.70.47.99:/data/newbrick2/br1 Brick4: 10.70.47.99:/data/newbrick2/br2 Brick5: 10.70.47.99:/data/newbrick2/br3 Brick6: 10.70.47.101:/data/newbrick2/br1 Brick7: 10.70.47.101:/data/newbrick2/br2 Brick8: 10.70.47.101:/data/newbrick2/br3 Brick9: 10.70.47.99:/data/newbrick2/br4 Brick10: 10.70.47.101:/data/newbrick2/br4 Brick11: 10.70.47.99:/data/newbrick2/brick-100 Brick12: 10.70.47.101:/data/newbrick2/brick-100 Options Reconfigured: features.quota: on [root@dht-rhs-20 ~]# gluster volume rebalance gv0 start volume rebalance: gv0: success: Starting rebalance on volume gv0 has been successful. ID: a701172a-0047-4e89-951b-d9ff95ceb057 [root@dht-rhs-20 ~]# [root@dht-rhs-20 ~]# gluster volume rebalance gv0 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 111 4.2MB 275 0 0 in progress 5.00 10.70.47.101 101 3.9MB 238 0 0 in progress 5.00 volume rebalance: gv0: success: [root@dht-rhs-20 ~]# gluster volume rebalance gv0 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 125 4.8MB 329 0 0 completed 10.00 10.70.47.101 148 5.6MB 307 0 0 completed 10.00 volume rebalance: gv0: success: [root@dht-rhs-20 ~]# mkdir /data/newbrick2/brick-200 [root@dht-rhs-20 ~]# gluster volume add-brick gv0 10.70.47.101:/data/newbrick2/brick-200 force volume add-brick: success [root@dht-rhs-20 ~]# gluster volume add-brick gv0 10.70.47.99:/data/newbrick2/brick-200 force volume add-brick: success [root@dht-rhs-20 ~]# [root@dht-rhs-20 ~]# [root@dht-rhs-20 ~]# mount /dev/vda2 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/vda1 on /boot type ext4 (rw) /dev/vdb1 on /data/newbrick1 type xfs (rw) /dev/vdb2 on /data/newbrick2 type xfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) localhost:gv0 on /var/run/gluster/gv0 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) 10.70.47.101:/gv0 on /mnt/gvmnt type nfs (rw,addr=10.70.47.101) [root@dht-rhs-20 ~]# [root@dht-rhs-20 ~]# rpm -qa | grep -i gluster gluster-swift-container-1.10.0-2.el6rhs.noarch glusterfs-fuse-3.4.0.70rhs-1.el6rhs.x86_64 gluster-swift-1.10.0-2.el6rhs.noarch glusterfs-libs-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-geo-replication-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-api-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.70rhs-1.el6rhs.x86_64 gluster-swift-proxy-1.10.0-2.el6rhs.noarch gluster-swift-account-1.10.0-2.el6rhs.noarch gluster-swift-plugin-1.10.0-5.el6rhs.noarch vdsm-gluster-4.13.0-24.el6rhs.noarch gluster-swift-object-1.10.0-2.el6rhs.noarch samba-glusterfs-3.6.9-167.10.el6rhs.x86_64 glusterfs-3.4.0.70rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.70rhs-1.el6rhs.x86_64 root@dht-rhs-20 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/vda2 34G 2.5G 30G 8% / tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/vda1 485M 59M 401M 13% /boot /dev/vdb1 20G 34M 20G 1% /data/newbrick1 /dev/vdb2 20G 41M 20G 1% /data/newbrick2 localhost:gv0 300G 595M 300G 1% /var/run/gluster/gv0 10.70.47.101:/gv0 300G 595M 300G 1% /mnt/gvmnt [root@dht-rhs-20 ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2014-1853.html