Description of problem: libvirt fails live-migration when brick processes occupy/use/block ports. By default glusterfsd processes start to listen on port 49152 and up, libvirt uses the same base port(range) for live-migration. Version-Release number of selected component (if applicable): glusterfs-3.4 How reproducible: 100% Steps to Reproduce: 1. configure several glusterfs volumes 2. install/start some virtual machines 3. try to live-migrate virtual machines with libvirt Actual results: libvirt tells QEMU to listen on a port that glusterfs is using. QEMU can not listen on the port and live-migration is failing. Expected results: live-migration works Additional info: See bug 987555. IRC discussion with Kaleb: 14:53 < kkeithley_wfh> Are we really sure we want it in upstream on the master branch? Or any other branch? 14:54 < ndevos> I do not think there is something against adding such an option, you never know what other software tries to use those ports too 14:56 < kkeithley_wfh> Then they would be just as broken as libvirt is. (At least we are well behaved and try other ports until we find an unused one.) 14:57 < kkeithley_wfh> they would be broken if they rely on getting 49152 (or any other port in that range) like libvirt does 14:57 < ndevos> yes, that is true, but it would be nice to be able to have a workaround for other broken softwate 14:57 < ndevos> *software 14:57 < ndevos> and, I am not sure if/how glusterd checks if a port is free... 14:57 < ndevos> is it not broken that way either? 14:58 < kkeithley_wfh> glusterd tries to bind to it, if it fails (because it's already in use) then it tries another. 14:59 < ndevos> ah, okay, but live-migration may be in progress when glusterfsd starts up? does glusterfsd fail-over to a different port? 15:01 < kkeithley_wfh> yes, it'll try to bind to 49152. If something else is already using that port, glusterd will try 49153, and so on, until it finds a port that isn't in use 15:02 < kkeithley_wfh> we are already well behaved 15:04 < ndevos> ah, thats pretty cool :) 15:18 < kkeithley_wfh> ndevos: http://review.gluster.org/6147
Verification that the patch works: 1. create and start a volume: [root@vm130-31 tmp]# gluster volume create somevol replica 2 vm130-31:/bricks/somevol/data vm130-32:/bricks/somevol/data volume create: somevol: success: please start the volume to access data [root@vm130-31 tmp]# gluster volume start somevol volume start: somevol: success 2. check the ports used (default is 49152) [root@vm130-31 tmp]# gluster volume status Status of volume: somevol Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick vm130-31:/bricks/somevol/data 49152 Y 1118 Brick vm130-32:/bricks/somevol/data 49152 Y 26308 NFS Server on localhost 2049 Y 1128 Self-heal Daemon on localhost N/A Y 1135 NFS Server on 192.168.130.32 2049 Y 26320 Self-heal Daemon on 192.168.130.32 N/A Y 26324 There are no active volume tasks 3. uncomment and change the base-port option (50152): [root@vm130-31 tmp]# vi /etc/glusterfs/glusterd.vol 4. restart the processes [root@vm130-31 tmp]# systemctl restart glusterfsd [root@vm130-31 tmp]# systemctl restart glusterd 5. check if the port has been modified [root@vm130-31 tmp]# gluster volume status Status of volume: somevol Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick vm130-31:/bricks/somevol/data 50152 Y 1204 Brick vm130-32:/bricks/somevol/data 49152 Y 26308 NFS Server on localhost 2049 Y 1212 Self-heal Daemon on localhost N/A Y 1219 NFS Server on 192.168.130.32 2049 Y 26320 Self-heal Daemon on 192.168.130.32 N/A Y 26324 There are no active volume tasks -> VERIFIED, vm130-31 uses port 50152 after updating base-port in glusterd.vol
glusterfs-3.4.1-3.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/glusterfs-3.4.1-3.fc18
glusterfs-3.4.1-3.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/glusterfs-3.4.1-3.fc19
glusterfs-3.4.1-3.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/glusterfs-3.4.1-3.fc20
Package glusterfs-3.4.1-3.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing glusterfs-3.4.1-3.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-20091/glusterfs-3.4.1-3.fc20 then log in and leave karma (feedback).
glusterfs-3.4.1-3.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
glusterfs-3.4.1-3.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.
glusterfs-3.4.1-3.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.