Bug 1023653

Summary:	Add an option to /etc/glusterfs/glusterd.vol to change the base-port of brick processes
Product:	[Fedora] Fedora	Reporter:	Niels de Vos <ndevos>
Component:	glusterfs	Assignee:	Niels de Vos <ndevos>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	19	CC:	barumuga, joe, jonathansteffan, kkeithle, ndevos, silas
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.4.1-3.fc19	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-12-14 03:05:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Niels de Vos 2013-10-26 15:10:20 UTC

Description of problem:
libvirt fails live-migration when brick processes occupy/use/block ports. By default glusterfsd processes start to listen on port 49152 and up, libvirt uses the same base port(range) for live-migration.

Version-Release number of selected component (if applicable):
glusterfs-3.4

How reproducible:
100%

Steps to Reproduce:
1. configure several glusterfs volumes
2. install/start some virtual machines
3. try to live-migrate virtual machines with libvirt

Actual results:
libvirt tells QEMU to listen on a port that glusterfs is using. QEMU can not listen on the port and live-migration is failing.

Expected results:
live-migration works

Additional info:
See bug 987555.

IRC discussion with Kaleb:
14:53 < kkeithley_wfh> Are we really sure we want it in upstream on the master
                       branch? Or any other branch?
14:54 < ndevos> I do not think there is something against adding such an
		option, you never know what other software tries to use those
		ports too
14:56 < kkeithley_wfh> Then they would be just as broken as libvirt is.  (At
		       least we are well behaved and try other ports until we
		       find an unused one.)
14:57 < kkeithley_wfh> they would be broken if they rely on getting 49152 (or
		       any other port in that range) like libvirt does
14:57 < ndevos> yes, that is true, but it would be nice to be able to have a
                workaround for other broken softwate
14:57 < ndevos> *software
14:57 < ndevos> and, I am not sure if/how glusterd checks if a port is free...
14:57 < ndevos> is it not broken that way either?
14:58 < kkeithley_wfh> glusterd tries to bind to it, if it fails (because it's
                       already in use) then it tries another.
14:59 < ndevos> ah, okay, but live-migration may be in progress when
		glusterfsd starts up? does glusterfsd fail-over to a different
		port?
15:01 < kkeithley_wfh> yes, it'll try to bind to 49152. If something else is
		       already using that port, glusterd will try 49153, and
		       so on, until it finds a port that isn't in use
15:02 < kkeithley_wfh> we are already well behaved
15:04 < ndevos> ah, thats pretty cool :)
15:18 < kkeithley_wfh> ndevos: http://review.gluster.org/6147

Comment 1 Niels de Vos 2013-10-27 13:44:36 UTC

Verification that the patch works:

1. create and start a volume:
[root@vm130-31 tmp]# gluster volume create somevol replica 2 vm130-31:/bricks/somevol/data vm130-32:/bricks/somevol/data 
volume create: somevol: success: please start the volume to access data
[root@vm130-31 tmp]# gluster volume start somevol 
volume start: somevol: success

2. check the ports used (default is 49152)

[root@vm130-31 tmp]# gluster volume status
Status of volume: somevol
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick vm130-31:/bricks/somevol/data			49152	Y	1118
Brick vm130-32:/bricks/somevol/data			49152	Y	26308
NFS Server on localhost					2049	Y	1128
Self-heal Daemon on localhost				N/A	Y	1135
NFS Server on 192.168.130.32				2049	Y	26320
Self-heal Daemon on 192.168.130.32			N/A	Y	26324
 
There are no active volume tasks

3. uncomment and change the base-port option (50152):

[root@vm130-31 tmp]# vi /etc/glusterfs/glusterd.vol

4. restart the processes

[root@vm130-31 tmp]# systemctl restart glusterfsd
[root@vm130-31 tmp]# systemctl restart glusterd

5. check if the port has been modified

[root@vm130-31 tmp]# gluster volume status
Status of volume: somevol
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick vm130-31:/bricks/somevol/data			50152	Y	1204
Brick vm130-32:/bricks/somevol/data			49152	Y	26308
NFS Server on localhost					2049	Y	1212
Self-heal Daemon on localhost				N/A	Y	1219
NFS Server on 192.168.130.32				2049	Y	26320
Self-heal Daemon on 192.168.130.32			N/A	Y	26324
 
There are no active volume tasks

-> VERIFIED, vm130-31 uses port 50152 after updating base-port in glusterd.vol

Comment 2 Fedora Update System 2013-10-27 14:32:47 UTC

glusterfs-3.4.1-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/glusterfs-3.4.1-3.fc18

Comment 3 Fedora Update System 2013-10-27 14:33:08 UTC

glusterfs-3.4.1-3.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/glusterfs-3.4.1-3.fc19

Comment 4 Fedora Update System 2013-10-27 14:33:28 UTC

glusterfs-3.4.1-3.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/glusterfs-3.4.1-3.fc20

Comment 5 Fedora Update System 2013-10-27 18:11:41 UTC

Package glusterfs-3.4.1-3.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing glusterfs-3.4.1-3.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-20091/glusterfs-3.4.1-3.fc20
then log in and leave karma (feedback).

Comment 6 Fedora Update System 2013-12-14 03:05:25 UTC

glusterfs-3.4.1-3.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 7 Fedora Update System 2013-12-15 03:33:57 UTC

glusterfs-3.4.1-3.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 8 Fedora Update System 2013-12-15 03:38:29 UTC

glusterfs-3.4.1-3.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.