Bug 1018178 - Glusterfs ports conflict with qemu live migration
Glusterfs ports conflict with qemu live migration
Product: GlusterFS
Classification: Community
Component: transport (Show other bugs)
Unspecified Unspecified
urgent Severity medium
: ---
: ---
Assigned To: Kaleb KEITHLEY
: Reopened
: 1196678 (view as bug list)
Depends On: 987555 1018695 1019058 1019237 1340368
Blocks: 1018383
  Show dependency treegraph
Reported: 2013-10-11 07:32 EDT by Kaleb KEITHLEY
Modified: 2016-06-20 13:20 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Prior to this update, migrating a virtual machine failed when the libvirtd service used a transmission control protocol (TCP) port that was already in use. Now, it is possible to predefine a custom migration TCP port range in case the default port is in use. In addition, libvirtd now ensures that the port it chooses from the custom range is not used by another process.
Story Points: ---
Clone Of: 987555
Last Closed: 2013-11-06 07:06:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Kaleb KEITHLEY 2013-10-11 07:32:52 EDT
+++ This bug was initially created as a clone of Bug #987555 +++

Description of problem:
Starting with GlusterFS 3.4, glusterfsd uses the IANA defined ephemeral port range (49152 and upward). If you happen to use the same network for storage and qemu-kvm live migration, sometimes you get a port conflict, and live migration aborts

Here's a log of a failed live migration on the destination host:

2013-07-23 15:54:32.619+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name ipasserelle QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name ipasserelle -S -M-M rhel6.4.0  -enable-kvm -m 20482048 -smp-smp 2,sockets=2,cores=1,threads=1  -uuid  8505958b-8227-0a46-91a7-41d3247544e2 -nodefconfig-nodefconfig -nodefaults  -chardev  socket,id=charmonitor,path=/var/lib/libvirt/qemu/ipasserelle.monitor,server,nowait -mon-mon chardev=charmonitor,id=monitor,mode=controlchardev=charmonitor,id=monitor,mode=control -rtc  base=utc  -no-shutdown -device  piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/gluster/ipasserelle.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=nonefile=/var/lib/libvirt/images/gluster/ipasserelle.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device  virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=rawif=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device  ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev-netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23  -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:2b:14:d7,bus=pci.0,addr=0x3 -netdev-netdev tap,fd=24,id=hostnet1,vhost=on,vhostfd=25 -device  virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:6d:4f:52,bus=pci.0,addr=0x4  -chardev pty,id=charserial0pty,id=charserial0 -device-device isa-serial,chardev=charserial0,id=serial0  -device usb-tablet,id=input0 -vnc-vnc -vga  cirrus  -device intel-hda,id=sound0,bus=pci.0,addr=0x5  -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -incoming  tcp:[::]:49152 -device-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
char device redirected to /dev/pts/2
inet_listen_opts: bind(ipv6,::,49152): Address already in use
inet_listen_opts: FAILED
Migrate: Failed to bind socket
Migration failed. Exit code tcp:[::]:49152(-1), exiting.
2013-07-23 15:54:33.016+0000: shutting down

[root@dd9 ~]# netstat -laputen | grep :49152
tcp        0      0     *                   LISTEN      0          82349      1927/glusterfsd     
tcp        0      0                 ESTABLISHED 0          82555      1952/glusterfs      
tcp        0      0           ESTABLISHED 0          82473      1927/glusterfsd     
tcp        0      0           ESTABLISHED 0          82344      1952/glusterfs      
tcp        0      0                 ESTABLISHED 0          82725      1927/glusterfsd     
tcp        0      0                 ESTABLISHED 0          82556      1927/glusterfsd     
tcp        0      0           ESTABLISHED 0          89092      1927/glusterfsd     
tcp        0      0                 ESTABLISHED 0          82724      2069/glusterfs      
tcp        0      0           ESTABLISHED 0          82784      2115/glusterfs

The exact same setup with GlusterFS 3.3.2 is working like a charm

Version-Release number of selected component (if applicable):

Host is CentOS 6.4 x86_64

gluster 3.4.0-2 (glusterfs glusterfs-server glusterfs-fuse), from the gluster.org RHEL repo
libvirt 0.10.2-18

How reproducible:

Not always, but frequently enough

Steps to Reproduce:

- Two hosts with a replicated glusterFS volume (both are gluster server and client)
- Libvirt on both nodes
- One private network used for gluster and live migration
- while glusterFS is working, try to live migrate a qemu-kvm VM, using the standard migration (virsh migrate --live vm qemu+ssh://user@other_node/system)
- From time to time (not always), the migration will fail because the qemu process on the destination host cannot bind to the choosed port

Actual results:
Live migration fails

Expected results:
Live migration shouldn't be bothered by Gluster

Additional info:
An option to configure the first port, or the port range used by Gluster would avoid this situation

--- Additional comment from Daniel on 2013-07-24 05:41:31 EDT ---

Just one more info: I have three GlusterFS volumes between the two nodes, and the first three migrations fail.

As qemu (or libvirt, not sure which one chooses the incomming migration port) increment the port number at each migration attempt, the fourth migration succeed (and the following migrations succeed too)

--- Additional comment from Caerie Houchins on 2013-10-02 17:18:14 EDT ---

We just hit this bug in a new setup today.  Verifying this still exists.  
CentOS release 6.4 (Final)
Linux SERVERNAME 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28 17:19:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

--- Additional comment from Gianluca Cecchi on 2013-10-09 18:52:29 EDT ---

Same problem with oVirt 3.3 and fedora 19 as Hypervisors.
See here:
the range 49152-49215 already used by libvirt years before the Gluster change from 3.3 to 3.4...
How could you miss it and worse not able to change at least for 3.4.1 as this bugzilla was opened in July?

At least could you provide a way to configure gluster to use another range so that if two nodes are both servers and client they can use another range?

You are limiting GlusterFS adoption itself as noone would implement oVirt on GlusterFS without migration available...

Thanks for reading
Comment 1 Kaleb KEITHLEY 2013-10-11 07:54:29 EDT
Out of curiosity, why isn't this a bug in qemu-kvm? Shouldn't qemu-kvm be trying another port if 49152 (or any other port) is in use? And using portmapper to register the port it does end up using?
Comment 2 Anand Avati 2013-10-11 07:55:01 EDT
REVIEW: http://review.gluster.org/6075 (xlators/mgmt/glusterd: ports conflict with qemu live migration) posted (#1) for review on master by Kaleb KEITHLEY (kkeithle@redhat.com)
Comment 3 Gianluca Cecchi 2013-10-12 13:04:12 EDT
I confirm that taking source rpm provided by fedora 19 (glusterfs-3.4.1-1.fc19.src.rpm), changing sources as in comment#2 , rebuilding them and updating my gluster rpms on a Fedora 19 system in oVirt now I have

[g.cecchi@f18ovn01 ~]$ ps -ef|grep glusterfs
root      2131     1  0 18:44 ?        00:00:00 /usr/sbin/glusterfsd -s f18ovn01.mydomain --volfile-id gvdata.f18ovn01.mydomain.gluster-DATA_GLUSTER-brick1 -p /var/lib/glusterd/vols/gvdata/run/f18ovn01.mydomain-gluster-DATA_GLUSTER-brick1.pid -S /var/run/3f792df94642bdbf308ce4c368dc05bf.socket --brick-name /gluster/DATA_GLUSTER/brick1 -l /var/log/glusterfs/bricks/gluster-DATA_GLUSTER-brick1.log --xlator-option *-posix.glusterd-uuid=ebaf2f1a-65a8-409a-b911-6e631a5f182f --brick-port 50152 --xlator-option gvdata-server.listen-port=50152

[g.cecchi@f18ovn01 ~]$ sudo lsof -Pp 2131 | egrep "491|501"
glusterfs 2131 root    9u     IPv4              28933      0t0    TCP f18ovn01.mydomain:50152->f18ovn01.mydomain:1011 (ESTABLISHED)
glusterfs 2131 root   10u     IPv4              25260      0t0    TCP *:50152 (LISTEN)
glusterfs 2131 root   13u     IPv4             101260      0t0    TCP f18ovn01.mydomain:50152->f18ovn01.mydomain:1001 (ESTABLISHED)

And I'm able to successfully live migrate my CentOS 6.4 VM from one host to the other one.
I have one brick that is replicated ditributed between two hosts

[g.cecchi@f18ovn01 ~]$ sudo gluster volume info gvdata
Volume Name: gvdata
Type: Replicate
Volume ID: ed71a4c2-6205-4aad-9aab-85da086d5ba3
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: f18ovn01.mydomain:/gluster/DATA_GLUSTER/brick1
Brick2: f18ovn03.mydomain:/gluster/DATA_GLUSTER/brick1
Options Reconfigured:
server.allow-insecure: on
storage.owner-uid: 36
storage.owner-gid: 36

Hope to see something like a configurable range via configuration file too.

Comment 5 Kaleb KEITHLEY 2013-10-21 14:27:13 EDT
RFC 6335 is very clear about this:

   8.1.2. Variances for Specific Port Number Ranges
      o  Ports in the Dynamic Ports range (49152-65535) have been
         specifically set aside for local and dynamic use and cannot be
         assigned through IANA.  Application software may simply use any
         dynamic port that is available on the local host, without any sort
         of assignment.  On the other hand, application software MUST NOT
         assume that a specific port number in the Dynamic Ports range will
         always be available for communication at all times...

This is clearly a libvirt bug.  Libvirt needs to handle the case when a port is not available. It cannot assume that it can exclusively use ports starting at 49152.

See Red Hat bugs 1018530, 1018696, and 1019237.
Comment 8 Anand Avati 2013-10-31 08:27:15 EDT
REVIEW: http://review.gluster.org/6210 (mgmt/glusterd: add option to specify a different base-port) posted (#1) for review on master by Kaleb KEITHLEY (kkeithle@redhat.com)
Comment 9 Anand Avati 2013-10-31 12:31:35 EDT
COMMIT: http://review.gluster.org/6210 committed in master by Anand Avati (avati@redhat.com) 
commit c47408e896c9bcaf21e7f8956bdae85633f873e0
Author: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Date:   Thu Oct 31 08:18:56 2013 -0400

    mgmt/glusterd: add option to specify a different base-port
    This is (arguably) a hack to work around a bug in libvirt which is not
    well behaved wrt to using TCP ports in the unreserved space between
    49152-65535. (See RFC 6335)
    Normally glusterd starts and binds to the first available port in range,
    usually 49152. libvirt's live migration also tries to use ports in this
    range, but has no fallback to use (an)other port(s) when the one it wants
    is already in use.
    Change-Id: Id8fe35c08b6ce4f268d46804bbb6dddab7a6b7bb
    BUG: 1018178
    Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
    Reviewed-on: http://review.gluster.org/6210
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Anand Avati <avati@redhat.com>
Comment 10 Niels de Vos 2013-11-06 07:06:38 EST
In fact, closing with UPSTREAM. Bug 987555 is tracked for an update in release-3.4.
Comment 11 Michal Skrivanek 2015-03-23 08:56:11 EDT
*** Bug 1196678 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.