Bug 1313507 - live-migration uses port range from ephemeral port range
Summary: live-migration uses port range from ephemeral port range
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 7.0 (Kilo)
Hardware: x86_64
OS: Linux
low
low
Target Milestone: ---
: ---
Assignee: Emilien Macchi
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks: 1573787 1573791 1573793 1573796
TreeView+ depends on / blocked
 
Reported: 2016-03-01 17:39 UTC by Sean Lee
Modified: 2019-10-10 11:29 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1573787 1573791 1573793 1573796 (view as bug list)
Environment:
Last Closed: 2018-05-18 14:33:49 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 561784 None MERGED Modify libvirt port range for live-migration 2020-04-17 14:02:06 UTC

Description Sean Lee 2016-03-01 17:39:36 UTC
By default, libvirtd uses ports 49152-49215 for live migration, as specified in qemu.conf:

#migration_port_min = 49152
#migration_port_max = 49215

However, these ports can also be randomly consumed by nova-api-metadata (or any service that accepts incoming connections), e.g.

nova-api-metada 15040 nova  9u IPv4  75481 0t0 TCP node1.example.com:49162->node2.example.com:amqp (ESTABLISHED)
nova-api-metada 15040 nova 10u IPv4 178462 0t0 TCP node1.example.com:49163->node2.example.com:amqp (ESTABLISHED)
nova-api-metada 15045 nova  9u IPv4 179423 0t0 TCP node1.example.com:49160->node2.example.com:amqp (ESTABLISHED)
nova-api-metada 15045 nova 10u IPv4 178458 0t0 TCP node1.example.com:49161->node2.example.com:amqp (ESTABLISHED)

The odd of nova-api-metadata using all the 64 ports is very low, but it does happen, and live-migration would fail with the following message:

Live Migration failure: internal error: Unable to find an unused port in range 'migration' (49152-49215)

Comment 1 Daniel Berrangé 2016-03-01 17:43:43 UTC
Changing the default range in libvirt itself is not an option. We have to expect that existing RHEL users will have configured their firewalls based on the existing port range. So if we changed it in libvirt, upgrades would cause a regression for existing RHEL users.

The only viable option is to have osp-director configure /etc/libvirt/qemu.conf to set a custom migration port range when deploying opentstack nova compute nodes. This will only affect new deployments, so minimises chance of regression for existing users

Comment 3 Mike Burns 2016-04-07 21:11:06 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 4 Sven Anderson 2016-10-14 15:06:28 UTC
@slee: where and in which circumstances did this occur? Is this an actual environment? Thr problem is, that the ports can be taken by any _outbound_ connection as a source port, because these are in the ephemeral port range which is from 32768 to 61000 on linux by default. The only way to avoid that for sure is to move the migration ports out of that range. The risk could be reduced by expanding the range.


Note You need to log in before you can comment on or make changes to this bug.