Bug 1313507 - live-migration uses same ports as nova-api-metadata (or any service that accepts incoming connections) [NEEDINFO]
live-migration uses same ports as nova-api-metadata (or any service that acce...
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
x86_64 Linux
low Severity low
: ---
: 11.0 (Ocata)
Assigned To: Angus Thomas
Arik Chernetsky
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-01 12:39 EST by Sean Lee
Modified: 2017-09-22 21:54 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
svanders: needinfo? (slee)


Attachments (Terms of Use)

  None (edit)
Description Sean Lee 2016-03-01 12:39:36 EST
By default, libvirtd uses ports 49152-49215 for live migration, as specified in qemu.conf:

#migration_port_min = 49152
#migration_port_max = 49215

However, these ports can also be randomly consumed by nova-api-metadata (or any service that accepts incoming connections), e.g.

nova-api-metada 15040 nova  9u IPv4  75481 0t0 TCP node1.example.com:49162->node2.example.com:amqp (ESTABLISHED)
nova-api-metada 15040 nova 10u IPv4 178462 0t0 TCP node1.example.com:49163->node2.example.com:amqp (ESTABLISHED)
nova-api-metada 15045 nova  9u IPv4 179423 0t0 TCP node1.example.com:49160->node2.example.com:amqp (ESTABLISHED)
nova-api-metada 15045 nova 10u IPv4 178458 0t0 TCP node1.example.com:49161->node2.example.com:amqp (ESTABLISHED)

The odd of nova-api-metadata using all the 64 ports is very low, but it does happen, and live-migration would fail with the following message:

Live Migration failure: internal error: Unable to find an unused port in range 'migration' (49152-49215)
Comment 1 Daniel Berrange 2016-03-01 12:43:43 EST
Changing the default range in libvirt itself is not an option. We have to expect that existing RHEL users will have configured their firewalls based on the existing port range. So if we changed it in libvirt, upgrades would cause a regression for existing RHEL users.

The only viable option is to have osp-director configure /etc/libvirt/qemu.conf to set a custom migration port range when deploying opentstack nova compute nodes. This will only affect new deployments, so minimises chance of regression for existing users
Comment 3 Mike Burns 2016-04-07 17:11:06 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 4 Sven Anderson 2016-10-14 11:06:28 EDT
@slee: where and in which circumstances did this occur? Is this an actual environment? Thr problem is, that the ports can be taken by any _outbound_ connection as a source port, because these are in the ephemeral port range which is from 32768 to 61000 on linux by default. The only way to avoid that for sure is to move the migration ports out of that range. The risk could be reduced by expanding the range.

Note You need to log in before you can comment on or make changes to this bug.