Bug 1313507

Summary: live-migration uses port range from ephemeral port range
Product: Red Hat OpenStack Reporter: Sean Lee <slee>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED WONTFIX QA Contact: Gurenko Alex <agurenko>
Severity: low Docs Contact:
Priority: low    
Version: 7.0 (Kilo)CC: berrange, dbecker, jcoufal, jschluet, kchamart, mbooth, mburns, morazi, owalsh, ratailor, rbalakri, rhel-osp-director-maint, sgordon, slee, svanders
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1573787 1573791 1573793 1573796 (view as bug list) Environment:
Last Closed: 2018-05-18 14:33:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1573787, 1573791, 1573793, 1573796    

Description Sean Lee 2016-03-01 17:39:36 UTC 
By default, libvirtd uses ports 49152-49215 for live migration, as specified in qemu.conf:

#migration_port_min = 49152
#migration_port_max = 49215

However, these ports can also be randomly consumed by nova-api-metadata (or any service that accepts incoming connections), e.g.

nova-api-metada 15040 nova  9u IPv4  75481 0t0 TCP node1.example.com:49162->node2.example.com:amqp (ESTABLISHED)
nova-api-metada 15040 nova 10u IPv4 178462 0t0 TCP node1.example.com:49163->node2.example.com:amqp (ESTABLISHED)
nova-api-metada 15045 nova  9u IPv4 179423 0t0 TCP node1.example.com:49160->node2.example.com:amqp (ESTABLISHED)
nova-api-metada 15045 nova 10u IPv4 178458 0t0 TCP node1.example.com:49161->node2.example.com:amqp (ESTABLISHED)

The odd of nova-api-metadata using all the 64 ports is very low, but it does happen, and live-migration would fail with the following message:

Live Migration failure: internal error: Unable to find an unused port in range 'migration' (49152-49215)
Comment 1 Daniel Berrangé 2016-03-01 17:43:43 UTC 
Changing the default range in libvirt itself is not an option. We have to expect that existing RHEL users will have configured their firewalls based on the existing port range. So if we changed it in libvirt, upgrades would cause a regression for existing RHEL users.

The only viable option is to have osp-director configure /etc/libvirt/qemu.conf to set a custom migration port range when deploying opentstack nova compute nodes. This will only affect new deployments, so minimises chance of regression for existing users
Comment 3 Mike Burns 2016-04-07 21:11:06 UTC 
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 4 Sven Anderson 2016-10-14 15:06:28 UTC 
@slee: where and in which circumstances did this occur? Is this an actual environment? Thr problem is, that the ports can be taken by any _outbound_ connection as a source port, because these are in the ephemeral port range which is from 32768 to 61000 on linux by default. The only way to avoid that for sure is to move the migration ports out of that range. The risk could be reduced by expanding the range.