Bug 1159245

Summary:	repeated migration with NBD fails
Product:	Red Hat Enterprise Linux 7	Reporter:	Ján Tomko <jtomko>
Component:	libvirt	Assignee:	Ján Tomko <jtomko>
Status:	CLOSED ERRATA	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.1	CC:	dyuan, jiahu, mzhan, rbalakri, shyu, zhwang, zpeng
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-1.2.8-6.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-03-05 07:46:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ján Tomko 2014-10-31 09:55:56 UTC

Description of problem:
After migration with NBD, libvirt doesn't mark the TCP port used by the NBD server as free in its internal data. After repeated migration with NBD, all the ports are marked as used and libvirtd cannot migrate until it's restarted.

Version-Release number of selected component (if applicable):
libvirt-1.2.8-5.el7.x86_64

Steps to Reproduce:
Repeatedly (~64 times) migrate with --copy-storage-all

Actual results:
After exhaustion of the migration_port range (default is 64 ports starting from 49152), migration fails with: 
Unable to find an unused port in range 'migration' (49152-49215)

Expected results:
Migration still works.

Additional info:
Weiwei Li sent a patch upstream:
https://www.redhat.com/archives/libvir-list/2014-October/msg01097.html

Comment 1 Ján Tomko 2014-10-31 11:32:04 UTC

Now pushed upstream:
commit be598c5ff84656d3498b950d473fafe5b86f87b4
Author:     weiwei li <weiweili821>
AuthorDate: 2014-10-31 16:16:22 +0800
Commit:     Ján Tomko <jtomko>
CommitDate: 2014-10-31 12:20:06 +0100

    qemu: Release nbd port from migrationPorts instead of remotePorts
    
    commit 3e1e16aa8d4238241a1806cb9bdb3b9ad60db777 (Use a port from the
    migration range for NBD as well) changed ndb port allocation from
    remotePorts to migrationPorts, but did not change the port releasing
    process, which makes an error when migrating several times (above 64):
    error: internal error: Unable to find an unused port in range
    'migration' (49152-49215)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1159245
    
    Signed-off-by: Weiwei Li <nuonuoli>
    Signed-off-by: Ján Tomko <jtomko>

git describe: v1.2.10-rc2-1-gbe598c5

Comment 4 zhenfeng wang 2014-12-05 03:02:09 UTC

I can reproduce this bug with libvirt-1.2.8-5.el7.x86_64, the reproduce steps as following
1.start a guest on the source host
# virsh list
 Id    Name                           State
----------------------------------------------------
 65    rhel7.0                        running

#virsh dumpxml rhel7.0
--
<disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source dev='/var/lib/libvirt/images/test.img'/>
--

2.create a image a the target host which has the same name and in the same directory with the guest'img on the source host
#qemu-img create -f qcow2 /var/lib/libvirt/images/test.img 1G

3.Migrate the guest from source to target more than 64 times, the guest will fail to migrate to the target migrating more than 64 times
# for i in {1..80}; do virsh -c qemu:///system start rhel7.0; sleep 3; virsh migrate --live --copy-storage-all rhel7.0 qemu+ssh://$target_ip/system; sleep 3; virsh -c qemu+ssh://$target_ip/system destroy rhel7.0;done
--
Domain rhel7.0 started

Domain rhel7.0 destroyed
--------------------------------
Domain rhel7.0 started

error: internal error: Unable to find an unused port in range 'migration' (49152-49215)

4.Restart the libvirtd, the guest could migrate successfully

Verify the bug with the libvirt-1.2.8-10.el7, The steps were the same with the reproduce steps and the guest could still be migrated successfully, even we migrate more 64 times

Comment 6 errata-xmlrpc 2015-03-05 07:46:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html