Bug 1568939

Summary: Libvirt-resolved migration address has to be less than 64 symbols
Product: [Community] Virtualization Tools Reporter: drokvlad
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED NOTABUG QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: berrange, libvirt-maint, rbalakri
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-18 12:38:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description drokvlad 2018-04-18 12:34:23 UTC 
Description of problem:

When trying to live migrate an instance and not providing explicit migrate_uri in migration parameters, and if destination hypervisor fqdn is longer than 64 symbols, the following error happens:

Apr 12 10:55:22 cmp1 libvirtd[4133]: 2018-04-12 10:55:22.133+0000: 4139: error : qemuMonitorJSONCheckError:392 : internal error: unable to execute QEMU command 'drive-mirror': error parsing address 'cmp0.sandriichenko-deploy-heat-virtual-mcp-pike-ovs-76.bud-mk.local:49153'

Version-Release number of selected component (if applicable):

libvirt 4.0.0

How reproducible:

When fqdn of the destination compute is longer than 64 symbols.

It was reproduced on openstack pike release installation.

Steps to Reproduce:
1. Boot an instance through openstack nova.
2. Request instance migration on a compute host with fqdn longer than 64 symbols, destination compute config file should not contain the following option https://github.com/openstack/nova/blob/stable/pike/nova/conf/libvirt.py#L239-L259 -- "openstack server migrate 9153f1e9-18bd-46c3-a55f-c8da174cc94b --block-migration --live cmp1"

Actual results:

migration fails

Apr 12 10:55:22 cmp1 libvirtd[4133]: 2018-04-12 10:55:22.133+0000: 4139: error : qemuMonitorJSONCheckError:392 : internal error: unable to execute QEMU command 'drive-mirror': error parsing address 'cmp0.sandriichenko-deploy-heat-virtual-mcp-pike-ovs-76.bud-mk.local:49153'
Apr 12 10:55:22 cmp1 bagpipe-bgp[5533]: 127.0.0.1 - - [12/Apr/2018 10:55:22] "GET /ping HTTP/1.1" 200 39
Apr 12 10:55:22 cmp1 libvirtd[4133]: 2018-04-12 10:55:22.492+0000: 4139: error : virNetClientProgramDispatchError:177 : migration successfully aborted
Apr 12 10:55:22 cmp1 nova-compute[23186]: Traceback (most recent call last):
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
Apr 12 10:55:22 cmp1 nova-compute[23186]:     listener.cb(fileno)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
Apr 12 10:55:22 cmp1 nova-compute[23186]:     result = function(*args, **kwargs)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 1004, in context_wrapper
Apr 12 10:55:22 cmp1 nova-compute[23186]:     return func(*args, **kwargs)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6493, in _live_migration_operation
Apr 12 10:55:22 cmp1 nova-compute[23186]:     LOG.error("Live Migration failure: %s", e, instance=instance)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 12 10:55:22 cmp1 nova-compute[23186]:     self.force_reraise()
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 12 10:55:22 cmp1 nova-compute[23186]:     six.reraise(self.type_, self.value, self.tb)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6487, in _live_migration_operation
Apr 12 10:55:22 cmp1 nova-compute[23186]:     bandwidth=CONF.libvirt.live_migration_bandwidth)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 670, in migrate
Apr 12 10:55:22 cmp1 nova-compute[23186]:     destination, params=params, flags=flags)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
Apr 12 10:55:22 cmp1 nova-compute[23186]:     result = proxy_call(self._autowrap, f, *args, **kwargs)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
Apr 12 10:55:22 cmp1 nova-compute[23186]:     rv = execute(f, *args, **kwargs)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
Apr 12 10:55:22 cmp1 nova-compute[23186]:     six.reraise(c, e, tb)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
Apr 12 10:55:22 cmp1 nova-compute[23186]:     rv = meth(*args, **kwargs)
Apr 12 10:55:22 cmp1 nova-compute[23186]:   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1702, in migrateToURI3
Apr 12 10:55:22 cmp1 nova-compute[23186]:     if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
Apr 12 10:55:22 cmp1 nova-compute[23186]: libvirtError: internal error: unable to execute QEMU command 'drive-mirror': error parsing address 'cmp0.sandriichenko-deploy-heat-virtual-mcp-pike-ovs-76.bud-mk.local:49153'

Expected results:

migration passes

Additional info:

As the option in step 2 is not present, this code https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/driver.py#L6454-L6463 leaves migrate_uri=None, then it gets passed furhter to https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/driver.py#L6502-L6507 and to libvirt through libvirt-python https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/guest.py#L664-L670.

Then it seems that libvirt resolves the destination compute fqdn and then tries to validate that the fqdn is shorted than 64 symbols. If the migration_uri is set explicitly to hosname (not fqdn) of destination compute, migration happens fine.
Comment 1 Daniel Berrangé 2018-04-18 12:38:04 UTC 
This error message is actually coming from QEMU, libvirt is just the messenger here.

QEMU's inet_parse method has horrible code that uses a fixed length buffer of 64 bytes together with scanf which you're hitting here. So recommend filing a bug against QEMU for this: https://www.qemu.org/contribute/report-a-bug/