971485 – Live migration fails if "vnc_listen" set in /etc/libvirt/qemu.conf

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 971485 - Live migration fails if "vnc_listen" set in /etc/libvirt/qemu.conf

Summary: Live migration fails if "vnc_listen" set in /etc/libvirt/qemu.conf

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	6.4
Hardware:	All
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Michal Privoznik
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2013-06-06 16:01 UTC by Roman
Modified:	2014-06-14 11:26 UTC (History)
CC List:	7 users (show)
Fixed In Version:	libvirt-0.10.2-20.el6
Doc Type:	Bug Fix
Doc Text:	Cause: If user specified no vnc listen address in their domain XML, the one from qemu.conf is used. However, uppon migrating, there was no difference between cases where the listen address was set by user in XML directly or copied from qemu.conf. Consequence: A domain couldn't be migrated. Fix: If the listen address is copied from qemu.conf, it is not transferred to the destination (which probably can't bind() to the old listen address anyway). Result: A domain can be migrated without any error.
Clone Of:
Environment:
Last Closed:	2013-11-21 09:02:25 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2013:1581	0	normal	SHIPPED_LIVE	libvirt bug fix and enhancement update	2013-11-21 01:11:35 UTC

Description Roman 2013-06-06 16:01:38 UTC

Description of problem:
Live migration fails if "vnc_listen" set in /etc/libvirt/qemu.conf

Version-Release number of selected component (if applicable):
libvirt-0.10.2-18.el6_4.5.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Setup vnc_listen option /etc/libvirt/qemu.conf to any ip.
2. Restart libvirtd, start any VM
3. Start live migration:
# virsh migrate --timeout 300 --live --verbose <VM> qemu+tcp://other-host/system

Actual results:
error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
inet_listen_opts: bind(ipv4,<IP>,5900): Cannot assign requested address
inet_listen_opts: FAILED
Failed to start VNC server on `<IP>:0'

Expected results:
migration completed

Additional info:

Comment 2 Roman 2013-06-07 16:14:23 UTC

Patch that breaks live migration:
http://libvirt.org/git/?p=libvirt.git;a=commit;h=d7f9d827531bc843b7c5aa9d3e8c08738a1de248

Comment 3 Dave Allan 2013-06-07 16:22:10 UTC

Roman, thanks for the bisect; that makes sense.  Michal, can you see what can be done about this?

Comment 4 Roman 2013-06-08 21:56:23 UTC

Simple workaround using libvirt hooks:

# cat /etc/libvirt/hooks/qemu-filter.xsl 
<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output omit-xml-declaration="yes"/>
  <xsl:template match="/domain/devices/graphics/listen"/>
  <xsl:template match="/domain/devices/graphics/@listen"/>
  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

# cat /etc/libvirt/hooks/qemu
#!/bin/bash

[ "$2" = "migrate" ] && xsltproc /etc/libvirt/hooks/qemu-filter.xsl -

exit 0

Comment 5 Michal Privoznik 2013-06-10 17:39:00 UTC

Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2013-June/msg00456.html

Comment 6 Michal Privoznik 2013-06-11 12:58:20 UTC

I've just pushed the patch upstream:

commit 6546017c50c104d0b9867137b64ab1f4a312e436
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Jun 10 17:51:46 2013 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue Jun 11 14:11:46 2013 +0200

    qemu_migrate: Dispose listen address if set from config
    
    https://bugzilla.redhat.com/show_bug.cgi?id=971485
    
    As of d7f9d827531bc843b7c5aa9d3e8c08738a1de248 we copy the listen
    address from the qemu.conf config file in case none has been provided
    via XML. But later, when migrating, we should not include such listen
    address in the migratable XML as it is something autogenerated, not
    requested by user. Moreover, the binding to the listen address will
    likely fail, unless the address is '0.0.0.0' or its IPv6 equivalent.
    This patch introduces a new boolean attribute to virDomainGraphicsListenDef
    to distinguish autofilled listen addresses. However, we must keep the
    attribute over libvirtd restarts, so it must be kept within status XML.

v1.0.6-61-g6546017

Comment 7 Michal Privoznik 2013-07-04 14:36:56 UTC

Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2013-July/msg00065.html

Comment 9 yanbing du 2013-07-16 08:23:47 UTC

# rpm -q libvirt
libvirt-0.10.2-20.el6.x86_64

Following the steps in comment0, when start migration, libvirtd on dest host will crash, which should cause by bug 984793.

Comment 10 yanbing du 2013-07-23 06:29:24 UTC

1. Set vnc_listen="10.66.106.30" in /etc/libvirt/qemu.conf
2. Start a guest
# virsh dumpxml spice
...
    <graphics type='vnc' port='5900' autoport='yes' listen='10.66.106.30'>
      <listen type='address' address='10.66.106.30'/>
    </graphics>
...

3. Do live migration
3.1 With libvirt-0.10.2-18.el6_4.5, when do migration:
# virsh migrate --timeout 300 --live --verbose spice qemu+tcp://10.66.106.31/system
error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/2
inet_listen_opts: bind(ipv4,10.66.106.30,5900): Cannot assign requested address
inet_listen_opts: FAILED
Failed to start VNC server on `10.66.106.30:0'

3.2. With libvirt-0.10.2-21.el6, when do migration:
# virsh migrate --timeout 300 --live --verbose spice qemu+tcp://10.66.106.31/system
Migration: [100 %]

and on the dest host:
# virsh list
 Id    Name                           State
----------------------------------------------------
 1     spice                          running

as vnc_listen="10.66.106.31 on the dest host qemu.conf, the guest listen address also changed
# virsh dumpxml spice
...
    <graphics type='vnc' port='5900' autoport='yes' listen='10.66.106.31'>
      <listen type='address' address='10.66.106.31'/>
    </graphics>
...

So move this bug to VERIFIED.

Comment 12 errata-xmlrpc 2013-11-21 09:02:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1581.html

Comment 13 pasquale.demichele 2014-06-14 11:26:33 UTC

(In reply to errata-xmlrpc from comment #12)
> Since the problem described in this bug report should be
> resolved in a recent advisory, it has been closed with a
> resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated
> files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> http://rhn.redhat.com/errata/RHBA-2013-1581.html

Hi all.

I have the same issue with this packages:

$ rpm -qa | grep libvirt

libvirt-python-0.10.2-29.el6_5.3.x86_64
libvirt-client-0.10.2-29.el6_5.3.x86_64
libvirt-0.10.2-29.el6_5.3.x86_64

OS version is the same, CentOS 6.4.

The cloud system is OpenNebula (4.4) and the migration fails also when the listen_vnc (different from 0.0.0.0) is setted directly in the template.

Note You need to log in before you can comment on or make changes to this bug.