Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 889961

Summary:

VDSM | Libvirt fails the migration process when removing 1st boot device from running VM.

Product:

Red Hat Enterprise Linux 6

Reporter:

Meni Yakove <myakove>

Component:

libvirt

Assignee:

Peter Krempa <pkrempa>

Status:

CLOSED ERRATA

QA Contact:

Meni Yakove <myakove>

Severity:

medium

Docs Contact:

Priority:

unspecified

Version:

6.3

CC:

acathrow, cwei, dallan, dyuan, jdenemar, lnovich, mzhan, pkrempa, weizhan

Target Milestone:

Keywords:

Upstream

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

libvirt-0.10.2-19.el6

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2013-11-21 08:36:18 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
libvirtd.log	none
libvirtd.log	none
vdsm log	none
source libvirtd log	none
target libvirtd log	none

Description Meni Yakove 2012-12-24 07:16:44 UTC

Created attachment 668340 [details]
libvirtd.log

Description of problem:
Using RHEVM and VDSM.
When VM xml is missing <boot order='1'> libvirt failes the migration process


How reproducible:
Alyeas

Steps to Reproduce:
1. rhevm create VM and select boot from network.
2. While the VM is running delete the network interface from the VM (using rhevm webadmin)
3. From rhevm webadmin migrate the VM.
  
Actual results:
Migration failed.

Expected results:
Migration should pass.

Additional info:

Thread-200896::DEBUG::2012-12-24 09:05:43,576::libvirtvm::454::vm.Vm::(_startUnderlyingMigration) vmId=`22ee2cb6-d617-44ce-b650-d16607825589`::starting migration to qemu+tls://orchid-vds1.qa.lab.tlv.redhat.com/system
Thread-200899::DEBUG::2012-12-24 09:05:43,577::libvirtvm::352::vm.Vm::(run) vmId=`22ee2cb6-d617-44ce-b650-d16607825589`::migration downtime thread started
Thread-200900::DEBUG::2012-12-24 09:05:43,580::libvirtvm::380::vm.Vm::(run) vmId=`22ee2cb6-d617-44ce-b650-d16607825589`::starting migration monitor thread
Thread-200896::DEBUG::2012-12-24 09:05:43,850::libvirtvm::367::vm.Vm::(cancel) vmId=`22ee2cb6-d617-44ce-b650-d16607825589`::canceling migration downtime thread
Thread-200896::DEBUG::2012-12-24 09:05:43,851::libvirtvm::417::vm.Vm::(stop) vmId=`22ee2cb6-d617-44ce-b650-d16607825589`::stopping migration monitor thread
Thread-200899::DEBUG::2012-12-24 09:05:43,851::libvirtvm::364::vm.Vm::(run) vmId=`22ee2cb6-d617-44ce-b650-d16607825589`::migration downtime thread exiting
Thread-200896::ERROR::2012-12-24 09:05:43,851::vm::181::vm.Vm::(_recover) vmId=`22ee2cb6-d617-44ce-b650-d16607825589`::internal error boot orders have to be contiguous and starting from 1
Thread-200896::ERROR::2012-12-24 09:05:44,099::vm::262::vm.Vm::(run) vmId=`22ee2cb6-d617-44ce-b650-d16607825589`::Failed to migrate
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 245, in run
    self._startUnderlyingMigration()
  File "/usr/share/vdsm/libvirtvm.py", line 478, in _startUnderlyingMigration
    None, maxBandwidth)
  File "/usr/share/vdsm/libvirtvm.py", line 518, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1103, in migrateToURI2
    if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: internal error boot orders have to be contiguous and starting from 1

Comment 2 Huang Wenlong 2012-12-25 05:29:11 UTC

I can reproduce this issue with steps comment 0 
libvirt-0.9.10-21.el6_3.7.x86_64
qemu-kvm-rhev-0.12.1.2-2.295.el6_3.10.x86_64
vdsm-4.9.6-44.2.el6_3.x86_64


I will attach vdsm.log and libvirtd.log

Comment 3 Huang Wenlong 2012-12-25 05:30:47 UTC

Created attachment 668712 [details]
libvirtd.log

Comment 4 Huang Wenlong 2012-12-25 05:36:30 UTC

Created attachment 668716 [details]
vdsm log

Comment 5 RHEL Program Management 2012-12-28 06:47:52 UTC

This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 7 Peter Krempa 2013-04-12 12:44:51 UTC

Fix upstream in v1.0.4-132-g039a328:

commit 039a3283fcde23572cd93e9e09cfbfb508e5d44b
Author: Peter Krempa <pkrempa>
Date:   Thu Apr 4 15:29:16 2013 +0200

    conf: Allow for non-contiguous device boot orders
    
    This patch adds the ability to configure non-contiguous boot orders on boot
    devices. This allows unplugging devices that have boot order specified without
    breaking migration.
    
    The new code now uses a slightly less memory efficient approach to store the
    boot order fields in a hashtable instead of a bitmap.

Comment 17 weizhang 2013-07-11 04:16:56 UTC

Verify failed on
libvirt-0.10.2-19.el6.x86_64
kernel-2.6.32-396.el6.x86_64
qemu-kvm-0.12.1.2-2.377.el6.x86_64

Steps
1. start a guest with interface
    <interface type='network'>
      <mac address='52:54:00:b0:b9:18'/>
      <source network='default'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
2. restore above xml in interface.xml
3. do 
# virsh detach-device guest interface.xml
4. do migration
# virsh migrate --live guest qemu+ssh://xxx/system
error: operation failed: migration job: unexpectedly failed

Comment 18 weizhang 2013-07-11 04:18:17 UTC

Created attachment 771968 [details]
source libvirtd log

Comment 19 weizhang 2013-07-11 04:19:21 UTC

Created attachment 771969 [details]
target libvirtd log

Comment 20 Peter Krempa 2013-07-11 09:42:35 UTC

Qemu apparently crashed during your migration attempt.

2013-07-11 04:11:31.540+0000: 15035: error : qemuMonitorIO:613 : internal error End of file from monitor
2013-07-11 04:11:31.540+0000: 15035: debug : qemuMonitorIO:646 : Error on monitor internal error End of file from monitor

A more accurate error might be available in the machine log file. This problem shouldn't be related in any way to this bug.

Comment 21 weizhang 2013-07-11 13:17:05 UTC

I test with the second scenario, in the old version
libvirt-0.10.2-18.el6.x86_64 with steps

1. start a guest with
    <interface type='network'>
      <mac address='52:54:00:b4:62:42'/>
      <source network='default'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:b5:62:42'/>
      <source network='default'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>

2. detach the first network
# virsh detach-device guest interface.xml

3. Do migration
# virsh migrate test qemu+tcp://10.66.82.145/system
error: internal error boot orders have to be contiguous and starting from 1

But on version libvirt-0.10.2-19.el6.x86_64
libvirt will report new error
error: operation failed: migration job: unexpectedly failed

I think this bug is blocked by the new bug. I will report a new one and add block bug on this bug.

Comment 22 weizhang 2013-07-11 16:07:03 UTC

finally I find the reason. It is because I use the guest without os in it. when I use the guest with os and after successful detach of network interface 1, then migration can succeed. so verify pass

Comment 24 errata-xmlrpc 2013-11-21 08:36:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1581.html