Bug 1313341 - Failed to migration from RHEV-H to RHEL
Summary: Failed to migration from RHEV-H to RHEL
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhev-hypervisor
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ovirt-3.6.6
: ---
Assignee: Fabian Deutsch
QA Contact: Huijuan Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-01 12:32 UTC by Israel Pinto
Modified: 2016-03-04 07:37 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-03-04 07:37:05 UTC
oVirt Team: Node
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
logs (682.81 KB, application/zip)
2016-03-01 13:28 UTC, Israel Pinto
no flags Details

Description Israel Pinto 2016-03-01 12:32:54 UTC
Description of problem:
Migration VM from RHEV-H to RHEL failed
Checked also (on the same hosts):
RHEL to RHEV-H - failed
RHEV-H to RHEV-H - pass


Version-Release number of selected component (if applicable):
RHEVM: 3.6.3.4-0.1.el6
RHEV-H host:
OS Version:RHEV Hypervisor - 7.2 - 20160225.0.el7ev
Kernel Version:3.10.0 - 327.10.1.el7.x86_64
KVM Version:2.3.0 - 31.el7_2.7
LIBVIRT Version:libvirt-1.2.17-13.el7_2.3
VDSM Version:vdsm-4.17.23-0.el7ev
SPICE Version:0.12.4 - 15.el7
RHEL host:
OS Version:RHEL - 7.2 - 9.el7_2.1
Kernel Version:3.10.0 - 327.12.1.el7.x86_64
KVM Version:2.3.0 - 31.el7_2.8
LIBVIRT Version:libvirt-1.2.17-13.el7_2.3
VDSM Version:vdsm-4.17.23-0.el7ev
SPICE Version:0.12.4 - 15.el7


How reproducible:
All the time


Steps to Reproduce:
1. Create VM on RHEV-H
2. Migrate VM to RHEVM host

Actual results:
Migration failed

Additional info:
jsonrpc.Executor/1::DEBUG::2016-03-01 11:18:59,772::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.migrate' in bridge with {u'params': {u'tunneled': u'false', u'dstqemu': u'10.35.161.150', u'autoConverge': u'false', u'src': u'10.35.106.13', u'dst': u'10.35.161.150:54321', u'vmId': u'1eaaf517-6028-4d66-aa81-b908c20369f7', u'abortOnError': u'true', u'compressed': u'false', u'method': u'online'}, u'vmID': u'1eaaf517-6028-4d66-aa81-b908c20369f7'}
jsonrpc.Executor/1::DEBUG::2016-03-01 11:18:59,773::API::546::vds::(migrate) {u'tunneled': u'false', u'dstqemu': u'10.35.161.150', u'autoConverge': u'false', u'src': u'10.35.106.13', u'dst': u'10.35.161.150:54321', u'vmId': u'1eaaf517-6028-4d66-aa81-b908c20369f7', u'abortOnError': u'true', u'compressed': u'false', u'method': u'online'}
jsonrpc.Executor/1::DEBUG::2016-03-01 11:18:59,775::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4301232530, "1eaaf517-6028-4d66-aa81-b908c20369f7": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|1eaaf517-6028-4d66-aa81-b908c20369f7"}
jsonrpc.Executor/1::DEBUG::2016-03-01 11:18:59,776::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'VM.migrate' in bridge with True
Thread-62::DEBUG::2016-03-01 11:18:59,800::migration::147::virt.vm::(_setupVdsConnection) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Initiating connection with destination
Thread-62::DEBUG::2016-03-01 11:18:59,801::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response
JsonRpc (StompReactor)::DEBUG::2016-03-01 11:18:59,802::stomp::477::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established
Thread-62::DEBUG::2016-03-01 11:18:59,805::migration::159::virt.vm::(_setupVdsConnection) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Destination server is: 10.35.161.150:54321
Thread-62::DEBUG::2016-03-01 11:18:59,808::migration::202::virt.vm::(_prepareGuest) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Migration started
Thread-62::DEBUG::2016-03-01 11:18:59,829::migration::287::virt.vm::(run) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::migration semaphore acquired after 0 seconds
Thread-62::DEBUG::2016-03-01 11:18:59,854::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response
Thread-62::INFO::2016-03-01 11:18:59,925::migration::334::virt.vm::(_startUnderlyingMigration) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Creation of destination VM took: 0 seconds
Thread-62::INFO::2016-03-01 11:18:59,926::migration::353::virt.vm::(_startUnderlyingMigration) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::starting migration to qemu+tls://10.35.161.150/system with miguri tcp://10.35.161.150
Thread-63::DEBUG::2016-03-01 11:18:59,926::migration::442::virt.vm::(run) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::migration downtime thread started (10 steps)
Thread-63::DEBUG::2016-03-01 11:18:59,927::migration::465::virt.vm::(_set_downtime) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::setting migration downtime to 51
Thread-64::DEBUG::2016-03-01 11:18:59,927::migration::499::virt.vm::(monitor_migration) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::starting migration monitor thread
Thread-62::DEBUG::2016-03-01 11:19:00,952::migration::557::virt.vm::(stop) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::stopping migration monitor thread
Thread-62::DEBUG::2016-03-01 11:19:00,953::migration::452::virt.vm::(stop) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::stopping migration downtime thread
Thread-62::ERROR::2016-03-01 11:19:00,953::migration::208::virt.vm::(_recover) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::internal error: early end of file from monitor: possible problem:
2016-03-01T11:19:00.479958Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2016-03-01T11:19:00.480245Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
2016-03-01T11:19:00.715359Z qemu-kvm: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x20000 in != 0x40000: Invalid argument
2016-03-01T11:19:00.715385Z qemu-kvm: error while loading state for instance 0x0 of device 'ram'
2016-03-01T11:19:00.715478Z qemu-kvm: load of migration failed: Invalid argument
Thread-62::DEBUG::2016-03-01 11:19:00,954::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response
Thread-63::DEBUG::2016-03-01 11:19:00,954::migration::449::virt.vm::(run) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::migration downtime thread exiting
Thread-62::DEBUG::2016-03-01 11:19:00,989::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4301233740, "1eaaf517-6028-4d66-aa81-b908c20369f7": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|1eaaf517-6028-4d66-aa81-b908c20369f7"}
Thread-62::ERROR::2016-03-01 11:19:00,990::migration::309::virt.vm::(run) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Failed to migrate
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/migration.py", line 297, in run
    self._startUnderlyingMigration(time.time())
  File "/usr/share/vdsm/virt/migration.py", line 363, in _startUnderlyingMigration
    self._perform_migration(duri, muri)
  File "/usr/share/vdsm/virt/migration.py", line 402, in _perform_migration
    self._vm._dom.migrateToURI3(duri, params, flags)
  File "/usr/share/vdsm/virt/virdomain.py", line 68, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in migrateToURI3
    if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
libvirtError: internal error: early end of file from monitor: possible problem:
2016-03-01T11:19:00.479958Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2016-03-01T11:19:00.480245Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
2016-03-01T11:19:00.715359Z qemu-kvm: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x20000 in != 0x40000: Invalid argument
2016-03-01T11:19:00.715385Z qemu-kvm: error while loading state for instance 0x0 of device 'ram'
2016-03-01T11:19:00.715478Z qemu-kvm: load of migration failed: Invalid argument
jsonrpc.Executor/2::DEBUG::2016-03-01 11:19:04,196::task::595::Storage.TaskManager.Task::(_updateState) Task=`965c1dc8-13a9-4854-be63-377e8ca6540d`::moving from state init -> state preparing
jsonrpc.Executor/2::INFO::2016-03-01 11:19:04,196::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None)

Comment 1 Israel Pinto 2016-03-01 13:28:22 UTC
Created attachment 1131938 [details]
logs

Comment 2 Yaniv Kaul 2016-03-01 13:43:32 UTC
Re-appearance of bug 1293566? Can you check relevant versions of the *pxe* packages?

Comment 3 Israel Pinto 2016-03-01 14:16:39 UTC
The RHEV-H and RHEL are installed from QE foreman.
The ipxe version on the rhev-h is:
ipxe-roms-qemu-20130517-7.1fm.gitc4bce43.el7sat.noarch
we have diff version then the rhel:
ipxe-roms-qemu-20130517-8.gitc4bce43.el7_2.1.noarch
It looks like the same problem in BZ:1302742, 
Installation problem,
closing the bug

*** This bug has been marked as a duplicate of bug 1302742 ***

Comment 4 Fabian Deutsch 2016-03-02 09:10:10 UTC
Re-opening this bug and moving it to Node

Comment 7 cshao 2016-03-02 10:49:29 UTC
RHEV-H QE can reproduce this bug.

Test version:
RHEV-M: rhevm-3.6.3.4-0.1.el6

RHEV-H:
RHEV Hypervisor - 7.2 - 20160225.0.el7ev
kernel-3.10.0-327.10.1.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.7.x86_64
qemu-kvm-common-rhev-2.3.0-31.el7_2.7.x86_64
libvirt-1.2.17-13.el7_2.3.x86_64
vdsm-4.17.23-0.el7ev.noarch
spice-server-0.12.4-15.el7.x86_64


RHEL:
Red Hat Enterprise Linux Server release 7.2
kernel-3.10.0-327.3.1.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.8.x86_64
libvirt-daemon-1.2.17-13.el7_2.2.x86_64
vdsm-4.17.23-0.el7ev.noarch
spice-server-0.12.4-15.el7.x86_64

Test steps:
1. Install RHEV-H * RHEL.
2. Register  2 hosts to RHEV-M.
3. Connect NFS storage
4. Create VM on RHEV-H
5. Migrate VM to RHEL host

Test result:
Get the same error with the original bug, failed to migration from RHEV-H to RHEL.

Comment 8 cshao 2016-03-04 03:14:22 UTC
Test version:
RHEV-M: rhevm-3.6.3.4-0.1.el6

RHEV-H:
rhev-hypervisor7-7.2-20160302.1
kernel-3.10.0-327.10.1.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.7.x86_64
libvirt-1.2.17-13.el7_2.3.x86_64
vdsm-4.17.23-0.el7ev.noarch
spice-server-0.12.4-15.el7.x86_64
ipxe-roms-qemu-20130517-8.gitc4bce43.el7_2.1.noarch

RHEL:
Red Hat Enterprise Linux Server release 7.2
kernel-3.10.0-327.3.1.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.8.x86_64
libvirt-daemon-1.2.17-13.el7_2.2.x86_64
vdsm-4.17.23-0.el7ev.noarch
spice-server-0.12.4-15.el7.x86_64

Test steps:
1. Install RHEV-H * RHEL.
2. Register  2 hosts to RHEV-M.
3. Connect NFS storage
4. Create VM on RHEL
5. Migrate VM to RHEV-H host
6. Migrate VM to RHEL host

Test result:
Migrate PASS.

Comment 10 Ying Cui 2016-03-04 07:31:43 UTC
Fabian, also need to attach this bug to errata(back to NEW_FILES), then I will handle errata process. Thanks.

Comment 11 Fabian Deutsch 2016-03-04 07:37:05 UTC
Fixed according to comment 8.


Note You need to log in before you can comment on or make changes to this bug.