Description of problem: Migration VM from RHEV-H to RHEL failed Checked also (on the same hosts): RHEL to RHEV-H - failed RHEV-H to RHEV-H - pass Version-Release number of selected component (if applicable): RHEVM: 3.6.3.4-0.1.el6 RHEV-H host: OS Version:RHEV Hypervisor - 7.2 - 20160225.0.el7ev Kernel Version:3.10.0 - 327.10.1.el7.x86_64 KVM Version:2.3.0 - 31.el7_2.7 LIBVIRT Version:libvirt-1.2.17-13.el7_2.3 VDSM Version:vdsm-4.17.23-0.el7ev SPICE Version:0.12.4 - 15.el7 RHEL host: OS Version:RHEL - 7.2 - 9.el7_2.1 Kernel Version:3.10.0 - 327.12.1.el7.x86_64 KVM Version:2.3.0 - 31.el7_2.8 LIBVIRT Version:libvirt-1.2.17-13.el7_2.3 VDSM Version:vdsm-4.17.23-0.el7ev SPICE Version:0.12.4 - 15.el7 How reproducible: All the time Steps to Reproduce: 1. Create VM on RHEV-H 2. Migrate VM to RHEVM host Actual results: Migration failed Additional info: jsonrpc.Executor/1::DEBUG::2016-03-01 11:18:59,772::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.migrate' in bridge with {u'params': {u'tunneled': u'false', u'dstqemu': u'10.35.161.150', u'autoConverge': u'false', u'src': u'10.35.106.13', u'dst': u'10.35.161.150:54321', u'vmId': u'1eaaf517-6028-4d66-aa81-b908c20369f7', u'abortOnError': u'true', u'compressed': u'false', u'method': u'online'}, u'vmID': u'1eaaf517-6028-4d66-aa81-b908c20369f7'} jsonrpc.Executor/1::DEBUG::2016-03-01 11:18:59,773::API::546::vds::(migrate) {u'tunneled': u'false', u'dstqemu': u'10.35.161.150', u'autoConverge': u'false', u'src': u'10.35.106.13', u'dst': u'10.35.161.150:54321', u'vmId': u'1eaaf517-6028-4d66-aa81-b908c20369f7', u'abortOnError': u'true', u'compressed': u'false', u'method': u'online'} jsonrpc.Executor/1::DEBUG::2016-03-01 11:18:59,775::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4301232530, "1eaaf517-6028-4d66-aa81-b908c20369f7": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|1eaaf517-6028-4d66-aa81-b908c20369f7"} jsonrpc.Executor/1::DEBUG::2016-03-01 11:18:59,776::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'VM.migrate' in bridge with True Thread-62::DEBUG::2016-03-01 11:18:59,800::migration::147::virt.vm::(_setupVdsConnection) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Initiating connection with destination Thread-62::DEBUG::2016-03-01 11:18:59,801::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response JsonRpc (StompReactor)::DEBUG::2016-03-01 11:18:59,802::stomp::477::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established Thread-62::DEBUG::2016-03-01 11:18:59,805::migration::159::virt.vm::(_setupVdsConnection) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Destination server is: 10.35.161.150:54321 Thread-62::DEBUG::2016-03-01 11:18:59,808::migration::202::virt.vm::(_prepareGuest) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Migration started Thread-62::DEBUG::2016-03-01 11:18:59,829::migration::287::virt.vm::(run) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::migration semaphore acquired after 0 seconds Thread-62::DEBUG::2016-03-01 11:18:59,854::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response Thread-62::INFO::2016-03-01 11:18:59,925::migration::334::virt.vm::(_startUnderlyingMigration) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Creation of destination VM took: 0 seconds Thread-62::INFO::2016-03-01 11:18:59,926::migration::353::virt.vm::(_startUnderlyingMigration) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::starting migration to qemu+tls://10.35.161.150/system with miguri tcp://10.35.161.150 Thread-63::DEBUG::2016-03-01 11:18:59,926::migration::442::virt.vm::(run) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::migration downtime thread started (10 steps) Thread-63::DEBUG::2016-03-01 11:18:59,927::migration::465::virt.vm::(_set_downtime) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::setting migration downtime to 51 Thread-64::DEBUG::2016-03-01 11:18:59,927::migration::499::virt.vm::(monitor_migration) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::starting migration monitor thread Thread-62::DEBUG::2016-03-01 11:19:00,952::migration::557::virt.vm::(stop) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::stopping migration monitor thread Thread-62::DEBUG::2016-03-01 11:19:00,953::migration::452::virt.vm::(stop) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::stopping migration downtime thread Thread-62::ERROR::2016-03-01 11:19:00,953::migration::208::virt.vm::(_recover) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::internal error: early end of file from monitor: possible problem: 2016-03-01T11:19:00.479958Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2016-03-01T11:19:00.480245Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config 2016-03-01T11:19:00.715359Z qemu-kvm: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x20000 in != 0x40000: Invalid argument 2016-03-01T11:19:00.715385Z qemu-kvm: error while loading state for instance 0x0 of device 'ram' 2016-03-01T11:19:00.715478Z qemu-kvm: load of migration failed: Invalid argument Thread-62::DEBUG::2016-03-01 11:19:00,954::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending response Thread-63::DEBUG::2016-03-01 11:19:00,954::migration::449::virt.vm::(run) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::migration downtime thread exiting Thread-62::DEBUG::2016-03-01 11:19:00,989::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4301233740, "1eaaf517-6028-4d66-aa81-b908c20369f7": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|1eaaf517-6028-4d66-aa81-b908c20369f7"} Thread-62::ERROR::2016-03-01 11:19:00,990::migration::309::virt.vm::(run) vmId=`1eaaf517-6028-4d66-aa81-b908c20369f7`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 297, in run self._startUnderlyingMigration(time.time()) File "/usr/share/vdsm/virt/migration.py", line 363, in _startUnderlyingMigration self._perform_migration(duri, muri) File "/usr/share/vdsm/virt/migration.py", line 402, in _perform_migration self._vm._dom.migrateToURI3(duri, params, flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: internal error: early end of file from monitor: possible problem: 2016-03-01T11:19:00.479958Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2016-03-01T11:19:00.480245Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config 2016-03-01T11:19:00.715359Z qemu-kvm: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x20000 in != 0x40000: Invalid argument 2016-03-01T11:19:00.715385Z qemu-kvm: error while loading state for instance 0x0 of device 'ram' 2016-03-01T11:19:00.715478Z qemu-kvm: load of migration failed: Invalid argument jsonrpc.Executor/2::DEBUG::2016-03-01 11:19:04,196::task::595::Storage.TaskManager.Task::(_updateState) Task=`965c1dc8-13a9-4854-be63-377e8ca6540d`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2016-03-01 11:19:04,196::logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Created attachment 1131938 [details] logs
Re-appearance of bug 1293566? Can you check relevant versions of the *pxe* packages?
The RHEV-H and RHEL are installed from QE foreman. The ipxe version on the rhev-h is: ipxe-roms-qemu-20130517-7.1fm.gitc4bce43.el7sat.noarch we have diff version then the rhel: ipxe-roms-qemu-20130517-8.gitc4bce43.el7_2.1.noarch It looks like the same problem in BZ:1302742, Installation problem, closing the bug *** This bug has been marked as a duplicate of bug 1302742 ***
Re-opening this bug and moving it to Node
RHEV-H QE can reproduce this bug. Test version: RHEV-M: rhevm-3.6.3.4-0.1.el6 RHEV-H: RHEV Hypervisor - 7.2 - 20160225.0.el7ev kernel-3.10.0-327.10.1.el7.x86_64 qemu-kvm-rhev-2.3.0-31.el7_2.7.x86_64 qemu-kvm-common-rhev-2.3.0-31.el7_2.7.x86_64 libvirt-1.2.17-13.el7_2.3.x86_64 vdsm-4.17.23-0.el7ev.noarch spice-server-0.12.4-15.el7.x86_64 RHEL: Red Hat Enterprise Linux Server release 7.2 kernel-3.10.0-327.3.1.el7.x86_64 qemu-kvm-rhev-2.3.0-31.el7_2.8.x86_64 libvirt-daemon-1.2.17-13.el7_2.2.x86_64 vdsm-4.17.23-0.el7ev.noarch spice-server-0.12.4-15.el7.x86_64 Test steps: 1. Install RHEV-H * RHEL. 2. Register 2 hosts to RHEV-M. 3. Connect NFS storage 4. Create VM on RHEV-H 5. Migrate VM to RHEL host Test result: Get the same error with the original bug, failed to migration from RHEV-H to RHEL.
Test version: RHEV-M: rhevm-3.6.3.4-0.1.el6 RHEV-H: rhev-hypervisor7-7.2-20160302.1 kernel-3.10.0-327.10.1.el7.x86_64 qemu-kvm-rhev-2.3.0-31.el7_2.7.x86_64 libvirt-1.2.17-13.el7_2.3.x86_64 vdsm-4.17.23-0.el7ev.noarch spice-server-0.12.4-15.el7.x86_64 ipxe-roms-qemu-20130517-8.gitc4bce43.el7_2.1.noarch RHEL: Red Hat Enterprise Linux Server release 7.2 kernel-3.10.0-327.3.1.el7.x86_64 qemu-kvm-rhev-2.3.0-31.el7_2.8.x86_64 libvirt-daemon-1.2.17-13.el7_2.2.x86_64 vdsm-4.17.23-0.el7ev.noarch spice-server-0.12.4-15.el7.x86_64 Test steps: 1. Install RHEV-H * RHEL. 2. Register 2 hosts to RHEV-M. 3. Connect NFS storage 4. Create VM on RHEL 5. Migrate VM to RHEV-H host 6. Migrate VM to RHEL host Test result: Migrate PASS.
Fabian, also need to attach this bug to errata(back to NEW_FILES), then I will handle errata process. Thanks.
Fixed according to comment 8.