Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1150636

Summary: [migration] Can't migrate a VM from RHEL6.6 host to RHEL7 host: virNetClientProgramDispatchError:174 : cannot read header '$path' Input/output errorr
Product: Red Hat Enterprise Virtualization Manager Reporter: Jiri Belka <jbelka>
Component: vdsmAssignee: Nobody <nobody>
Status: CLOSED WORKSFORME QA Contact: meital avital <mavital>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.5.0CC: bazulay, ecohen, gklein, iheim, jbelka, lpeer, lsurette, michal.skrivanek, ofrenkel, vfeenstr, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: virt
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-19 11:17:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sosreport-LogCollector-20141008163953.tar.xz none

Description Jiri Belka 2014-10-08 14:50:24 UTC 
Created attachment 945045 [details]
sosreport-LogCollector-20141008163953.tar.xz

Description of problem:

Just live migrating a VM from RHEL6.6 (vdsm-4.16.6-1.el6ev/libvirt-1.1.1-29.el7_0.3) host to RHEL7 (vdsm-4.16.6-1.el7/libvirt-1.1.1-29.el7_0.3) host.

...
2014-10-08 14:26:17.585+0000: 2310: error : virDomainObjGetMetadata:16650 : metadata not found: Requested metadata element is not present
2014-10-08 14:26:20.698+0000: 2308: error : virNetClientProgramDispatchError:174 : cannot read header '/rhev/data-center/00000002-0002-0002-0002-0000000001f1/322e3360-066e-4e98-a0e9-dd44e82f0992/images/e0bcb03f-8b25-4841-b39e-8575d112ee55/91743eb0-5857-4155-9bb3-2e3cc991d15f': Input/output error

...
Thread-15584::DEBUG::2014-10-08 16:26:20,742::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 38 edom: 18 level: 2 message: cannot read header '/rhev/data-center/00000002-0002-0002-0002-0000000001f1/322e3360-066e-4e98-a0e9-dd44e82f0992/images/e0bcb03f-8b25-4841-b39e-8575d112ee55/91743eb0-5857-4155-9bb3-2e3cc991d15f': Input/output error
Thread-15584::DEBUG::2014-10-08 16:26:20,742::migration::375::vm.Vm::(cancel) vmId=`63d1ebb8-3614-4685-8ea6-a022d8725e1d`::canceling migration downtime thread
Thread-15584::DEBUG::2014-10-08 16:26:20,743::migration::469::vm.Vm::(stop) vmId=`63d1ebb8-3614-4685-8ea6-a022d8725e1d`::stopping migration monitor thread
Thread-15585::DEBUG::2014-10-08 16:26:20,743::migration::372::vm.Vm::(run) vmId=`63d1ebb8-3614-4685-8ea6-a022d8725e1d`::migration downtime thread exiting
Thread-15584::ERROR::2014-10-08 16:26:20,744::migration::160::vm.Vm::(_recover) vmId=`63d1ebb8-3614-4685-8ea6-a022d8725e1d`::cannot read header '/rhev/data-center/00000002-0002-0002-0002-0000000001f1/322e3360-066e-4e98-a0e9-dd44e82f0992/images/e0bcb03f-8b25-4841-b39e-8575d112ee55/91743eb0-5857-4155-9bb3-2e3cc991d15f': Input/output error
Thread-15584::ERROR::2014-10-08 16:26:20,890::migration::259::vm.Vm::(run) vmId=`63d1ebb8-3614-4685-8ea6-a022d8725e1d`::Failed to migrate
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/migration.py", line 245, in run
    self._startUnderlyingMigration(time.time())
  File "/usr/share/vdsm/virt/migration.py", line 324, in _startUnderlyingMigration
    None, maxBandwidth)
  File "/usr/share/vdsm/virt/vm.py", line 670, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1210, in migrateToURI2
    if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: cannot read header '/rhev/data-center/00000002-0002-0002-0002-0000000001f1/322e3360-066e-4e98-a0e9-dd44e82f0992/images/e0bcb03f-8b25-4841-b39e-8575d112ee55/91743eb0-5857-4155-9bb3-2e3cc991d15f': Input/output error


Version-Release number of selected component (if applicable):
vdsm-4.16.6-1.el7.x86_64 libvirt-daemon-1.1.1-29.el7_0.3.x86_64 qemu-kvm-rhev-1.5.3-60.el7_0.10.x86_64

vdsm-4.16.6-1.el6ev.x86_64 libvirt-0.10.2-46.el6.x86_64 qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64

How reproducible:


Steps to Reproduce:
1. migrate vm from host being rhel6.6 to a host being rhel7
2.
3.

Actual results:
migration failed

Expected results:
should pass

Additional info:
Comment 1 Jiri Belka 2014-10-08 14:53:26 UTC 
How reproducible:

  once it didn't work, 2nd try worked

anyway, please investigate.
Comment 2 Michal Skrivanek 2014-10-10 17:30:08 UTC 
looks like some storage problem?
Comment 3 Vinzenz Feenstra [evilissimo] 2014-10-13 07:31:36 UTC 
(In reply to Michal Skrivanek from comment #2)
> looks like some storage problem?

It most certainly was a storage problem, the other possibility I could have thought of would have been an SELinux denial, however since the 2nd try worked that can be excluded.

This looks simply like a hiccup with the remote storage. There's nothing else what could be deduced from that nor any other hint to anything code related.
Comment 4 Omer Frenkel 2014-10-13 08:56:05 UTC 
Jiri, does this reproduce?
assuming this really was a temp storage issue, there is nothing much we can do.
Comment 5 Jiri Belka 2014-10-17 12:33:51 UTC 
I can't reproduce on vt6: totally clean environment, VM started on rhel6.6 based host and migrated successfully to rhel7.0 based host.

Feel free to close if you haven't found anything suspicious in the logs. Thanks!
Comment 6 Omer Frenkel 2014-10-19 11:17:55 UTC 
closing, please re-open if needed