Bug 1014645 - RHEV-M: VM migration fails from 3.3 host to 3.2 host - cannot open disk image file
RHEV-M: VM migration fails from 3.3 host to 3.2 host - cannot open disk image...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Libvirt Maintainers
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-02 09:19 EDT by Pavel Novotny
Modified: 2013-10-03 04:21 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-03 04:21:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
libvirtd.log from the source 3.3 host (37.62 KB, text/plain)
2013-10-02 09:19 EDT, Pavel Novotny
no flags Details
libvirtd.log from the target 3.2 host (25.79 KB, text/plain)
2013-10-02 11:25 EDT, Pavel Novotny
no flags Details
VDSM log from the source 3.3 host (239.63 KB, text/plain)
2013-10-02 11:26 EDT, Pavel Novotny
no flags Details
VDSM log from the target 3.2 host (1.12 MB, text/plain)
2013-10-02 11:27 EDT, Pavel Novotny
no flags Details

  None (edit)
Description Pavel Novotny 2013-10-02 09:19:13 EDT
Created attachment 806453 [details]
libvirtd.log from the source 3.3 host

Description of problem:
Using RHEV-M with two hosts (3.3 and 3.2), migration of a VM from 3.3 host to 3.2 host fails. Libvirt throws an exception that it cannot find the VM disk image file, despite it's present on the filesystem.
Note: SELinux was set to permissive mode on both hosts.

Version-Release number of selected component (if applicable):
3.3 host:
libvirt-0.10.2-27.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.355.el6_4.7.x86_64
vdsm-4.12.0-156.git6e499d6.el6ev.x86_64

3.2 host:
libvirt-0.10.2-18.el6_4.14.x86_64
qemu-kvm-rhev-0.12.1.2-2.355.el6_4.2.x86_64
vdsm-4.10.2-25.1.el6ev.x86_64


How reproducible:
100%

Steps to Reproduce:
1. In RHEV-M, have 3.3 and 3.2 host in a 3.2 cluster.
2. Create new VM with a bootable disk and run it on the 3.3 host.
3. Migrate the VM to the 3.2 host.

Actual results:
Migration fails.
Error in libvirt log (attached): "virNetClientProgramDispatchError:174 : cannot open file '/var/run/vdsm/storage/84f6d841-6c6a-49be-a0f5-b7d80f876573/1985711b-46fd-4cb7-909b-1168f60b5547/548c7b86-b75e-441e-97b9-c163ffdacaca': No such file or directory".

Expected results:
Migration completes successfully.

Additional info:

The file in /var/run/vdsm/storage/ actually exists:
# ls -lh /var/run/vdsm/storage/84f6d841-6c6a-49be-a0f5-b7d80f876573/1985711b-46fd-4cb7-909b-1168f60b5547/548c7b86-b75e-441e-97b9-c163ffdacaca
-rw-rw----. 1 vdsm kvm 1,0G  2. říj 13.41 /var/run/vdsm/storage/84f6d841-6c6a-49be-a0f5-b7d80f876573/1985711b-46fd-4cb7-909b-1168f60b5547/548c7b86-b75e-441e-97b9-c163ffdacaca

Libvirt log file from the source 3.3 host attached (libvirt log on target host is empty).
Comment 2 Jiri Denemark 2013-10-02 10:29:35 EDT
The error is coming from the destination host and we need libvirtd logs from there. If the logs there are empty, I think there are some configuration issues on that host. Also where did you run the "ls -lh" command? On the source or destination host?
Comment 3 Pavel Novotny 2013-10-02 11:23:49 EDT
(In reply to Jiri Denemark from comment #2)
> The error is coming from the destination host and we need libvirtd logs from
> there. If the logs there are empty, I think there are some configuration
> issues on that host. ...

Ok, after a little bit of investigation I found the libvirt log on the target host, it was just at slightly different location then on the source host :) Put in attachment.

> Also where did you run the "ls -lh" command? On the
> source or destination host?

I ran the `ls` command on the source host.
On the destination host there was even no directory `/var/run/vdsm/storage/` when I checked after the failed migration (but it could have been already removed by VDSM). Attaching also the VDSM logs from both hosts.
Comment 4 Pavel Novotny 2013-10-02 11:25:20 EDT
Created attachment 806534 [details]
libvirtd.log from the target 3.2 host
Comment 5 Pavel Novotny 2013-10-02 11:26:22 EDT
Created attachment 806535 [details]
VDSM log from the source 3.3 host
Comment 6 Pavel Novotny 2013-10-02 11:27:11 EDT
Created attachment 806536 [details]
VDSM log from the target 3.2 host
Comment 7 Jiri Denemark 2013-10-03 04:21:49 EDT
Please, could you do a little bit more investigation before filing a bug next time, esp. if the error message is so clear? It may always be a bug but it's much better to check obvious thing before creating a bug report. I have no idea why the /var/run/vdsm/storage directory is not present on your 3.2 host but that's also part of the investigation since the most obvious reason is a bad configuration of vdsm on that host.

In any case, I'm closing this bug. Feel free to file a new one if you find a real bug there.

Note You need to log in before you can comment on or make changes to this bug.