Bug 1014645 - RHEV-M: VM migration fails from 3.3 host to 3.2 host - cannot open disk image file
Summary: RHEV-M: VM migration fails from 3.3 host to 3.2 host - cannot open disk image...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Libvirt Maintainers
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-10-02 13:19 UTC by Pavel Novotny
Modified: 2013-10-03 08:21 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-10-03 08:21:49 UTC
Target Upstream Version:


Attachments (Terms of Use)
libvirtd.log from the source 3.3 host (37.62 KB, text/plain)
2013-10-02 13:19 UTC, Pavel Novotny
no flags Details
libvirtd.log from the target 3.2 host (25.79 KB, text/plain)
2013-10-02 15:25 UTC, Pavel Novotny
no flags Details
VDSM log from the source 3.3 host (239.63 KB, text/plain)
2013-10-02 15:26 UTC, Pavel Novotny
no flags Details
VDSM log from the target 3.2 host (1.12 MB, text/plain)
2013-10-02 15:27 UTC, Pavel Novotny
no flags Details

Description Pavel Novotny 2013-10-02 13:19:13 UTC
Created attachment 806453 [details]
libvirtd.log from the source 3.3 host

Description of problem:
Using RHEV-M with two hosts (3.3 and 3.2), migration of a VM from 3.3 host to 3.2 host fails. Libvirt throws an exception that it cannot find the VM disk image file, despite it's present on the filesystem.
Note: SELinux was set to permissive mode on both hosts.

Version-Release number of selected component (if applicable):
3.3 host:
libvirt-0.10.2-27.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.355.el6_4.7.x86_64
vdsm-4.12.0-156.git6e499d6.el6ev.x86_64

3.2 host:
libvirt-0.10.2-18.el6_4.14.x86_64
qemu-kvm-rhev-0.12.1.2-2.355.el6_4.2.x86_64
vdsm-4.10.2-25.1.el6ev.x86_64


How reproducible:
100%

Steps to Reproduce:
1. In RHEV-M, have 3.3 and 3.2 host in a 3.2 cluster.
2. Create new VM with a bootable disk and run it on the 3.3 host.
3. Migrate the VM to the 3.2 host.

Actual results:
Migration fails.
Error in libvirt log (attached): "virNetClientProgramDispatchError:174 : cannot open file '/var/run/vdsm/storage/84f6d841-6c6a-49be-a0f5-b7d80f876573/1985711b-46fd-4cb7-909b-1168f60b5547/548c7b86-b75e-441e-97b9-c163ffdacaca': No such file or directory".

Expected results:
Migration completes successfully.

Additional info:

The file in /var/run/vdsm/storage/ actually exists:
# ls -lh /var/run/vdsm/storage/84f6d841-6c6a-49be-a0f5-b7d80f876573/1985711b-46fd-4cb7-909b-1168f60b5547/548c7b86-b75e-441e-97b9-c163ffdacaca
-rw-rw----. 1 vdsm kvm 1,0G  2. říj 13.41 /var/run/vdsm/storage/84f6d841-6c6a-49be-a0f5-b7d80f876573/1985711b-46fd-4cb7-909b-1168f60b5547/548c7b86-b75e-441e-97b9-c163ffdacaca

Libvirt log file from the source 3.3 host attached (libvirt log on target host is empty).

Comment 2 Jiri Denemark 2013-10-02 14:29:35 UTC
The error is coming from the destination host and we need libvirtd logs from there. If the logs there are empty, I think there are some configuration issues on that host. Also where did you run the "ls -lh" command? On the source or destination host?

Comment 3 Pavel Novotny 2013-10-02 15:23:49 UTC
(In reply to Jiri Denemark from comment #2)
> The error is coming from the destination host and we need libvirtd logs from
> there. If the logs there are empty, I think there are some configuration
> issues on that host. ...

Ok, after a little bit of investigation I found the libvirt log on the target host, it was just at slightly different location then on the source host :) Put in attachment.

> Also where did you run the "ls -lh" command? On the
> source or destination host?

I ran the `ls` command on the source host.
On the destination host there was even no directory `/var/run/vdsm/storage/` when I checked after the failed migration (but it could have been already removed by VDSM). Attaching also the VDSM logs from both hosts.

Comment 4 Pavel Novotny 2013-10-02 15:25:20 UTC
Created attachment 806534 [details]
libvirtd.log from the target 3.2 host

Comment 5 Pavel Novotny 2013-10-02 15:26:22 UTC
Created attachment 806535 [details]
VDSM log from the source 3.3 host

Comment 6 Pavel Novotny 2013-10-02 15:27:11 UTC
Created attachment 806536 [details]
VDSM log from the target 3.2 host

Comment 7 Jiri Denemark 2013-10-03 08:21:49 UTC
Please, could you do a little bit more investigation before filing a bug next time, esp. if the error message is so clear? It may always be a bug but it's much better to check obvious thing before creating a bug report. I have no idea why the /var/run/vdsm/storage directory is not present on your 3.2 host but that's also part of the investigation since the most obvious reason is a bad configuration of vdsm on that host.

In any case, I'm closing this bug. Feel free to file a new one if you find a real bug there.


Note You need to log in before you can comment on or make changes to this bug.