Red Hat Bugzilla – Bug 1138231
Report better error when backing chain detection fails
Last modified: 2015-03-05 02:43:53 EST
Description v2v:libguestfs error: could not create appliance through libvirt Version: libguestfs-tools-c-1.27.32-1.1.el7.x86_64 libguestfs-1.27.32-1.1.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.Copy xen guest xml and image to virt-v2v server,use virt-v2v to convert it to localhost,error shows as blow. # virt-v2v -i libvirtxml -o local -os /var/tmp rhel6.6-pv-x64-test.xml [ 0.0] Opening the source -i libvirtxml rhel6.6-pv-x64-test.xml [ 0.0] Creating an overlay to protect the source from being modified [ 0.0] Opening the overlay virt-v2v: libguestfs error: could not create appliance through libvirt. Try running qemu directly without libvirt using this environment variable: export LIBGUESTFS_BACKEND=direct Original error from libvirt: internal error: process exited while connecting to monitor: Warning: option deprecated, use lost_tick_policy property of kvm-pit instead. qemu-kvm: -drive file=/tmp/v2vovlec242c.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=unsafe,discard=unmap: could not open disk image /tmp/v2vovlec242c.qcow2: Could not open file: Permission denied [code=1 domain=10] 2.# virt-v2v -i disk -o local -os /var/tmp rhel6.6-pv-x64-test.img [ 0.0] Opening the source -i disk rhel6.6-pv-x64-test.img libguestfs: error: invalid backend: [ 0.0] Creating an overlay to protect the source from being modified [ 0.0] Opening the overlay libguestfs: error: invalid backend: virt-v2v: libguestfs error: could not create appliance through libvirt. Try running qemu directly without libvirt using this environment variable: export LIBGUESTFS_BACKEND=direct Original error from libvirt: internal error: process exited while connecting to monitor: Warning: option deprecated, use lost_tick_policy property of kvm-pit instead. qemu-kvm: -drive file=/tmp/v2vovlb7760b.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=unsafe,discard=unmap: could not open disk image /tmp/v2vovlb7760b.qcow2: Could not open file: Permission denied [code=1 domain=10] 3.After run the below command,virt-v2v can work well in step 1 and step 2. # export LIBGUESTFS_BACKEND=direct Actual results： Libguestfs doesn't execute command "export LIBGUESTFS_BACKEND=direct" automatically,the warning info is not so clear and friendly,the error info from libvirt is a bit confusing. Expected results: Libguestfs execute command "export LIBGUESTFS_BACKEND=direct" automatically. If step1 and step2 should run "export LIBGUESTFS_BACKEND=direct" first,it should be documented in virt-v2v manual/guide. Improve the warning info to be more clear and friendly. Additional info:
(In reply to tingting zheng from comment #0) > Description > v2v:libguestfs error: could not create appliance through libvirt > > Version: > libguestfs-tools-c-1.27.32-1.1.el7.x86_64 > libguestfs-1.27.32-1.1.el7.x86_64 > > How reproducible: > 100% > > Steps to Reproduce: > 1.Copy xen guest xml and image to virt-v2v server,use virt-v2v to convert it > to localhost,error shows as blow. > # virt-v2v -i libvirtxml -o local -os /var/tmp rhel6.6-pv-x64-test.xml > [ 0.0] Opening the source -i libvirtxml rhel6.6-pv-x64-test.xml > [ 0.0] Creating an overlay to protect the source from being modified > [ 0.0] Opening the overlay > virt-v2v: libguestfs error: could not create appliance through libvirt. > > Try running qemu directly without libvirt using this environment variable: > export LIBGUESTFS_BACKEND=direct > > Original error from libvirt: internal error: process exited while connecting > to monitor: Warning: option deprecated, use lost_tick_policy property of > kvm-pit instead. > qemu-kvm: -drive > file=/tmp/v2vovlec242c.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2, > cache=unsafe,discard=unmap: could not open disk image > /tmp/v2vovlec242c.qcow2: Could not open file: Permission denied > [code=1 domain=10] The fundamental problem here is that you need to use qemu-kvm-rhev from here: https://brewweb.devel.redhat.com/packageinfo?packageID=35491 I haven't worked out how to express the dependencies correctly for RHEL, but it's possible we may end up (on RHEL) splitting out the virt-v2v package and adding a dependency on qemu-kvm-rhev. Unfortunately we are also still waiting for some fixes to go into qemu-kvm-rhev, see: https://bugzilla.redhat.com/show_bug.cgi?id=1132569#c5 I have patched my copy of qemu-kvm-rhev to add these fixes, and it's likely that without those patches, any attempt to import from ESX will fail for you (although *with* those patches, it should work). The patches are: https://bugzilla.redhat.com/show_bug.cgi?id=1132569#c6 You can grab that patch and apply it to qemu-kvm-rhev from dist-git, but let me know if you need a scratch build.
While I'm at it, also you'll need libguestfs-winsupport from: https://brewweb.devel.redhat.com/buildinfo?buildID=380616
(In reply to Richard W.M. Jones from comment #1) > The fundamental problem here is that you need to use qemu-kvm-rhev > from here: > > https://brewweb.devel.redhat.com/packageinfo?packageID=35491 > > I haven't worked out how to express the dependencies correctly for > RHEL, but it's possible we may end up (on RHEL) splitting out the > virt-v2v package and adding a dependency on qemu-kvm-rhev. I tried the scratch qemu-kvm-rhev you sent to me: qemu-kvm-rhev-2.1.0-3.rwmj3.el7.x86_64 # echo $LIBGUESTFS_BACKEND Run virt-v2v,there is still such error info. # virt-v2v -i disk -o local -os /var/tmp rhel6.6-i386-hvm.img [ 0.0] Opening the source -i disk rhel6.6-i386-hvm.img [ 0.0] Creating an overlay to protect the source from being modified [ 0.0] Opening the overlay virt-v2v: libguestfs error: could not create appliance through libvirt. Try running qemu directly without libvirt using this environment variable: export LIBGUESTFS_BACKEND=direct Original error from libvirt: invalid argument: Backing file '/root/rhel6.6-i386-hvm.img' of image '/tmp/v2vovlba582a.qcow2' is missing. [code=8 domain=10]
I thought I understood this bug, but apparently not. It *might* be because of libvirt: http://libguestfs.org/guestfs-faq.1.html#permission-denied-when-running-libguestfs-as-root I've cleaned up the -i disk code (in 1.27.38) so it should produce clearer error messages next time.
Tingting: Please let me know if this still happens with the 1.27.41 (or later) package.
(In reply to Richard W.M. Jones from comment #6) > Tingting: Please let me know if this still happens with the > 1.27.41 (or later) package. So it probably *will* still happen, but it would be nice to get updated output, with 'virt-v2v -v -x' flags, because later versions of virt-v2v provide more detailed logging in this part of the code.
(In reply to Richard W.M. Jones from comment #7) > (In reply to Richard W.M. Jones from comment #6) > > Tingting: Please let me know if this still happens with the > > 1.27.41 (or later) package. > > So it probably *will* still happen, but it would be nice > to get updated output, with 'virt-v2v -v -x' flags, because > later versions of virt-v2v provide more detailed logging > in this part of the code. Yes,the error is still there,detailed debug info is attached.
Created attachment 935970 [details] Detailed log file
More info about this bug: Tried the below version: # rpm -qa libguestfs virt-v2v libguestfs-1.27.41-1.1.el7.x86_64 virt-v2v-1.27.41-1.1.el7.x86_64 1.virt-v2v -i disk will show the same error,the log file has been attached in comment 9. # virt-v2v -i disk -o local -os /var/tmp rhel6.6-pv-x64-test.img [ 0.0] Opening the source -i disk rhel6.6-pv-x64-test.img [ 0.0] Creating an overlay to protect the source from being modified [ 0.0] Opening the overlay virt-v2v: error: libguestfs error: could not create appliance through libvirt. Try running qemu directly without libvirt using this environment variable: export LIBGUESTFS_BACKEND=direct Original error from libvirt: invalid argument: Backing file '/root/rhel6.6-pv-x64-test.img' of image '/tmp/v2vovlfb8258.qcow2' is missing. [code=8 domain=10] If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] 2.virt-v2v -ic xen+ssh [...],different error shows that there is libvirt bug. # virt-v2v -ic xen+ssh://10.66.106.64 -os default rhel6.6-pv-x64-test [ 0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 rhel6.6-pv-x64-test libvirt: Remote Driver error : unknown procedure: 212 virt-v2v: error: because of libvirt bug https://bugzilla.redhat.com/show_bug.cgi?id=1134592 you must set this environment variable: export LIBGUESTFS_BACKEND=direct and then rerun the virt-v2v command. If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...]
(In reply to tingting zheng from comment #10) > Original error from libvirt: invalid argument: Backing file > '/root/rhel6.6-pv-x64-test.img' of image '/tmp/v2vovlfb8258.qcow2' is > missing. [code=8 domain=10] I have to think that either this is a libvirt bug, or it's happening because the file is located in /root and perhaps libvirt cannot open /root because it's not running as root (maybe it runs as another user?) I am not able to reproduce this on my machine. However here are some other things to try: (1) Does the bug still happen if you move the file from /root to /tmp ? (2) If it still fails, what is the output when you enable libvirt debugging: export LIBVIRT_DEBUG=1
Adding Dan, maybe he has some idea. The code for checking backing files in libvirt is rather complex.
I tried to put the file to /tmp,then it can be converted successfully. # virt-v2v -i disk -o local -os /var/tmp /tmp/rhel6.6-pv-x64-test.img [ 0.0] Opening the source -i disk /tmp/rhel6.6-pv-x64-test.img [ 0.0] Creating an overlay to protect the source from being modified [ 0.0] Opening the overlay [ 10.0] Initializing the target -o local -os /var/tmp [ 10.0] Inspecting the overlay [ 16.0] Checking for sufficient free disk space in the guest [ 16.0] Converting Red Hat Enterprise Linux Server release 6.6 Beta (Santiago) to run on KVM virt-v2v: warning: /files/boot/grub/device.map/hd0 references unknown device "xvda". You may have to fix this entry manually after conversion. [ 82.0] Mapping filesystem data to avoid copying unused and blank areas [ 83.0] Closing the overlay [ 83.0] Copying disk 1/1 to /var/tmp/rhel6.6-pv-x64-test-sda (raw) (100.00/100%) [ 118.0] Creating output metadata [ 119.0] Finishing off
I am preliminarily reassigning this bug to libvirt, since I think it at least needs investigation. I am still trying to reproduce the exact error.
The good news is I can reproduce this error, using: - RHEL 7.1 - libvirt-1.2.8-2.el7.x86_64 - virt-v2v-1.27.42-1.2.el7.x86_64 I put a dummy (empty) file in /root/test1.img and ran the following command as root: # virt-v2v -i disk test1.img -o local -os /var/tmp [ 0.0] Opening the source -i disk test1.img [ 0.0] Creating an overlay to protect the source from being modified [ 0.0] Opening the overlay virt-v2v: error: libguestfs error: could not create appliance through libvirt. Try running qemu directly without libvirt using this environment variable: export LIBGUESTFS_BACKEND=direct Original error from libvirt: invalid argument: Backing file '/root/test1.img' of image '/tmp/v2vovl493098.qcow2' is missing. [code=8 domain=10] Obviously the file does exist: # ll /root/test1.img -rw-r--r--. 1 root root 104857600 Sep 11 12:17 /root/test1.img The error message from libvirt would indicate that it is missing, but it's possible libvirt means something else, eg. that the file is inaccessible (SELinux? permissions?)
I turned off SELinux, but the error still occurs => NOT SELinux. However it is a permissions problem. I straced the libvirtd process and it fails here: 4993 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fc2f1c939d0) = 5214 4993 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 4993 wait4(5214, <unfinished ...> [ a bunch of sigaction calls omitted ] 5214 setregid(107, 107) = 0 5214 setgroups(2, [107, 36]) = 0 5214 setreuid(107, 107) = 0 5214 access("/root/test1.img", F_OK) = -1 EACCES (Permission denied) 5214 exit_group(13) = ? 5214 +++ exited with 13 +++ 4993 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 13}], 0, NULL) = 5214 So libvirtd seems to fork, change its own ownership, and then test if the file is access F_OK. Because the file is located in /root this test fails.
I guess the logic above comes from virFileAccessibleAs. I think what is happening is that libvirt is trying to check that each file is accessible as qemu.qemu before it runs qemu (which would fail later on). It sounds like a good thing, BUT the error message definitely needs to not say "Backing file .. is missing", but something like: Backing file FOO is not accessible to user qemu.qemu, starting the guest would fail. Make sure that the backing file and parent directories are readable and/or writable by user qemu.qemu.
We should report errors from backing chain detection right away rather than erroring out with the generic error message from the backing chain checker.
Fixed upstream: commit eaf43e37552e6b40468b72b2819d9b7a5874095e Author: Peter Krempa <pkrempa@redhat.com> Date: Thu Sep 11 19:09:48 2014 +0200 storage: Improve error message when traversing backing chains Report also the name of the parent file and uid/gid used to access it to help debugging broken storage configurations. commit 639a00984a1b6b328093c2e89f2b329ed4f341e4 Author: Peter Krempa <pkrempa@redhat.com> Date: Thu Sep 11 18:59:32 2014 +0200 qemu: Report better errors from broken backing chains Request erroring out from the backing chain traveller and drop qemu's internal backing chain integrity tester. The backing chain traveller reports errors by itself with possibly more detail than qemuDiskChainCheckBroken ever could. We also need to make sure that we reconnect to existing qemu instances even at the cost of losing the backing chain info (this really should be stored in the XML rather than reloaded from disk, but that needs some work). commit 172ca0e748714799bc0abec1144d7896e681728b Author: Peter Krempa <pkrempa@redhat.com> Date: Thu Sep 18 11:21:10 2014 +0200 qemu: Sanitize argument names and empty disk check in qemuDomainDetermineDiskChain Reuse virStorageSourceIsEmpty and rename "force" argument to "force_probe". commit b8549877a1e3ca1854fab2527508637b38e63f0d Author: Peter Krempa <pkrempa@redhat.com> Date: Thu Sep 11 18:28:47 2014 +0200 util: storage: Allow metadata crawler to report useful errors Add a new parameter to virStorageFileGetMetadata that will break the backing chain detection process and report useful error message rather than having to use virStorageFileChainGetBroken. This patch just introduces the option, usage will be provided separately.
I tested with: qemu-kvm-rhev-2.1.2-3.el7.x86_64 virt-v2v-1.27.62-1.1.el7.x86_64 libvirt-1.2.8-5.el7.x86_64 # echo $LIBGUESTFS_BACKEND [root@localhost ~]# virt-v2v -i disk test.img -o local -os /var/tmp [ 0.0] Opening the source -i disk test.img libguestfs: error: invalid backend: [ 0.0] Creating an overlay to protect the source from being modified libguestfs: error: invalid backend: libguestfs: error: invalid backend: [ 0.0] Opening the overlay libguestfs: error: invalid backend: virt-v2v: error: libguestfs error: could not create appliance through libvirt. Try running qemu directly without libvirt using this environment variable: export LIBGUESTFS_BACKEND=direct Original error from libvirt: Cannot access backing file '/root/test.img' of storage file '/var/tmp/v2vovl6324e4.qcow2' (as uid:107, gid:107): Permission denied [code=38 domain=18] If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Now there is still permission denied error,pls help to check.
(In reply to tingting zheng from comment #22) > I tested with: > qemu-kvm-rhev-2.1.2-3.el7.x86_64 > virt-v2v-1.27.62-1.1.el7.x86_64 > libvirt-1.2.8-5.el7.x86_64 > > # echo $LIBGUESTFS_BACKEND > > [root@localhost ~]# virt-v2v -i disk test.img -o local -os /var/tmp > [ 0.0] Opening the source -i disk test.img > libguestfs: error: invalid backend: > [ 0.0] Creating an overlay to protect the source from being modified > libguestfs: error: invalid backend: > libguestfs: error: invalid backend: > [ 0.0] Opening the overlay > libguestfs: error: invalid backend: So I guess that LIBGUESTFS_BACKEND is an empty string, because: $ LIBGUESTFS_BACKEND="" guestfish -a /dev/null run libguestfs: error: invalid backend: I have filed a new bug about this. https://bugzilla.redhat.com/show_bug.cgi?id=1153515 Tingting: could you try with: unset LIBGUESTFS_BACKEND
(In reply to Richard W.M. Jones from comment #23) > (In reply to tingting zheng from comment #22) > > I tested with: > > qemu-kvm-rhev-2.1.2-3.el7.x86_64 > > virt-v2v-1.27.62-1.1.el7.x86_64 > > libvirt-1.2.8-5.el7.x86_64 > > > > # echo $LIBGUESTFS_BACKEND > > > > [root@localhost ~]# virt-v2v -i disk test.img -o local -os /var/tmp > > [ 0.0] Opening the source -i disk test.img > > libguestfs: error: invalid backend: > > [ 0.0] Creating an overlay to protect the source from being modified > > libguestfs: error: invalid backend: > > libguestfs: error: invalid backend: > > [ 0.0] Opening the overlay > > libguestfs: error: invalid backend: > > So I guess that LIBGUESTFS_BACKEND is an empty string, because: > > $ LIBGUESTFS_BACKEND="" guestfish -a /dev/null run > libguestfs: error: invalid backend: > > I have filed a new bug about this. > > https://bugzilla.redhat.com/show_bug.cgi?id=1153515 > > Tingting: could you try with: > > unset LIBGUESTFS_BACKEND # unset LIBGUESTFS_BACKEND [root@localhost ~]# virt-v2v -i disk test.img -o local -os /var/tmp [ 0.0] Opening the source -i disk test.img [ 0.0] Creating an overlay to protect the source from being modified [ 0.0] Opening the overlay virt-v2v: error: libguestfs error: could not create appliance through libvirt. Try running qemu directly without libvirt using this environment variable: export LIBGUESTFS_BACKEND=direct Original error from libvirt: Cannot access backing file '/root/test.img' of storage file '/var/tmp/v2vovl440814.qcow2' (as uid:107, gid:107): Permission denied [code=38 domain=18] If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...]
It seems that the error has propagated as it was requested. /root/ is not accessible to uid 107.
I would say the new error message is accurate now: Original error from libvirt: Cannot access backing file '/root/test.img' of storage file '/var/tmp/v2vovl6324e4.qcow2' (as uid:107, gid:107): Permission denied [code=38 domain=18] So this bug can be VERIFIED.
(In reply to Richard W.M. Jones from comment #26) > I would say the new error message is accurate now: > > Original error from libvirt: Cannot access backing file '/root/test.img' of > storage file '/var/tmp/v2vovl6324e4.qcow2' (as uid:107, gid:107): > Permission denied [code=38 domain=18] > > So this bug can be VERIFIED. Refer to comment 26,set the bug to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html