Hide Forgot
Created attachment 1850233 [details] import-50021ef8-643e-45ac-89d0-61fa381bb125-20220111T202633.log Description of problem: Failed to import VM when selecting OVA as a source on RHV webadmin Version-Release number of selected component (if applicable): RHV NODE: libvirt-7.10.0-1.module+el8.6.0+13502+4f24a11d.x86_64 virt-v2v-1.42.0-18.module+el8.6.0+13447+4b5d0856.x86_64 libguestfs-1.44.0-5.module+el8.6.0+13732+b2b9b31d.x86_64 vdsm-4.40.100.2-1.el8ev.x86_64 RHV Server: Red Hat Virtualization Manager Web Administration Software Version:4.4.10.2-0.1.el8ev How reproducible: 100% Steps to Reproduce: 1. Login with administration portal, System->Virtual Machines. 2. Click "Import" button, then "Import Virtual Machine(s)" window pop up. Then choose "Data Center" and "Source" as follows. ->Data Center: Default ->Source: Virtual Appliance (OVA) Fill the "Host" and "Path" contents, then click the "Load" button. ->Host: Using the default rhv node ->File path: /home/ 3. After loading, then select guest name list in "Virtual Machines on Source" to import, then click "Next" , then click "OK" in next page, then guest import starts. VM name - esx6_5-rhel6.9-x86_64 Actual results: Failed to import VM via OVA on RHV webadmin with the following error: Failed to import Vm esx6_5-rhel6.9-x86_64 to Data Center Default, Cluster Default # cat import-50021ef8-643e-45ac-89d0-61fa381bb125-20220111T202633.log ... libguestfs: trace: v2v: launch = -1 (error) { "message": "libguestfs error: could not create appliance through libvirt.\n\nTry running qemu directly without libvirt using this environment variable:\nexport LIBGUESTFS_BACKEND=direct\n\nOriginal error from libvirt: internal error: process exited while connecting to monitor: 2022-01-11T12:26:40.398699Z qemu-kvm: -blockdev {\"driver\":\"file\",\"filename\":\"/home//esx6_5-rhel6.9-x86_64.ova\",\"node-name\":\"libvirt-4-storage\",\"cache\":{\"direct\":false,\"no-flush\":true},\"auto-read-only\":true,\"discard\":\"unmap\"}: Could not open '/home//esx6_5-rhel6.9-x86_64.ova': Permission denied [code=1 int1=-1]", "timestamp": "2022-01-11T20:26:40.405338319+08:00", "type": "error" } virt-v2v: error: libguestfs error: could not create appliance through libvirt. Try running qemu directly without libvirt using this environment variable: export LIBGUESTFS_BACKEND=direct Original error from libvirt: internal error: process exited while connecting to monitor: 2022-01-11T12:26:40.398699Z qemu-kvm: -blockdev {"driver":"file","filename":"/home//esx6_5-rhel6.9-x86_64.ova","node-name":"libvirt-4-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: Could not open '/home//esx6_5-rhel6.9-x86_64.ova': Permission denied ... Expected results: Fix it. Additional info: I can convert VM from OVA file on the rhv node successfully. # virt-v2v -i ova /home/esx6_5-rhel6.9-x86_64.ova -o null [ 0.0] Opening the source -i ova /home/esx6_5-rhel6.9-x86_64.ova virt-v2v: warning: making OVA directory public readable to work around libvirt bug https://bugzilla.redhat.com/1045069 [ 4.3] Creating an overlay to protect the source from being modified [ 4.4] Opening the overlay libvirt needs authentication to connect to libvirt URI qemu:///system (see also: http://libvirt.org/auth.html http://libvirt.org/uri.html) Please enter your authentication name: test Please enter your password: [ 12.7] Inspecting the overlay [ 14.9] Checking for sufficient free disk space in the guest [ 14.9] Estimating space required on target for each disk [ 14.9] Converting Red Hat Enterprise Linux Server release 6.9 (Santiago) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 68.7] Mapping filesystem data to avoid copying unused and blank areas [ 69.2] Closing the overlay [ 70.6] Assigning disks to buses [ 70.6] Checking if the guest needs BIOS or UEFI to boot [ 70.6] Initializing the target -o null [ 70.6] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw) (100.00/100%) [ 84.3] Creating output metadata [ 84.3] Finishing off
This seems most likely to be either: (1) regular permissions or (2) SELinux To exclude (2), can you see if there are any SELinux events around the time of the failure? They should be present in the journal (use "journalctl" command). But I think it's most likely to be (1). RHV runs everything as the VDSM user (UID 36). Can the VDSM user access files in /home? What are the permissions on /home? What are the permissions on the file? Does it work if you do: chmod ugo+rx /home chmod ugo+rx /home/esx6_5-rhel6.9-x86_64.ova
(In reply to Richard W.M. Jones from comment #1) > This seems most likely to be either: > > (1) regular permissions or > > (2) SELinux > > To exclude (2), can you see if there are any SELinux events around the > time of the failure? They should be present in the journal (use "journalctl" > command). 1. After I copy the OVA file from our shared NFS server, the permission of the file is as follows: # ll /home/esx6_5-rhel6.9-x86_64.ova -rw-r--r--. 1 root root 1634396160 Jan 13 10:31 /home/esx6_5-rhel6.9-x86_64.ova Then I failed to import VM from this OVA file on RHV. I get the journalctl log with "journalctl -f" command. """ Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/home//esx6_5-rhel6.9-x86_64.ova' (errno 1) Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: Unable to read from monitor: Connection reset by peer Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: internal error: qemu unexpectedly closed the monitor: 2022-01-13T02:33:18.856322Z qemu-kvm: -blockdev {"driver":"file","filename":"/home//esx6_5-rhel6.9-x86_64.ova","node-name":"libvirt-4-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: Could not open '/home//esx6_5-rhel6.9-x86_64.ova': Permission denied Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: internal error: process exited while connecting to monitor: 2022-01-13T02:33:18.856322Z qemu-kvm: -blockdev {"driver":"file","filename":"/home//esx6_5-rhel6.9-x86_64.ova","node-name":"libvirt-4-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: Could not open '/home//esx6_5-rhel6.9-x86_64.ova': Permission denied Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /var/tmp/v2vovl17b887.qcow2 Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /tmp/libguestfsErBeys/overlay1.qcow2 Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /tmp/libguestfsaFmgBu/console.sock Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /tmp/libguestfsaFmgBu/guestfsd.sock Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /var/tmp/.guestfs-36/appliance.d/kernel Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /var/tmp/.guestfs-36/appliance.d/initrd Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com vdsm[623418]: ERROR Job 'b2e7c762-4a11-4957-9c4b-9763f7b80645' failed Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/v2v.py", line 864, in _run self._import() File "/usr/lib/python3.6/site-packages/vdsm/v2v.py", line 889, in _import self._proc.returncode)) vdsm.v2v.V2VProcessError: Job 'b2e7c762-4a11-4957-9c4b-9763f7b80645' process failed exit-code: 1 Jan 13 10:33:21 bootp-73-194-236.rhts.eng.pek2.redhat.com vdsm[623418]: WARN Nic ovirtmgmt is not configured for IPv6 monitoring. Jan 13 10:33:21 bootp-73-194-236.rhts.eng.pek2.redhat.com dbus-daemon[1032]: [system] Activating service name='org.fedoraproject.Setroubleshootd' requested by ':1.1700' (uid=0 pid=989 comm="/usr/sbin/sedispatch " label="system_u:system_r:auditd_t:s0") (using servicehelper) Jan 13 10:33:22 bootp-73-194-236.rhts.eng.pek2.redhat.com dbus-daemon[1032]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd' Jan 13 10:33:23 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: AnalyzeThread.run(): Cancel pending alarm Jan 13 10:33:23 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: failed to retrieve rpm info for /home/esx6_5-rhel6.9-x86_64.ova Jan 13 10:33:23 bootp-73-194-236.rhts.eng.pek2.redhat.com dbus-daemon[1032]: [system] Activating service name='org.fedoraproject.SetroubleshootPrivileged' requested by ':1.2623' (uid=979 pid=715968 comm="/usr/libexec/platform-python -Es /usr/sbin/setroub" label="system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023") (using servicehelper) Jan 13 10:33:23 bootp-73-194-236.rhts.eng.pek2.redhat.com dbus-daemon[1032]: [system] Successfully activated service 'org.fedoraproject.SetroubleshootPrivileged' Jan 13 10:33:24 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: SELinux is preventing /usr/libexec/qemu-kvm from read access on the file /home/esx6_5-rhel6.9-x86_64.ova. For complete SELinux messages run: sealert -l be72b649-b8d0-4284-8de1-94d21a4ac403 Jan 13 10:33:24 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: SELinux is preventing /usr/libexec/qemu-kvm from read access on the file /home/esx6_5-rhel6.9-x86_64.ova. ***** Plugin restorecon (62.6 confidence) suggests ************************ If you want to fix the label. /home/esx6_5-rhel6.9-x86_64.ova default label should be default_t. Then you can run restorecon. The access attempt may have been stopped due to insufficient permissions to access a parent directory in which case try to change the following command accordingly. Do # /sbin/restorecon -v /home/esx6_5-rhel6.9-x86_64.ova ***** Plugin qemu_file_image (37.7 confidence) suggests ******************* If esx6_5-rhel6.9-x86_64.ova is a virtualization target Then you need to change the label on esx6_5-rhel6.9-x86_64.ova' Do # semanage fcontext -a -t virt_image_t '/home/esx6_5-rhel6.9-x86_64.ova' # restorecon -v '/home/esx6_5-rhel6.9-x86_64.ova' ***** Plugin catchall (1.12 confidence) suggests ************************** If you believe that qemu-kvm should be allowed read access on the esx6_5-rhel6.9-x86_64.ova file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'qemu-kvm' --raw | audit2allow -M my-qemukvm # semodule -X 300 -i my-qemukvm.pp Jan 13 10:33:24 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: AnalyzeThread.run(): Set alarm timeout to 10 """ > > But I think it's most likely to be (1). RHV runs everything as the VDSM > user (UID 36). Can the VDSM user access files in /home? What are the > permissions on /home? What are the permissions on the file? Does it work > if you do: > > chmod ugo+rx /home > chmod ugo+rx /home/esx6_5-rhel6.9-x86_64.ova 2. Change the permission of the OVA file. # chmod ugo+rx /home/esx6_5-rhel6.9-x86_64.ova # ll /home/esx6_5-rhel6.9-x86_64.ova -rwxr-xr-x. 1 root root 1634396160 Jan 13 10:31 /home/esx6_5-rhel6.9-x86_64.ova Test result - Still failed to import VM via OVA file. 3. And as you guessed, if I change the owner of the OVA file to "36:36", then I can import VM from OVA file successfully. # chown 36:36 /home/esx6_5-rhel6.9-x86_64.ova # ll /home/esx6_5-rhel6.9-x86_64.ova -rwxr-xr-x. 1 vdsm kvm 1634396160 Jan 13 10:31 /home/esx6_5-rhel6.9-x86_64.ova Test result - PASS
Can I check - what are the permissions on the /home directory?
(In reply to Richard W.M. Jones from comment #3) > Can I check - what are the permissions on the /home directory? # ll / ... drwxr-xr-x. 7 root root 4096 Jan 13 10:30 home Does this satisfy your answer, thanks.
It sounds to me as if RHV should try to set the label on the OVA file before importing it. Is this a new bug which has just started to happen, or is this a new test scenario that we've not tested before? Arik do you have any idea about this? Does vdsm try to set a label on OVA files before importing them?
(In reply to Richard W.M. Jones from comment #5) > Arik do you have any idea about this? Does vdsm try to set a label > on OVA files before importing them? Looking at the VDSM code, it doesn't seem like VDSM tries to set a label on the OVA file before importing it (Tomas, can you confirm? maybe I'm missing it) But note that the oVirt/RHV documentation is very strict about ensuring that the OVA file has permissions allowing read/write access to the qemu user (UID 36) and the kvm group (GID 36): https://www.ovirt.org/documentation/virtual_machine_management_guide/#Importing_a_virtual_machine_from_a_host
(In reply to Arik from comment #6) > (In reply to Richard W.M. Jones from comment #5) > > Arik do you have any idea about this? Does vdsm try to set a label > > on OVA files before importing them? > > Looking at the VDSM code, it doesn't seem like VDSM tries to set a label on > the OVA file before importing it (Tomas, can you confirm? maybe I'm missing > it) No, VDSM should not change any labels. Could we see what is the security context on the OVA after copying from NFS and after (unsuccessful) import? > But note that the oVirt/RHV documentation is very strict about ensuring that > the OVA file has permissions allowing read/write access to the qemu user > (UID 36) and the kvm group (GID 36): > https://www.ovirt.org/documentation/virtual_machine_management_guide/ > #Importing_a_virtual_machine_from_a_host As confirmed in comment #2 it succeeds: > 3. And as you guessed, if I change the owner of the OVA file to "36:36", then I can import VM from OVA file successfully. So I would say it all works fine.
My assumption is virt-v2v uses libvirtd from user's session which prevents it changing security context for root owned file.
(In reply to Tomáš Golembiovský from comment #8) > My assumption is virt-v2v uses libvirtd from user's session which prevents > it changing security context for root owned file. I'm going to guess that vdsm runs virt-v2v as its user (ie. 36:36), in which case it will be running a session libvirtd as the same user. Since the file was originally owned by root, it does seem plausible that the session libvirtd would not be able to change the file context.
(In reply to Tomáš Golembiovský from comment #7) > (In reply to Arik from comment #6) > > (In reply to Richard W.M. Jones from comment #5) > > > Arik do you have any idea about this? Does vdsm try to set a label > > > on OVA files before importing them? > > > > Looking at the VDSM code, it doesn't seem like VDSM tries to set a label on > > the OVA file before importing it (Tomas, can you confirm? maybe I'm missing > > it) > > No, VDSM should not change any labels. > > Could we see what is the security context on the OVA after copying from NFS > and after (unsuccessful) import? Hi Tomas, I may find that I will get different final results when copying the OVA file from NFS server to the local '/home' directory. 1.Copying with '-a' option -> Succeeded to import from OVA # cp -a ova-images/esx6_5-rhel6.9-x86_64.ova /home/ # ll -Z esx6_5-rhel6.9-x86_64.ova -rw-r--r--. 1 nobody nobody system_u:object_r:nfs_t:s0 1634396160 Jun 6 2018 esx6_5-rhel6.9-x86_64.ova Test result: Converting OVA file to RHV successfully, Checking the permission again, getting the same test result as above. # ll -Z esx6_5-rhel6.9-x86_64.ova -rw-r--r--. 1 nobody nobody system_u:object_r:nfs_t:s0 1634396160 Jun 6 2018 esx6_5-rhel6.9-x86_64.ova 2. Copying direclty without any extra option -> Failed to import from OVA. # cp /opt/ova-images/esx6_5-rhel6.9-x86_64.ova /home/ # ll -Z esx6_5-rhel6.9-x86_64.ova -rw-r--r--. 1 root root unconfined_u:object_r:home_root_t:s0 1634396160 Jan 18 10:56 esx6_5-rhel6.9-x86_64.ova Test result: Reproduce the bug issue, failed to import OVA file. And checking the permission of the OVA file again, getting the same result. # ll -Z esx6_5-rhel6.9-x86_64.ova -rw-r--r--. 1 root root unconfined_u:object_r:home_root_t:s0 1634396160 Jan 18 10:56 esx6_5-rhel6.9-x86_64.ova
Thanks, I don't see anything suspicious happening here.
Upstream in: https://github.com/libguestfs/virt-v2v/commit/3a66d0fece08897744b4a9cc62de148abbe49b49 I'm moving this to RHEL 9.1 since it's a documentation fix. (The libguestfs.org website documentation will be updated in a few days).
Verified with virt-v2v-2.0.3-1.el9.x86_64 Test step: It is just a document update. # man virt-v2v-input-vmware ... OVA: Permissions issues with oVirt/RHV import oVirt/RHV provides a graphical user interface for importing from OVA files which uses this method. It requires that RHV is able to access the OVA file which can be a problem if the file is owned by root (RHV runs as a non-root user). The suggested workaround is to copy the OVA to a public directory such as /var/tmp before doing the import and perhaps change the user and group ownership of the file. For more information see these links: • https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/virtual_machine_management_guide/index#Importing_a_virtual_machine_from_a_host • https://bugzilla.redhat.com/show_bug.cgi?id=2039597 Test Summary - The document is readable and clear, so I move this bug from ON_QA to VERIFIED status, thanks.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7968