RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2039597 - Failed to import VM when selecting OVA as a source on RHV webadmin
Summary: Failed to import VM when selecting OVA as a source on RHV webadmin
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: virt-v2v
Version: 9.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Virtualization Maintenance
QA Contact: zhoujunqin
URL:
Whiteboard: V2V_RHV_INT
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-12 02:50 UTC by zhoujunqin
Modified: 2022-11-15 10:23 UTC (History)
11 users (show)

Fixed In Version: virt-v2v-2.0.7-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-15 09:55:51 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
import-50021ef8-643e-45ac-89d0-61fa381bb125-20220111T202633.log (16.35 KB, text/plain)
2022-01-12 02:50 UTC, zhoujunqin 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-107511 0 None None None 2022-01-12 02:53:02 UTC
Red Hat Product Errata RHSA-2022:7968 0 None None None 2022-11-15 09:56:22 UTC

Description zhoujunqin 2022-01-12 02:50:29 UTC 
Created attachment 1850233 [details]
import-50021ef8-643e-45ac-89d0-61fa381bb125-20220111T202633.log

Description of problem:
Failed to import VM when selecting OVA as a source on RHV webadmin

Version-Release number of selected component (if applicable):
RHV NODE:
libvirt-7.10.0-1.module+el8.6.0+13502+4f24a11d.x86_64
virt-v2v-1.42.0-18.module+el8.6.0+13447+4b5d0856.x86_64
libguestfs-1.44.0-5.module+el8.6.0+13732+b2b9b31d.x86_64
vdsm-4.40.100.2-1.el8ev.x86_64

RHV Server:
Red Hat Virtualization Manager Web Administration
Software Version:4.4.10.2-0.1.el8ev

How reproducible:
100%

Steps to Reproduce:
1. Login with administration portal, System->Virtual Machines.
2. Click "Import" button, then "Import Virtual Machine(s)" window pop up. Then choose "Data Center" and "Source" as follows.

->Data Center: Default
->Source: Virtual Appliance (OVA)

Fill the "Host" and "Path" contents, then click the "Load" button.
->Host: Using the default rhv node
->File path: /home/

3. After loading, then select guest name list in "Virtual Machines on Source" to import, then click "Next" , then click "OK" in next page, then guest import starts.

VM name - esx6_5-rhel6.9-x86_64

Actual results:
Failed to import VM via OVA on RHV webadmin with the following error:
Failed to import Vm esx6_5-rhel6.9-x86_64 to Data Center Default, Cluster Default

# cat import-50021ef8-643e-45ac-89d0-61fa381bb125-20220111T202633.log
...
libguestfs: trace: v2v: launch = -1 (error)
{ "message": "libguestfs error: could not create appliance through libvirt.\n\nTry running qemu directly without libvirt using this environment variable:\nexport LIBGUESTFS_BACKEND=direct\n\nOriginal error from libvirt: internal error: process exited while connecting to monitor: 2022-01-11T12:26:40.398699Z qemu-kvm: -blockdev {\"driver\":\"file\",\"filename\":\"/home//esx6_5-rhel6.9-x86_64.ova\",\"node-name\":\"libvirt-4-storage\",\"cache\":{\"direct\":false,\"no-flush\":true},\"auto-read-only\":true,\"discard\":\"unmap\"}: Could not open '/home//esx6_5-rhel6.9-x86_64.ova': Permission denied [code=1 int1=-1]", "timestamp": "2022-01-11T20:26:40.405338319+08:00", "type": "error" }
virt-v2v: error: libguestfs error: could not create appliance through
libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct

Original error from libvirt: internal error: process exited while
connecting to monitor: 2022-01-11T12:26:40.398699Z qemu-kvm: -blockdev
{"driver":"file","filename":"/home//esx6_5-rhel6.9-x86_64.ova","node-name":"libvirt-4-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}:
Could not open '/home//esx6_5-rhel6.9-x86_64.ova': Permission denied
...

Expected results:
Fix it.

Additional info:
I can convert VM from OVA file on the rhv node successfully.

# virt-v2v -i ova /home/esx6_5-rhel6.9-x86_64.ova -o null
[   0.0] Opening the source -i ova /home/esx6_5-rhel6.9-x86_64.ova
virt-v2v: warning: making OVA directory public readable to work around 
libvirt bug https://bugzilla.redhat.com/1045069
[   4.3] Creating an overlay to protect the source from being modified
[   4.4] Opening the overlay
libvirt needs authentication to connect to libvirt URI qemu:///system
(see also: http://libvirt.org/auth.html http://libvirt.org/uri.html)
Please enter your authentication name: test
Please enter your password: 
[  12.7] Inspecting the overlay
[  14.9] Checking for sufficient free disk space in the guest
[  14.9] Estimating space required on target for each disk
[  14.9] Converting Red Hat Enterprise Linux Server release 6.9 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  68.7] Mapping filesystem data to avoid copying unused and blank areas
[  69.2] Closing the overlay
[  70.6] Assigning disks to buses
[  70.6] Checking if the guest needs BIOS or UEFI to boot
[  70.6] Initializing the target -o null
[  70.6] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
    (100.00/100%)
[  84.3] Creating output metadata
[  84.3] Finishing off
Comment 1 Richard W.M. Jones 2022-01-12 09:52:59 UTC 
This seems most likely to be either:

(1) regular permissions or

(2) SELinux

To exclude (2), can you see if there are any SELinux events around the
time of the failure?  They should be present in the journal (use "journalctl"
command).

But I think it's most likely to be (1).  RHV runs everything as the VDSM
user (UID 36).  Can the VDSM user access files in /home?  What are the
permissions on /home?  What are the permissions on the file?  Does it work if you do:

  chmod ugo+rx /home
  chmod ugo+rx /home/esx6_5-rhel6.9-x86_64.ova
Comment 2 zhoujunqin 2022-01-13 03:00:27 UTC 
(In reply to Richard W.M. Jones from comment #1)
> This seems most likely to be either:
> 
> (1) regular permissions or
> 
> (2) SELinux
> 
> To exclude (2), can you see if there are any SELinux events around the
> time of the failure?  They should be present in the journal (use "journalctl"
> command).

1. After I copy the OVA file from our shared NFS server, the permission of the file is as follows:
# ll /home/esx6_5-rhel6.9-x86_64.ova
-rw-r--r--. 1 root root 1634396160 Jan 13 10:31 /home/esx6_5-rhel6.9-x86_64.ova

Then I failed to import VM from this OVA file on RHV.

I get the journalctl log with "journalctl -f" command.

"""
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/home//esx6_5-rhel6.9-x86_64.ova' (errno 1)
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: Unable to read from monitor: Connection reset by peer
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: internal error: qemu unexpectedly closed the monitor: 2022-01-13T02:33:18.856322Z qemu-kvm: -blockdev {"driver":"file","filename":"/home//esx6_5-rhel6.9-x86_64.ova","node-name":"libvirt-4-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: Could not open '/home//esx6_5-rhel6.9-x86_64.ova': Permission denied
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: internal error: process exited while connecting to monitor: 2022-01-13T02:33:18.856322Z qemu-kvm: -blockdev {"driver":"file","filename":"/home//esx6_5-rhel6.9-x86_64.ova","node-name":"libvirt-4-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: Could not open '/home//esx6_5-rhel6.9-x86_64.ova': Permission denied
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /var/tmp/v2vovl17b887.qcow2
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /tmp/libguestfsErBeys/overlay1.qcow2
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /tmp/libguestfsaFmgBu/console.sock
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /tmp/libguestfsaFmgBu/guestfsd.sock
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /var/tmp/.guestfs-36/appliance.d/kernel
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com libvirtd[715860]: cannot lookup default selinux label for /var/tmp/.guestfs-36/appliance.d/initrd
Jan 13 10:33:18 bootp-73-194-236.rhts.eng.pek2.redhat.com vdsm[623418]: ERROR Job 'b2e7c762-4a11-4957-9c4b-9763f7b80645' failed
                                                                        Traceback (most recent call last):
                                                                          File "/usr/lib/python3.6/site-packages/vdsm/v2v.py", line 864, in _run
                                                                            self._import()
                                                                          File "/usr/lib/python3.6/site-packages/vdsm/v2v.py", line 889, in _import
                                                                            self._proc.returncode))
                                                                        vdsm.v2v.V2VProcessError: Job 'b2e7c762-4a11-4957-9c4b-9763f7b80645' process failed exit-code: 1
Jan 13 10:33:21 bootp-73-194-236.rhts.eng.pek2.redhat.com vdsm[623418]: WARN Nic ovirtmgmt is not configured for IPv6 monitoring.
Jan 13 10:33:21 bootp-73-194-236.rhts.eng.pek2.redhat.com dbus-daemon[1032]: [system] Activating service name='org.fedoraproject.Setroubleshootd' requested by ':1.1700' (uid=0 pid=989 comm="/usr/sbin/sedispatch " label="system_u:system_r:auditd_t:s0") (using servicehelper)
Jan 13 10:33:22 bootp-73-194-236.rhts.eng.pek2.redhat.com dbus-daemon[1032]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Jan 13 10:33:23 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: AnalyzeThread.run(): Cancel pending alarm
Jan 13 10:33:23 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: failed to retrieve rpm info for /home/esx6_5-rhel6.9-x86_64.ova
Jan 13 10:33:23 bootp-73-194-236.rhts.eng.pek2.redhat.com dbus-daemon[1032]: [system] Activating service name='org.fedoraproject.SetroubleshootPrivileged' requested by ':1.2623' (uid=979 pid=715968 comm="/usr/libexec/platform-python -Es /usr/sbin/setroub" label="system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023") (using servicehelper)
Jan 13 10:33:23 bootp-73-194-236.rhts.eng.pek2.redhat.com dbus-daemon[1032]: [system] Successfully activated service 'org.fedoraproject.SetroubleshootPrivileged'
Jan 13 10:33:24 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: SELinux is preventing /usr/libexec/qemu-kvm from read access on the file /home/esx6_5-rhel6.9-x86_64.ova. For complete SELinux messages run: sealert -l be72b649-b8d0-4284-8de1-94d21a4ac403
Jan 13 10:33:24 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: SELinux is preventing /usr/libexec/qemu-kvm from read access on the file /home/esx6_5-rhel6.9-x86_64.ova.
                                                                                  
                                                                                  *****  Plugin restorecon (62.6 confidence) suggests   ************************
                                                                                  
                                                                                  If you want to fix the label. 
                                                                                  /home/esx6_5-rhel6.9-x86_64.ova default label should be default_t.
                                                                                  Then you can run restorecon. The access attempt may have been stopped due to insufficient permissions to access a parent directory in which case try to change the following command accordingly.
                                                                                  Do
                                                                                  # /sbin/restorecon -v /home/esx6_5-rhel6.9-x86_64.ova
                                                                                  
                                                                                  *****  Plugin qemu_file_image (37.7 confidence) suggests   *******************
                                                                                  
                                                                                  If esx6_5-rhel6.9-x86_64.ova is a virtualization target
                                                                                  Then you need to change the label on esx6_5-rhel6.9-x86_64.ova'
                                                                                  Do
                                                                                  # semanage fcontext -a -t virt_image_t '/home/esx6_5-rhel6.9-x86_64.ova'
                                                                                  # restorecon -v '/home/esx6_5-rhel6.9-x86_64.ova'
                                                                                  
                                                                                  *****  Plugin catchall (1.12 confidence) suggests   **************************
                                                                                  
                                                                                  If you believe that qemu-kvm should be allowed read access on the esx6_5-rhel6.9-x86_64.ova file by default.
                                                                                  Then you should report this as a bug.
                                                                                  You can generate a local policy module to allow this access.
                                                                                  Do
                                                                                  allow this access for now by executing:
                                                                                  # ausearch -c 'qemu-kvm' --raw | audit2allow -M my-qemukvm
                                                                                  # semodule -X 300 -i my-qemukvm.pp
                                                                                  
Jan 13 10:33:24 bootp-73-194-236.rhts.eng.pek2.redhat.com setroubleshoot[715968]: AnalyzeThread.run(): Set alarm timeout to 10

"""

> 
> But I think it's most likely to be (1).  RHV runs everything as the VDSM
> user (UID 36).  Can the VDSM user access files in /home?  What are the
> permissions on /home?  What are the permissions on the file?  Does it work
> if you do:
> 
>   chmod ugo+rx /home
>   chmod ugo+rx /home/esx6_5-rhel6.9-x86_64.ova

2. Change the permission of the OVA file.

# chmod ugo+rx /home/esx6_5-rhel6.9-x86_64.ova

# ll /home/esx6_5-rhel6.9-x86_64.ova
-rwxr-xr-x. 1 root root 1634396160 Jan 13 10:31 /home/esx6_5-rhel6.9-x86_64.ova

Test result - Still failed to import VM via OVA file.

3. And as you guessed, if I change the owner of the OVA file to "36:36", then I can import VM from OVA file successfully.
# chown 36:36 /home/esx6_5-rhel6.9-x86_64.ova

# ll /home/esx6_5-rhel6.9-x86_64.ova
-rwxr-xr-x. 1 vdsm kvm 1634396160 Jan 13 10:31 /home/esx6_5-rhel6.9-x86_64.ova

Test result - PASS
Comment 3 Richard W.M. Jones 2022-01-13 12:26:55 UTC 
Can I check - what are the permissions on the /home directory?
Comment 4 zhoujunqin 2022-01-14 02:17:04 UTC 
(In reply to Richard W.M. Jones from comment #3)
> Can I check - what are the permissions on the /home directory?
# ll /
...
drwxr-xr-x.   7 root   root    4096 Jan 13 10:30 home

Does this satisfy your answer, thanks.
Comment 5 Richard W.M. Jones 2022-01-14 10:05:48 UTC 
It sounds to me as if RHV should try to set the label on the OVA file
before importing it.  Is this a new bug which has just started to happen,
or is this a new test scenario that we've not tested before?

Arik do you have any idea about this?  Does vdsm try to set a label
on OVA files before importing them?
Comment 6 Arik 2022-01-16 12:29:03 UTC 
(In reply to Richard W.M. Jones from comment #5)
> Arik do you have any idea about this?  Does vdsm try to set a label
> on OVA files before importing them?

Looking at the VDSM code, it doesn't seem like VDSM tries to set a label on the OVA file before importing it (Tomas, can you confirm? maybe I'm missing it)
But note that the oVirt/RHV documentation is very strict about ensuring that the OVA file has permissions allowing read/write access to the qemu user (UID 36) and the kvm group (GID 36):
https://www.ovirt.org/documentation/virtual_machine_management_guide/#Importing_a_virtual_machine_from_a_host
Comment 7 Tomáš Golembiovský 2022-01-17 12:09:44 UTC 
(In reply to Arik from comment #6)
> (In reply to Richard W.M. Jones from comment #5)
> > Arik do you have any idea about this?  Does vdsm try to set a label
> > on OVA files before importing them?
> 
> Looking at the VDSM code, it doesn't seem like VDSM tries to set a label on
> the OVA file before importing it (Tomas, can you confirm? maybe I'm missing
> it)

No, VDSM should not change any labels.

Could we see what is the security context on the OVA after copying from NFS and after (unsuccessful) import?



> But note that the oVirt/RHV documentation is very strict about ensuring that
> the OVA file has permissions allowing read/write access to the qemu user
> (UID 36) and the kvm group (GID 36):
> https://www.ovirt.org/documentation/virtual_machine_management_guide/
> #Importing_a_virtual_machine_from_a_host


As confirmed in comment #2 it succeeds:

> 3. And as you guessed, if I change the owner of the OVA file to "36:36", then I can import VM from OVA file successfully.

So I would say it all works fine.
Comment 8 Tomáš Golembiovský 2022-01-17 12:12:31 UTC 
My assumption is virt-v2v uses libvirtd from user's session which prevents it changing security context for root owned file.
Comment 9 Richard W.M. Jones 2022-01-17 13:11:55 UTC 
(In reply to Tomáš Golembiovský from comment #8)
> My assumption is virt-v2v uses libvirtd from user's session which prevents
> it changing security context for root owned file.

I'm going to guess that vdsm runs virt-v2v as its user (ie. 36:36), in
which case it will be running a session libvirtd as the same user.  Since
the file was originally owned by root, it does seem plausible that
the session libvirtd would not be able to change the file context.
Comment 10 zhoujunqin 2022-01-18 03:06:35 UTC 
(In reply to Tomáš Golembiovský from comment #7)
> (In reply to Arik from comment #6)
> > (In reply to Richard W.M. Jones from comment #5)
> > > Arik do you have any idea about this?  Does vdsm try to set a label
> > > on OVA files before importing them?
> > 
> > Looking at the VDSM code, it doesn't seem like VDSM tries to set a label on
> > the OVA file before importing it (Tomas, can you confirm? maybe I'm missing
> > it)
> 
> No, VDSM should not change any labels.
> 
> Could we see what is the security context on the OVA after copying from NFS
> and after (unsuccessful) import?

Hi Tomas,
I may find that I will get different final results when copying the OVA file from NFS server to the local '/home' directory.

1.Copying with '-a' option -> Succeeded to import from OVA
# cp -a ova-images/esx6_5-rhel6.9-x86_64.ova /home/

# ll -Z  esx6_5-rhel6.9-x86_64.ova 
-rw-r--r--. 1 nobody nobody system_u:object_r:nfs_t:s0 1634396160 Jun  6  2018 esx6_5-rhel6.9-x86_64.ova

Test result: Converting OVA file to RHV successfully, 

Checking the permission again, getting the same test result as above.
# ll -Z  esx6_5-rhel6.9-x86_64.ova 
-rw-r--r--. 1 nobody nobody system_u:object_r:nfs_t:s0 1634396160 Jun  6  2018 esx6_5-rhel6.9-x86_64.ova


2. Copying direclty without any extra option -> Failed to import from OVA.

# cp /opt/ova-images/esx6_5-rhel6.9-x86_64.ova /home/

# ll -Z esx6_5-rhel6.9-x86_64.ova 
-rw-r--r--. 1 root root unconfined_u:object_r:home_root_t:s0 1634396160 Jan 18 10:56 esx6_5-rhel6.9-x86_64.ova

Test result: Reproduce the bug issue, failed to import OVA file.

And checking the permission of the OVA file again, getting the same result.

# ll -Z esx6_5-rhel6.9-x86_64.ova 
-rw-r--r--. 1 root root unconfined_u:object_r:home_root_t:s0 1634396160 Jan 18 10:56 esx6_5-rhel6.9-x86_64.ova
Comment 11 Tomáš Golembiovský 2022-01-18 08:26:15 UTC 
Thanks, I don't see anything suspicious happening here.
Comment 16 Richard W.M. Jones 2022-03-07 14:20:45 UTC 
Upstream in:

https://github.com/libguestfs/virt-v2v/commit/3a66d0fece08897744b4a9cc62de148abbe49b49

I'm moving this to RHEL 9.1 since it's a documentation fix.  (The libguestfs.org
website documentation will be updated in a few days).
Comment 19 zhoujunqin 2022-04-20 04:42:41 UTC 
Verified with virt-v2v-2.0.3-1.el9.x86_64

Test step:
It is just a document update.

# man virt-v2v-input-vmware
...
   OVA: Permissions issues with oVirt/RHV import
       oVirt/RHV provides a graphical user interface for importing from OVA files which uses this
       method.  It requires that RHV is able to access the OVA file which can be a problem if the
       file is owned by root (RHV runs as a non-root user).

       The suggested workaround is to copy the OVA to a public directory such as /var/tmp before
       doing the import and perhaps change the user and group ownership of the file.

       For more information see these links:

       •   https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/virtual_machine_management_guide/index#Importing_a_virtual_machine_from_a_host

       •   https://bugzilla.redhat.com/show_bug.cgi?id=2039597


Test Summary - The document is readable and clear, so I move this bug from ON_QA to VERIFIED status, thanks.
Comment 21 errata-xmlrpc 2022-11-15 09:55:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7968
Note You need to log in before you can comment on or make changes to this bug.