Bug 1717088

+++ This bug was initially created as a clone of Bug #1698437 +++

Description of problem:
Need to set environment variable "export LIBGUESTFS_BACKEND=direct" during virt-v2v openstack conversion if selinux is enabled and use VMware VDDK as a transport

Version-Release number of selected component (if applicable):
virt-v2v-1.40.2-2.el7.x86_64
libguestfs-1.40.2-2.el7.x86_64
libvirt-client-4.5.0-10.el7_6.6.x86_64
qemu-kvm-rhev-2.12.0-18.el7_6.4.x86_64
nbdkit-1.8.0-1.el7.x86_64
nbdkit-plugin-vddk-1.8.0-1.el7.x86_64



How reproducible:
100%

Steps to Reproduce:

Set up environment:
1.Prepare a guest which have installed python2-openstackclient and virt-v2v on openstack environment

# openstack server list
+--------------------------------------+-------------------------------+--------+-----------------------+-----------------------+-----------+
| ID                                   | Name                          | Status | Networks              | Image                 | Flavor    |
+--------------------------------------+-------------------------------+--------+-----------------------+-----------------------+-----------+
| ff074bf3-51ed-4897-9a81-2ba97a69d81e | rhel7.6-v2v-conversion-server | ACTIVE | public01=10.73.224.5  | esx6.7-rhel7.6-x86_64 | m1.medium |
+--------------------------------------+-------------------------------+--------+-----------------------+-----------------------+-----------+

2.Copy keystone_admin from openstack server to guest "rhel7.6-v2v-conversion-server", then source keystone_admin to authenticate with openstack

#source keystone_admin


Scenario1: enable selinux on v2v conversion appliance of openstack
1.1 Enable selinux on v2v
# getenforce
Enforcing

1.2 Convert a guest from vmware to openstack by virt-v2v with vddk
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -n default esx6.7-rhel6.10-x86_64 --password-file /tmp/passwd -o openstack -oo server-id=rhel7.6-v2v-conversion-server
[   1.8] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel6.10-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[   3.7] Creating an overlay to protect the source from being modified
[   7.1] Opening the overlay
virt-v2v: error: libguestfs error: could not create appliance through 
libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct

Original error from libvirt: internal error: process exited while 
connecting to monitor: 2019-04-10 11:06:44.854+0000: Domain id=27 is 
tainted: custom-argv
2019-04-10T11:06:45.003428Z qemu-kvm: -drive 
file=/var/tmp/v2vovl946f1d.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=unsafe,copy-on-read=on,discard=unmap: 
Could not open backing file: Failed to connect socket 
/var/tmp/vddk.V77HRV/nbdkit0.sock: Permission denied [code=1 int1=-1]

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]



1.3 Convert a guest from vmware to openstack by virt-v2v without vddk
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1  esx6.7-rhel6.10-x86_64 --password-file /tmp/passwd -o openstack -oo server-id=rhel7.6-v2v-conversion-server
[   1.9] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel6.10-x86_64
[   4.1] Creating an overlay to protect the source from being modified
[   5.1] Opening the overlay
[  62.2] Inspecting the overlay
[ 225.9] Checking for sufficient free disk space in the guest
[ 225.9] Estimating space required on target for each disk
[ 225.9] Converting Red Hat Enterprise Linux Server release 6.10 (Santiago) to run on KVM
^C


Scenario2: disable selinux on v2v conversion appliance of openstack
2.1 Disable selinux on v2v
# getenforce 
Permissive

2.2 Convert a guest from vmware to openstack by virt-v2v with vddk
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -n default esx6.7-rhel6.10-x86_64 --password-file /tmp/passwd -o openstack -oo server-id=rhel7.6-v2v-conversion-server
[   1.8] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel6.10-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[   3.6] Creating an overlay to protect the source from being modified
[   6.8] Opening the overlay
[  59.3] Inspecting the overlay
[ 125.2] Checking for sufficient free disk space in the guest
[ 125.2] Estimating space required on target for each disk
[ 125.2] Converting Red Hat Enterprise Linux Server release 6.10 (Santiago) to run on KVM
^C


2.3 Convert a guest from vmware to openstack by virt-v2v without vddk
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1  esx6.7-rhel6.10-x86_64 --password-file /tmp/passwd -o openstack -oo server-id=rhel7.6-v2v-conversion-server
[   1.7] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel6.10-x86_64
[   3.8] Creating an overlay to protect the source from being modified
[   4.6] Opening the overlay
[  61.9] Inspecting the overlay
[ 226.5] Checking for sufficient free disk space in the guest
[ 226.5] Estimating space required on target for each disk
[ 226.5] Converting Red Hat Enterprise Linux Server release 6.10 (Santiago) to run on KVM
^C


Actual results:
As above description

Expected results:
There is no need to set environment variable "export LIBGUESTFS_BACKEND=direct" during virt-v2v openstack conversion if selinux is enabled and convert with vddk

Additional info:

--- Additional comment from Richard W.M. Jones on 2019-04-10 17:29:17 CEST ---

Could you grab the SELinux audit messages.  The way to do it is:

(1) Run the virt-v2v command so it fails.

(2) Within 10 minutes of (1), run:

ausearch -m avc -ts recent

The ausearch command should print the SELinux failures.

--- Additional comment from mxie on 2019-04-11 04:35:52 CEST ---

(In reply to Richard W.M. Jones from comment #2)
> Could you grab the SELinux audit messages.  The way to do it is:
> 
> (1) Run the virt-v2v command so it fails.
> 
> (2) Within 10 minutes of (1), run:
> 
> ausearch -m avc -ts recent
> 
> The ausearch command should print the SELinux failures.

# ausearch -m avc -ts recent----
time->Wed Apr 10 22:34:01 2019
type=PROCTITLE msg=audit(1554950041.115:50316): proctitle=2F7573722F6C6962657865632F71656D752D6B766D002D6E616D650067756573743D677565737466732D673834696D3061683279686734346A352C64656275672D746872656164733D6F6E002D53002D6F626A656374007365637265742C69643D6D61737465724B6579302C666F726D61743D7261772C66696C653D2F766172
type=SYSCALL msg=audit(1554950041.115:50316): arch=c000003e syscall=42 success=no exit=-13 a0=f a1=7fff31a9fed0 a2=6e a3=21 items=0 ppid=1 pid=11198 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_tcg_t:s0:c814,c876 key=(null)
type=AVC msg=audit(1554950041.115:50316): avc:  denied  { connectto } for  pid=11198 comm="qemu-kvm" path="/var/tmp/vddk.7JkpoI/nbdkit0.sock" scontext=system_u:system_r:svirt_tcg_t:s0:c814,c876 tcontext=system_u:object_r:svirt_t:s0 tclass=unix_stream_socket permissive=0

--- Additional comment from Richard W.M. Jones on 2019-04-11 09:47:32 CEST ---

audit2allow says:

#============= svirt_tcg_t ==============
allow svirt_tcg_t svirt_t:unix_stream_socket connectto;

--- Additional comment from Richard W.M. Jones on 2019-04-11 09:50:12 CEST ---

So it's similar to bug 1632231 but not quite the same.  However libvirt is still failing
to label the socket correctly so I'm reassigning this one to libvirt.

--- Additional comment from Richard W.M. Jones on 2019-04-11 10:01:43 CEST ---

I'm able to reproduce this fairly easily.  Note that you will first need to do:

# setsebool -P virt_use_xserver=1

to get past another error.

Run these commands *AS NON ROOT* to reproduce the problem:

  cd /var/tmp
  rm -f sock overlay.qcow2
  nbdkit -f --exit-with-parent -U sock memory size=64M &
  sleep 5
  qemu-img create -f qcow2 overlay.qcow2 -b nbd:unix:$PWD/sock
  guestfish -a overlay.qcow2 run : list-devices

For me the SELinux error is slightly different, but I think that's just because
I'm using KVM instead of TCG:

time->Thu Apr 11 08:58:47 2019
type=PROCTITLE msg=audit(1554969527.537:9927): proctitle=2F7573722F6C6962657865632F71656D752D6B766D002D6E616D650067756573743D677565737466732D756F6A6A6268313136337478717A747A2C64656275672D746872656164733D6F6E002D53002D6F626A656374007365637265742C69643D6D61737465724B6579302C666F726D61743D7261772C66696C653D2F686F6D
type=SYSCALL msg=audit(1554969527.537:9927): arch=c000003e syscall=42 success=no exit=-13 a0=10 a1=7ffd85a58850 a2=6e a3=ffffffc0 items=0 ppid=1 pid=20635 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=750 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=unconfined_u:unconfined_r:svirt_t:s0:c49,c502 key=(null)
type=AVC msg=audit(1554969527.537:9927): avc:  denied  { connectto } for  pid=20635 comm="qemu-kvm" path="/var/tmp/sock" scontext=unconfined_u:unconfined_r:svirt_t:s0:c49,c502 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0


#============= svirt_t ==============
allow svirt_t unconfined_t:unix_stream_socket connectto;

--- Additional comment from Daniel Berrange on 2019-04-11 13:14:45 CEST ---

It isn't entirely clear to me that we should be relabelling the sockets in this case. In fact I'm not sure its even possible because UNIX sockets have two distinct objects - the path on disk and the FD in memory. IIRC, they have distinct labels and setting label on the path on disk won't change the label of the server FD that nbdkit holds open.

One option would to launch nbdkit under a particular SELinux domain such that its sockets automatically get a "nbdkit_sock_t" type and then just grant svirt_image_t the permission to connect to nbdkit_sock_t.

Obviously this is not a fine grained restriction though since it won't include the per-VM unique MCS. 

There's a chicken & egg problem with improving that - we can't launch nbdkit with a MCS setting until QEMU is launched, because we don't know the MCS setting to use. But we can't launch QEMU until nbdkit is running.

Assuming nbdkit used some kind of authentication (TLS PSK for example) the missing MCS wouldn't be the end of the world.

--- Additional comment from Richard W.M. Jones on 2019-04-11 13:25:48 CEST ---

So this is a good point - we in fact already have an nbdkit flag
for relabelling sockets (--selinux-label).  It basically runs this
function:

https://github.com/libguestfs/nbdkit/blob/6c70ff4de085dd1ff5815cf0acc446cd256750e3/server/sockets.c#L57

As you say there are really *two* labels, and this only affects one of them.

However in the case of this bug (not my reproducer) we're already using this flag.

If you look in the log file attached to this bug you'll see that the nbdkit
command being run is:

LD_LIBRARY_PATH=/home/vmware-vix-disklib-distrib/lib64 'nbdkit' '--verbose' '--readonly' '--foreground' '--exit-with-parent' '--newstyle' '--exportname' '/' '--selinux-label' 'system_u:object_r:svirt_t:s0' 'vddk' 'server=10.73.73.141' 'user=root' 'password=+/tmp/passwd' 'vm=moref=vm-609' 'libdir=/home/vmware-vix-disklib-distrib' 'thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA' '--pidfile' '/var/tmp/vddk.KxTLgY/nbdkit0.pid' '--unix' '/var/tmp/vddk.KxTLgY/nbdkit0.sock' 'file=[esx6.7] esx6.7-rhel6.10-x86_64/esx6.7-rhel6.10-x86_64.vmdk'

I modified my reproducer to add --selinux-label=system_u:object_r:svirt_t:s0 and the reproducer
now *works* (which just means for some reason it's not a good reproducer - the bug is evidently
still present because virt-v2v uses that flag and it doesn't work).

--- Additional comment from Richard W.M. Jones on 2019-04-11 13:30:25 CEST ---

> There's a chicken & egg problem with improving that - we can't launch nbdkit with
> a MCS setting until QEMU is launched, because we don't know the MCS setting to use.
> But we can't launch QEMU until nbdkit is running.

Slightly off-topic but this is I think another reason why if we're going to
split out block drivers from qemu then we need to have libvirt coordinate
qemu + the other process.  There's currently ongoing discussion as to whether
split block drivers would use NBD or vhost-user as a protocol, there being
(IMO) swings and roundabouts both ways.  vhost-user isn't really a protocol,
rather a collection of Linux features brought together, but it is likely to be faster.

--- Additional comment from Daniel Berrange on 2019-04-11 13:41:02 CEST ---

Oh, i just noticed the AVC from comment #3 is different from the AVC in comment #6. The one in comment #3 makes sense given that you passed "selinux-label=system_u:object_r:svirt_t:s0"

type=AVC msg=audit(1554950041.115:50316): avc:  denied  { connectto } for  pid=11198 comm="qemu-kvm" path="/var/tmp/vddk.7JkpoI/nbdkit0.sock" scontext=system_u:system_r:svirt_tcg_t:s0:c814,c876 tcontext=system_u:object_r:svirt_t:s0 tclass=unix_stream_socket permissive=0

IIRC,  when "system_u:object_r:svirt_t:s0" is used with a disk image,  any QEMU is permitted to write to it (ie the shared-writable disk use case).

I guess we probably just need this to work the same way for UNIX sockets, which would need a selinux policy change.

--- Additional comment from Martin Kletzander on 2019-05-22 14:43:53 CEST ---

So what you are saying is that the only problem is the fact that the actioon being denied is "connectto", otherwise it's fine?  Since we still did not start shipping our own policy, can I switch this BZ to selinux-policy?

--- Additional comment from Daniel Berrange on 2019-05-22 14:51:50 CEST ---

Yes, I think we need "connectto" allowed for svirt_t  target.

--- Additional comment from Martin Kletzander on 2019-05-22 15:23:56 CEST ---

Actually, maybe it would be enough to use a different label.

See possibilities for svirt_t:

# sesearch -A -s svirt_t -c unix_stream_socket -p connectto
allow domain setrans_t:unix_stream_socket connectto;
allow domain unconfined_service_t:unix_stream_socket { connectto getattr };
allow svirt_t openvswitch_t:unix_stream_socket connectto;
allow svirt_t sssd_t:unix_stream_socket connectto;
allow svirt_t svirt_t:unix_stream_socket { accept append bind connect connectto create getattr getopt ioctl listen lock read setattr setopt shutdown write };
allow syslog_client_type kernel_t:unix_stream_socket { connectto getattr };
allow syslog_client_type syslogd_t:unix_stream_socket connectto;
allow virt_domain pcscd_t:unix_stream_socket connectto; [ virt_use_pcscd ]:True
allow virt_domain sanlock_t:unix_stream_socket connectto; [ virt_use_sanlock ]:True
allow virt_domain svirt_socket_t:unix_stream_socket { accept append bind connect connectto create getattr getopt ioctl listen lock read setattr setopt shutdown write };
allow virt_domain virtd_t:unix_stream_socket { accept connectto getattr getopt read write };
allow virt_domain xserver_t:unix_stream_socket connectto; [ virt_use_xserver ]:True

and for svirt_tcg_t:

# sesearch -A -s svirt_tcg_t -c unix_stream_socket -p connectto
allow domain setrans_t:unix_stream_socket connectto;
allow domain unconfined_service_t:unix_stream_socket { connectto getattr };
allow svirt_tcg_t sssd_t:unix_stream_socket connectto;
allow svirt_tcg_t svirt_tcg_t:unix_stream_socket { accept append bind connect connectto create getattr getopt ioctl listen lock read setattr setopt shutdown write };
allow syslog_client_type kernel_t:unix_stream_socket { connectto getattr };
allow syslog_client_type syslogd_t:unix_stream_socket connectto;
allow virt_domain pcscd_t:unix_stream_socket connectto; [ virt_use_pcscd ]:True
allow virt_domain sanlock_t:unix_stream_socket connectto; [ virt_use_sanlock ]:True
allow virt_domain svirt_socket_t:unix_stream_socket { accept append bind connect connectto create getattr getopt ioctl listen lock read setattr setopt shutdown write };
allow virt_domain virtd_t:unix_stream_socket { accept connectto getattr getopt read write };
allow virt_domain xserver_t:unix_stream_socket connectto; [ virt_use_xserver ]:True

The most fitting seems to be svirt_t or svirt_tcg_t, but it seems weird that you'd have to select the right one based on whether the VM is running in KVM or not.  So I guess better choice would be svirt_socket_t.

Is it possible that you tested it with different label (I see two different ones in this BZ)?

--- Additional comment from Richard W.M. Jones on 2019-05-23 16:33:44 CEST ---

I'm afraid I have no idea.  I don't think anyone tried relabelling the socket.

--- Additional comment from Lukas Vrabec on 2019-05-27 11:26:46 CEST ---

Hi All, 

Could you please try your scenario after this local testing policy is loaded? 

# cat svirt_tcg_svirt_connectto.cil 
(allow svirt_tcg_t svirt_t (unix_stream_socket (connectto)))

# semodule -i svirt_tcg_svirt_connectto.cil

Thanks,
Lukas.

--- Additional comment from Martin Kletzander on 2019-05-27 11:26:48 CEST ---

According to our tests proper label exists, but it was not used, so it will probably be fixed in virt-v2v.

--- Additional comment from mxie on 2019-05-27 12:43:07 CEST ---

(In reply to Lukas Vrabec from comment #18)
> Hi All, 
> 
> Could you please try your scenario after this local testing policy is
> loaded? 
> 
> # cat svirt_tcg_svirt_connectto.cil 
> (allow svirt_tcg_t svirt_t (unix_stream_socket (connectto)))
> 
> # semodule -i svirt_tcg_svirt_connectto.cil
> 
> Thanks,
> Lukas.

After setting above on v2v conversion appliance guest, still do not need to set environment variable "export LIBGUESTFS_BACKEND=direct" during virt-v2v openstack conversion with comment15's scratch builds. which means the bug will not be happened.

--- Additional comment from Martin Kletzander on 2019-05-27 12:45:31 CEST ---

(In reply to Lukas Vrabec from comment #18)
Sorry, this was identified as fixable one layer above by using the svirt_socket_t label.  If that is not accepted upstream, we'll keep you posted.

If, in any future is the above the case, it would need the same thing other way around to work in all scenarios.  That is:

(allow svirt_t svirt_tcg_t (unix_stream_socket (connectto)))

But I hope it is not the case.

--- Additional comment from Martin Kletzander on 2019-05-27 14:13:15 CEST ---

Patch proposed upstream:

https://www.redhat.com/archives/libguestfs/2019-May/msg00204.html

--- Additional comment from Pino Toscano on 2019-06-04 18:45:14 CEST ---

This was fixed upstream with commit c2918b8b74506523a723b804d452816a059c5e50.