Bug 1698437

Could you grab the SELinux audit messages.  The way to do it is:

(1) Run the virt-v2v command so it fails.

(2) Within 10 minutes of (1), run:

ausearch -m avc -ts recent

The ausearch command should print the SELinux failures.

(In reply to Richard W.M. Jones from comment #2)
> Could you grab the SELinux audit messages.  The way to do it is:
> 
> (1) Run the virt-v2v command so it fails.
> 
> (2) Within 10 minutes of (1), run:
> 
> ausearch -m avc -ts recent
> 
> The ausearch command should print the SELinux failures.

# ausearch -m avc -ts recent----
time->Wed Apr 10 22:34:01 2019
type=PROCTITLE msg=audit(1554950041.115:50316): proctitle=2F7573722F6C6962657865632F71656D752D6B766D002D6E616D650067756573743D677565737466732D673834696D3061683279686734346A352C64656275672D746872656164733D6F6E002D53002D6F626A656374007365637265742C69643D6D61737465724B6579302C666F726D61743D7261772C66696C653D2F766172
type=SYSCALL msg=audit(1554950041.115:50316): arch=c000003e syscall=42 success=no exit=-13 a0=f a1=7fff31a9fed0 a2=6e a3=21 items=0 ppid=1 pid=11198 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_tcg_t:s0:c814,c876 key=(null)
type=AVC msg=audit(1554950041.115:50316): avc:  denied  { connectto } for  pid=11198 comm="qemu-kvm" path="/var/tmp/vddk.7JkpoI/nbdkit0.sock" scontext=system_u:system_r:svirt_tcg_t:s0:c814,c876 tcontext=system_u:object_r:svirt_t:s0 tclass=unix_stream_socket permissive=0

audit2allow says:

#============= svirt_tcg_t ==============
allow svirt_tcg_t svirt_t:unix_stream_socket connectto;

So it's similar to bug 1632231 but not quite the same.  However libvirt is still failing
to label the socket correctly so I'm reassigning this one to libvirt.

I'm able to reproduce this fairly easily.  Note that you will first need to do:

# setsebool -P virt_use_xserver=1

to get past another error.

Run these commands *AS NON ROOT* to reproduce the problem:

  cd /var/tmp
  rm -f sock overlay.qcow2
  nbdkit -f --exit-with-parent -U sock memory size=64M &
  sleep 5
  qemu-img create -f qcow2 overlay.qcow2 -b nbd:unix:$PWD/sock
  guestfish -a overlay.qcow2 run : list-devices

For me the SELinux error is slightly different, but I think that's just because
I'm using KVM instead of TCG:

time->Thu Apr 11 08:58:47 2019
type=PROCTITLE msg=audit(1554969527.537:9927): proctitle=2F7573722F6C6962657865632F71656D752D6B766D002D6E616D650067756573743D677565737466732D756F6A6A6268313136337478717A747A2C64656275672D746872656164733D6F6E002D53002D6F626A656374007365637265742C69643D6D61737465724B6579302C666F726D61743D7261772C66696C653D2F686F6D
type=SYSCALL msg=audit(1554969527.537:9927): arch=c000003e syscall=42 success=no exit=-13 a0=10 a1=7ffd85a58850 a2=6e a3=ffffffc0 items=0 ppid=1 pid=20635 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=750 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=unconfined_u:unconfined_r:svirt_t:s0:c49,c502 key=(null)
type=AVC msg=audit(1554969527.537:9927): avc:  denied  { connectto } for  pid=20635 comm="qemu-kvm" path="/var/tmp/sock" scontext=unconfined_u:unconfined_r:svirt_t:s0:c49,c502 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0


#============= svirt_t ==============
allow svirt_t unconfined_t:unix_stream_socket connectto;

It isn't entirely clear to me that we should be relabelling the sockets in this case. In fact I'm not sure its even possible because UNIX sockets have two distinct objects - the path on disk and the FD in memory. IIRC, they have distinct labels and setting label on the path on disk won't change the label of the server FD that nbdkit holds open.

One option would to launch nbdkit under a particular SELinux domain such that its sockets automatically get a "nbdkit_sock_t" type and then just grant svirt_image_t the permission to connect to nbdkit_sock_t.

Obviously this is not a fine grained restriction though since it won't include the per-VM unique MCS. 

There's a chicken & egg problem with improving that - we can't launch nbdkit with a MCS setting until QEMU is launched, because we don't know the MCS setting to use. But we can't launch QEMU until nbdkit is running.

Assuming nbdkit used some kind of authentication (TLS PSK for example) the missing MCS wouldn't be the end of the world.

So this is a good point - we in fact already have an nbdkit flag
for relabelling sockets (--selinux-label).  It basically runs this
function:

https://github.com/libguestfs/nbdkit/blob/6c70ff4de085dd1ff5815cf0acc446cd256750e3/server/sockets.c#L57

As you say there are really *two* labels, and this only affects one of them.

However in the case of this bug (not my reproducer) we're already using this flag.

If you look in the log file attached to this bug you'll see that the nbdkit
command being run is:

LD_LIBRARY_PATH=/home/vmware-vix-disklib-distrib/lib64 'nbdkit' '--verbose' '--readonly' '--foreground' '--exit-with-parent' '--newstyle' '--exportname' '/' '--selinux-label' 'system_u:object_r:svirt_t:s0' 'vddk' 'server=10.73.73.141' 'user=root' 'password=+/tmp/passwd' 'vm=moref=vm-609' 'libdir=/home/vmware-vix-disklib-distrib' 'thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA' '--pidfile' '/var/tmp/vddk.KxTLgY/nbdkit0.pid' '--unix' '/var/tmp/vddk.KxTLgY/nbdkit0.sock' 'file=[esx6.7] esx6.7-rhel6.10-x86_64/esx6.7-rhel6.10-x86_64.vmdk'

I modified my reproducer to add --selinux-label=system_u:object_r:svirt_t:s0 and the reproducer
now *works* (which just means for some reason it's not a good reproducer - the bug is evidently
still present because virt-v2v uses that flag and it doesn't work).

> There's a chicken & egg problem with improving that - we can't launch nbdkit with
> a MCS setting until QEMU is launched, because we don't know the MCS setting to use.
> But we can't launch QEMU until nbdkit is running.

Slightly off-topic but this is I think another reason why if we're going to
split out block drivers from qemu then we need to have libvirt coordinate
qemu + the other process.  There's currently ongoing discussion as to whether
split block drivers would use NBD or vhost-user as a protocol, there being
(IMO) swings and roundabouts both ways.  vhost-user isn't really a protocol,
rather a collection of Linux features brought together, but it is likely to be faster.

Oh, i just noticed the AVC from comment #3 is different from the AVC in comment #6. The one in comment #3 makes sense given that you passed "selinux-label=system_u:object_r:svirt_t:s0"

type=AVC msg=audit(1554950041.115:50316): avc:  denied  { connectto } for  pid=11198 comm="qemu-kvm" path="/var/tmp/vddk.7JkpoI/nbdkit0.sock" scontext=system_u:system_r:svirt_tcg_t:s0:c814,c876 tcontext=system_u:object_r:svirt_t:s0 tclass=unix_stream_socket permissive=0

IIRC,  when "system_u:object_r:svirt_t:s0" is used with a disk image,  any QEMU is permitted to write to it (ie the shared-writable disk use case).

I guess we probably just need this to work the same way for UNIX sockets, which would need a selinux policy change.

So what you are saying is that the only problem is the fact that the actioon being denied is "connectto", otherwise it's fine?  Since we still did not start shipping our own policy, can I switch this BZ to selinux-policy?

Yes, I think we need "connectto" allowed for svirt_t  target.

Actually, maybe it would be enough to use a different label.

See possibilities for svirt_t:

# sesearch -A -s svirt_t -c unix_stream_socket -p connectto
allow domain setrans_t:unix_stream_socket connectto;
allow domain unconfined_service_t:unix_stream_socket { connectto getattr };
allow svirt_t openvswitch_t:unix_stream_socket connectto;
allow svirt_t sssd_t:unix_stream_socket connectto;
allow svirt_t svirt_t:unix_stream_socket { accept append bind connect connectto create getattr getopt ioctl listen lock read setattr setopt shutdown write };
allow syslog_client_type kernel_t:unix_stream_socket { connectto getattr };
allow syslog_client_type syslogd_t:unix_stream_socket connectto;
allow virt_domain pcscd_t:unix_stream_socket connectto; [ virt_use_pcscd ]:True
allow virt_domain sanlock_t:unix_stream_socket connectto; [ virt_use_sanlock ]:True
allow virt_domain svirt_socket_t:unix_stream_socket { accept append bind connect connectto create getattr getopt ioctl listen lock read setattr setopt shutdown write };
allow virt_domain virtd_t:unix_stream_socket { accept connectto getattr getopt read write };
allow virt_domain xserver_t:unix_stream_socket connectto; [ virt_use_xserver ]:True

and for svirt_tcg_t:

# sesearch -A -s svirt_tcg_t -c unix_stream_socket -p connectto
allow domain setrans_t:unix_stream_socket connectto;
allow domain unconfined_service_t:unix_stream_socket { connectto getattr };
allow svirt_tcg_t sssd_t:unix_stream_socket connectto;
allow svirt_tcg_t svirt_tcg_t:unix_stream_socket { accept append bind connect connectto create getattr getopt ioctl listen lock read setattr setopt shutdown write };
allow syslog_client_type kernel_t:unix_stream_socket { connectto getattr };
allow syslog_client_type syslogd_t:unix_stream_socket connectto;
allow virt_domain pcscd_t:unix_stream_socket connectto; [ virt_use_pcscd ]:True
allow virt_domain sanlock_t:unix_stream_socket connectto; [ virt_use_sanlock ]:True
allow virt_domain svirt_socket_t:unix_stream_socket { accept append bind connect connectto create getattr getopt ioctl listen lock read setattr setopt shutdown write };
allow virt_domain virtd_t:unix_stream_socket { accept connectto getattr getopt read write };
allow virt_domain xserver_t:unix_stream_socket connectto; [ virt_use_xserver ]:True

The most fitting seems to be svirt_t or svirt_tcg_t, but it seems weird that you'd have to select the right one based on whether the VM is running in KVM or not.  So I guess better choice would be svirt_socket_t.

Is it possible that you tested it with different label (I see two different ones in this BZ)?

I'm afraid I have no idea.  I don't think anyone tried relabelling the socket.

Hi All, 

Could you please try your scenario after this local testing policy is loaded? 

# cat svirt_tcg_svirt_connectto.cil 
(allow svirt_tcg_t svirt_t (unix_stream_socket (connectto)))

# semodule -i svirt_tcg_svirt_connectto.cil

Thanks,
Lukas.

According to our tests proper label exists, but it was not used, so it will probably be fixed in virt-v2v.

(In reply to Lukas Vrabec from comment #18)
> Hi All, 
> 
> Could you please try your scenario after this local testing policy is
> loaded? 
> 
> # cat svirt_tcg_svirt_connectto.cil 
> (allow svirt_tcg_t svirt_t (unix_stream_socket (connectto)))
> 
> # semodule -i svirt_tcg_svirt_connectto.cil
> 
> Thanks,
> Lukas.

After setting above on v2v conversion appliance guest, still do not need to set environment variable "export LIBGUESTFS_BACKEND=direct" during virt-v2v openstack conversion with comment15's scratch builds. which means the bug will not be happened.

(In reply to Lukas Vrabec from comment #18)
Sorry, this was identified as fixable one layer above by using the svirt_socket_t label.  If that is not accepted upstream, we'll keep you posted.

If, in any future is the above the case, it would need the same thing other way around to work in all scenarios.  That is:

(allow svirt_t svirt_tcg_t (unix_stream_socket (connectto)))

But I hope it is not the case.

Patch proposed upstream:

https://www.redhat.com/archives/libguestfs/2019-May/msg00204.html

This was fixed upstream with commit c2918b8b74506523a723b804d452816a059c5e50.

Verify the bug with below builds:
virt-v2v-1.40.2-5.el7.x86_64
libguestfs-1.40.2-5.el7.x86_64
libvirt-4.5.0-21.el7.x86_64
qemu-kvm-rhev-2.12.0-32.el7.x86_64
nbdkit-1.8.0-1.el7.x86_64


Steps:
1.Prepare a guest which have installed python2-openstackclient and virt-v2v packages on openstack environment
# openstack server list
+--------------------------------------+-------------------------------+--------+-----------------------+-----------------------+-----------+
| ID                                   | Name                          | Status | Networks              | Image                 | Flavor    |
+--------------------------------------+-------------------------------+--------+-----------------------+-----------------------+-----------+
| ff074bf3-51ed-4897-9a81-2ba97a69d81e | rhel7.6-v2v-conversion-server | ACTIVE | public01=10.73.224.5  | esx6.7-rhel7.6-x86_64 | m1.medium |
+--------------------------------------+-------------------------------+--------+-----------------------+-----------------------+-----------+

2.Copy keystone_admin from openstack host to guest "rhel7.6-v2v-conversion-server", then source keystone_admin to authenticate with openstack
#source keystone_admin

3.Enable selinux on v2v conversion appliance guest and do not set direct for LIBGUESTFS_BACKEND
# getenforce
Enforcing
# echo $LIBGUESTFS_BACKEND
nothing

4.Convert a guest from vmware to openstack by virt-v2v via vddk
#  virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -n default esx6.7-rhel7.7-x86_64 --password-file /tmp/passwd -o openstack -oo server-id=rhel7.6-v2v-conversion-server
[   1.9] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel7.7-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[   7.2] Creating an overlay to protect the source from being modified
[  12.5] Opening the overlay
[ 170.2] Inspecting the overlay
[ 274.0] Checking for sufficient free disk space in the guest
[ 274.0] Estimating space required on target for each disk
[ 274.0] Converting Red Hat Enterprise Linux Server 7.7 Beta (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[2989.8] Mapping filesystem data to avoid copying unused and blank areas
[3012.2] Closing the overlay
[3014.5] Assigning disks to buses
[3014.5] Checking if the guest needs BIOS or UEFI to boot
[3014.5] Initializing the target -o openstack
Failed to set volume read-only access mode flag: Invalid volume: Volume cc3769b2-8cf0-4e35-8439-c6a1eca177de status must be available to update readonly flag, but current status is: creating. (HTTP 400) (Request-ID: req-3c06ca37-1dfb-48cf-ab77-bc4504ed2e25)
[3044.0] Copying disk 1/1 to /dev/disk/by-id/virtio-cc3769b2-8cf0-4e35-8 (raw)
    (100.00/100%)
[3311.7] Creating output metadata
[3321.3] Finishing off

Result:
   No need to set environment variable "export LIBGUESTFS_BACKEND=direct" during virt-v2v openstack conversion if selinux is enabled and use VMware VDDK as a transport, so the bug has been fixed and move the bug from ON_QA to VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2096