Bug 1695567

Summary: [downstream clone - 4.3.4] [RFE] Support VMs with VNC console on a FIPS enabled hypervisor
Product: Red Hat Enterprise Virtualization Manager Reporter: RHV bug bot <rhv-bugzilla-bot>
Component: vdsmAssignee: Tomasz BaraƄski <tbaransk>
Status: CLOSED ERRATA QA Contact: Liran Rotenberg <lrotenbe>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.2.3CC: bugzilla-qe-rhv, emarcus, gveitmic, jcall, lsurette, mavital, michal.skrivanek, mkalinin, mtessun, rbarry, srevivo, tbaransk, ycui
Target Milestone: ovirt-4.3.4Keywords: FutureFeature, ZStream
Target Release: 4.3.1Flags: lrotenbe: testing_plan_complete+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: vdsm-4.30.15 Doc Type: Enhancement
Doc Text:
When a host is running in FIPS mode, VNC must use SASL authorization instead of regular passwords because of the weak algorithm inherent in the VNC protocol. In order to facilitate that process, the Ansible role 'ovirt-host-setup-vnc-sasl' is provided. It must be run manually on all FIPS hosts. The role does the following: * Creates an (empty) SASL password database * Prepares an SASL configuration file for qemu * Changes the libvirt configuration file for qemu
Story Points: ---
Clone Of: 1595536 Environment:
Last Closed: 2019-06-20 14:48:41 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1595536    
Bug Blocks:    

Description RHV bug bot 2019-04-03 11:31:47 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1595536 +++
======================================================================

Description of problem:

A VM with a VNC console will fail to start on a host which is having fips enabled. It will fail with the error below.

2018-06-27 10:22:12,054+0530 ERROR (vm/e876d0c5) [virt.vm] (vmId='e876d0c5-6fa0-45e0-8a10-e44012a74f94') The vm start process failed (vm:943)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2872, in _run
    dom.createWithFlags(flags)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error: process exited while connecting to monitor: 2018-06-27T04:52:11.931930Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2018-06-27T04:52:11.973768Z qemu-kvm: -vnc 10.65.177.137:0,password: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or SASL authentication methods as an alternative

If the host is operating in "FIPS mode", the VM will be created with "-enable-fips" which will disable the VNC password authentication. So the VM will fail to start with the error above.

The VM with a spice console will work fine.


Version-Release number of selected component (if applicable):

vdsm-4.20.27.2-1.el7ev.x86_64


How reproducible:

100%

Steps to Reproduce:
1. Create a FIPS compliant host.

cat /proc/sys/crypto/fips_enabled
1

2. Start a VM with VNC graphics console on this host.
 
3. This will fail with the error as mentioned above.


Actual results:

Not possible to start a VM with VNC console on a FIPS compliant host.


Expected results:

It should be possible to start a VM with VNC console on a FIPS compliant host.

Additional info:

(Originally by Nijin Ashok)

Comment 1 RHV bug bot 2019-04-03 11:31:49 UTC
would require securing VNC first, and change authentication method form OTP to something else
Alternatively, we can ditch VNC for FIPS hosts

(Originally by michal.skrivanek)

Comment 2 RHV bug bot 2019-04-03 11:31:51 UTC
Re-targeting to 4.3.1 since it is missing a patch, an acked blocker flag, or both

(Originally by Ryan Barry)

Comment 5 RHV bug bot 2019-04-03 11:31:56 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops

(Originally by rhv-bugzilla-bot)

Comment 7 RHV bug bot 2019-04-03 11:31:59 UTC
Verification failed on:
ovirt-engine-4.3.3.1-0.1.el7.noarch
vdsm-4.30.12-1.el7ev.x86_64

Steps:
1. Enabled FIPS on the host
# yum -y install prelink dracut-fips
# prelink -u -a
# dracut -f
# df /boot
Take the Filesystem value (for example /dev/vda1 or /dev/sda1)
# blkid $filesystem
for example: # blkid /dev/sda1
Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14
Edit /etc/default/grub fips=1 and boot=$uuid:
GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....."
Regenerate grub, BIOS host:
# grub2-mkconfig -o /boot/grub2/grub.cfg
# reboot

2. Check FIPS enabled:
# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
# cat /proc/sys/crypto/fips_enabled 
1

3. Run the new ansible playbook:
Copy ssh-key:
# ssh-copy-id -i <key_path> <user>@<host>
Edit /etc/ansible/hosts
Add:
<host> ansible_ssh_private_key_file=<path>
Run:
# ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml
3. Edit a VM to VNC console.
4. Run the VM on the FIPS enabled host.

Results:
Run VM failed.
Engine log:
2019-04-02 15:30:58,045+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_DOWN_ERROR(119), VM golden_env_mixed_virtio_0 is down with error
. Exit message: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial 
NUMA mappings is obsoleted and will be removed in future
2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S
ASL authentication methods as an alternative.
2019-04-02 15:30:58,045+03 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] add VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'(golden_env_mixed_virtio_0) to rerun treatment
2019-04-02 15:30:58,051+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-11) [] Rerun VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'. Called from VDS 'host_mixed_2'
2019-04-02 15:30:58,060+03 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3309) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run 
VM golden_env_mixed_virtio_0 on Host host_mixed_2.

VDSM:
2019-04-02 15:30:57,721+0300 ERROR (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') The vm start process failed (vm:937)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 866, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2864, in _run
    dom.createWithFlags(flags)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NU
MA mappings is obsoleted and will be removed in future
2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S
ASL authentication methods as an alternative
2019-04-02 15:30:57,724+0300 INFO  (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Changed state to Down: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z q
emu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S
ASL authentication methods as an alternative (code=1) (vm:1675)
2019-04-02 15:30:57,727+0300 INFO  (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Stopping connection (guestagent:455)

Additional information:
I suspect that we miss vdsm patch on 4.3 branch: https://gerrit.ovirt.org/#/c/97381/

(Originally by Liran Rotenberg)

Comment 8 RHV bug bot 2019-04-03 11:32:01 UTC
(In reply to Liran Rotenberg from comment #7)
> I suspect that we miss vdsm patch on 4.3 branch:
> https://gerrit.ovirt.org/#/c/97381/

indeed. too late for 4.3.3 unfortunately

(Originally by michal.skrivanek)

Comment 10 Liran Rotenberg 2019-05-21 10:04:24 UTC
Verified on:
ovirt-engine-4.3.4-0.1.el7.noarch
vdsm-4.30.15-1.el7ev.x86_64

Steps:
1. Enabled FIPS on the host
# yum -y install prelink dracut-fips
# prelink -u -a
# dracut -f
# df /boot
Take the Filesystem value (for example /dev/vda1 or /dev/sda1)
# blkid $filesystem
for example: # blkid /dev/sda1
Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14
Edit /etc/default/grub fips=1 and boot=$uuid:
GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....."
Regenerate grub, BIOS host:
# grub2-mkconfig -o /boot/grub2/grub.cfg
# reboot

2. Check FIPS enabled:
# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
# cat /proc/sys/crypto/fips_enabled 
1

3. Set FIPS enbaled in the engine (accordingly, it possible not to add fips=1 to the kernel and redeploy+reboot the host after this step) 
Compute->Hosts->Edit host->Kernel->FIPS mode

4. Run the new ansible playbook:
Copy ssh-key:
# ssh-copy-id -i <key_path> <user>@<host>
Edit /etc/ansible/hosts
Add:
<host> ansible_ssh_private_key_file=<path>
Run:
# ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml
5. Edit a VM to VNC console.
6. Run the VM on the FIPS enabled host.

Results:
Run VM succeed.

Additional information:
The host must be set as VNC Encrypted.

Comment 12 errata-xmlrpc 2019-06-20 14:48:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1567

Comment 13 John Call 2019-10-01 03:53:37 UTC
It should be noted that the hypervisors must have FIPS enabled via the RHVM GUI.  Specifically, HOST -> Edit -> Kernel -> Reset -> FIPS mode.  It is not sufficient to deploy the RHV-H host and choose a Security Profile that includes FIPS --e.g. "VPP - Protection Profile for Virtualization v. 1.0 for Red Hat Enterprise Linux Hypervisor (RHELH)"