Bug 1695567 - [downstream clone - 4.3.4] [RFE] Support VMs with VNC console on a FIPS enabled hypervisor
Summary: [downstream clone - 4.3.4] [RFE] Support VMs with VNC console on a FIPS enabl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 4.2.3
Hardware: All
OS: Linux
medium
medium
Target Milestone: ovirt-4.3.4
: 4.3.1
Assignee: Tomasz Barański
QA Contact: Liran Rotenberg
URL:
Whiteboard:
Depends On: 1595536
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-03 11:31 UTC by RHV bug bot
Modified: 2021-09-09 15:34 UTC (History)
13 users (show)

Fixed In Version: vdsm-4.30.15
Doc Type: Enhancement
Doc Text:
When a host is running in FIPS mode, VNC must use SASL authorization instead of regular passwords because of the weak algorithm inherent in the VNC protocol. In order to facilitate that process, the Ansible role 'ovirt-host-setup-vnc-sasl' is provided. It must be run manually on all FIPS hosts. The role does the following: * Creates an (empty) SASL password database * Prepares an SASL configuration file for qemu * Changes the libvirt configuration file for qemu
Clone Of: 1595536
Environment:
Last Closed: 2019-06-20 14:48:41 UTC
oVirt Team: Virt
Target Upstream Version:
Embargoed:
lrotenbe: testing_plan_complete+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-43597 0 None None None 2021-09-09 15:34:36 UTC
Red Hat Product Errata RHBA-2019:1567 0 None None None 2019-06-20 14:48:49 UTC
oVirt gerrit 95985 0 'None' ABANDONED [WIP] virt: Remove VNC password when FIPS is enabled 2021-01-12 10:06:48 UTC
oVirt gerrit 97070 0 'None' MERGED core: Don't use password auth in FIPS mode 2021-01-12 10:06:48 UTC
oVirt gerrit 97318 0 'None' MERGED ansible: Setup SASL authentication for VNC 2021-01-12 10:07:26 UTC
oVirt gerrit 97381 0 'None' MERGED virt: In FIPS mode, use SASL auth instead of qemu passwords 2021-01-12 10:06:46 UTC
oVirt gerrit 98661 0 'None' MERGED ansible: Setup SASL authentication for VNC 2021-01-12 10:06:46 UTC
oVirt gerrit 99185 0 'None' ABANDONED core: Don't use password auth in FIPS mode 2021-01-12 10:06:49 UTC
oVirt gerrit 99186 0 'None' MERGED virt: In FIPS mode, use SASL auth instead of qemu passwords 2021-01-12 10:06:46 UTC
oVirt gerrit 99630 0 'None' MERGED virt: Files missing from Makefiles. 2021-01-12 10:07:27 UTC
oVirt gerrit 99631 0 'None' MERGED virt: Wrong attribute tested for VNC security 2021-01-12 10:06:49 UTC

Description RHV bug bot 2019-04-03 11:31:47 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1595536 +++
======================================================================

Description of problem:

A VM with a VNC console will fail to start on a host which is having fips enabled. It will fail with the error below.

2018-06-27 10:22:12,054+0530 ERROR (vm/e876d0c5) [virt.vm] (vmId='e876d0c5-6fa0-45e0-8a10-e44012a74f94') The vm start process failed (vm:943)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2872, in _run
    dom.createWithFlags(flags)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error: process exited while connecting to monitor: 2018-06-27T04:52:11.931930Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2018-06-27T04:52:11.973768Z qemu-kvm: -vnc 10.65.177.137:0,password: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or SASL authentication methods as an alternative

If the host is operating in "FIPS mode", the VM will be created with "-enable-fips" which will disable the VNC password authentication. So the VM will fail to start with the error above.

The VM with a spice console will work fine.


Version-Release number of selected component (if applicable):

vdsm-4.20.27.2-1.el7ev.x86_64


How reproducible:

100%

Steps to Reproduce:
1. Create a FIPS compliant host.

cat /proc/sys/crypto/fips_enabled
1

2. Start a VM with VNC graphics console on this host.
 
3. This will fail with the error as mentioned above.


Actual results:

Not possible to start a VM with VNC console on a FIPS compliant host.


Expected results:

It should be possible to start a VM with VNC console on a FIPS compliant host.

Additional info:

(Originally by Nijin Ashok)

Comment 1 RHV bug bot 2019-04-03 11:31:49 UTC
would require securing VNC first, and change authentication method form OTP to something else
Alternatively, we can ditch VNC for FIPS hosts

(Originally by michal.skrivanek)

Comment 2 RHV bug bot 2019-04-03 11:31:51 UTC
Re-targeting to 4.3.1 since it is missing a patch, an acked blocker flag, or both

(Originally by Ryan Barry)

Comment 5 RHV bug bot 2019-04-03 11:31:56 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops

(Originally by rhv-bugzilla-bot)

Comment 7 RHV bug bot 2019-04-03 11:31:59 UTC
Verification failed on:
ovirt-engine-4.3.3.1-0.1.el7.noarch
vdsm-4.30.12-1.el7ev.x86_64

Steps:
1. Enabled FIPS on the host
# yum -y install prelink dracut-fips
# prelink -u -a
# dracut -f
# df /boot
Take the Filesystem value (for example /dev/vda1 or /dev/sda1)
# blkid $filesystem
for example: # blkid /dev/sda1
Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14
Edit /etc/default/grub fips=1 and boot=$uuid:
GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....."
Regenerate grub, BIOS host:
# grub2-mkconfig -o /boot/grub2/grub.cfg
# reboot

2. Check FIPS enabled:
# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
# cat /proc/sys/crypto/fips_enabled 
1

3. Run the new ansible playbook:
Copy ssh-key:
# ssh-copy-id -i <key_path> <user>@<host>
Edit /etc/ansible/hosts
Add:
<host> ansible_ssh_private_key_file=<path>
Run:
# ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml
3. Edit a VM to VNC console.
4. Run the VM on the FIPS enabled host.

Results:
Run VM failed.
Engine log:
2019-04-02 15:30:58,045+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_DOWN_ERROR(119), VM golden_env_mixed_virtio_0 is down with error
. Exit message: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial 
NUMA mappings is obsoleted and will be removed in future
2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S
ASL authentication methods as an alternative.
2019-04-02 15:30:58,045+03 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] add VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'(golden_env_mixed_virtio_0) to rerun treatment
2019-04-02 15:30:58,051+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-11) [] Rerun VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'. Called from VDS 'host_mixed_2'
2019-04-02 15:30:58,060+03 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3309) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run 
VM golden_env_mixed_virtio_0 on Host host_mixed_2.

VDSM:
2019-04-02 15:30:57,721+0300 ERROR (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') The vm start process failed (vm:937)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 866, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2864, in _run
    dom.createWithFlags(flags)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NU
MA mappings is obsoleted and will be removed in future
2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S
ASL authentication methods as an alternative
2019-04-02 15:30:57,724+0300 INFO  (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Changed state to Down: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z q
emu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S
ASL authentication methods as an alternative (code=1) (vm:1675)
2019-04-02 15:30:57,727+0300 INFO  (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Stopping connection (guestagent:455)

Additional information:
I suspect that we miss vdsm patch on 4.3 branch: https://gerrit.ovirt.org/#/c/97381/

(Originally by Liran Rotenberg)

Comment 8 RHV bug bot 2019-04-03 11:32:01 UTC
(In reply to Liran Rotenberg from comment #7)
> I suspect that we miss vdsm patch on 4.3 branch:
> https://gerrit.ovirt.org/#/c/97381/

indeed. too late for 4.3.3 unfortunately

(Originally by michal.skrivanek)

Comment 10 Liran Rotenberg 2019-05-21 10:04:24 UTC
Verified on:
ovirt-engine-4.3.4-0.1.el7.noarch
vdsm-4.30.15-1.el7ev.x86_64

Steps:
1. Enabled FIPS on the host
# yum -y install prelink dracut-fips
# prelink -u -a
# dracut -f
# df /boot
Take the Filesystem value (for example /dev/vda1 or /dev/sda1)
# blkid $filesystem
for example: # blkid /dev/sda1
Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14
Edit /etc/default/grub fips=1 and boot=$uuid:
GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....."
Regenerate grub, BIOS host:
# grub2-mkconfig -o /boot/grub2/grub.cfg
# reboot

2. Check FIPS enabled:
# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
# cat /proc/sys/crypto/fips_enabled 
1

3. Set FIPS enbaled in the engine (accordingly, it possible not to add fips=1 to the kernel and redeploy+reboot the host after this step) 
Compute->Hosts->Edit host->Kernel->FIPS mode

4. Run the new ansible playbook:
Copy ssh-key:
# ssh-copy-id -i <key_path> <user>@<host>
Edit /etc/ansible/hosts
Add:
<host> ansible_ssh_private_key_file=<path>
Run:
# ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml
5. Edit a VM to VNC console.
6. Run the VM on the FIPS enabled host.

Results:
Run VM succeed.

Additional information:
The host must be set as VNC Encrypted.

Comment 12 errata-xmlrpc 2019-06-20 14:48:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1567

Comment 13 John Call 2019-10-01 03:53:37 UTC
It should be noted that the hypervisors must have FIPS enabled via the RHVM GUI.  Specifically, HOST -> Edit -> Kernel -> Reset -> FIPS mode.  It is not sufficient to deploy the RHV-H host and choose a Security Profile that includes FIPS --e.g. "VPP - Protection Profile for Virtualization v. 1.0 for Red Hat Enterprise Linux Hypervisor (RHELH)"


Note You need to log in before you can comment on or make changes to this bug.