Bug 1595536
Summary: | [RFE] Support VMs with VNC console on a FIPS enabled hypervisor | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Virtualization Manager | Reporter: | nijin ashok <nashok> | ||||
Component: | vdsm | Assignee: | Tomasz Barański <tbaransk> | ||||
Status: | CLOSED ERRATA | QA Contact: | Liran Rotenberg <lrotenbe> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | medium | ||||||
Version: | 4.2.3 | CC: | ahadas, bugzilla-qe-rhv, gveitmic, hhaberma, jcall, lsurette, mavital, michal.skrivanek, mkalinin, mtessun, rob, skavishw, srevivo, tbaransk, usurse, ycui | ||||
Target Milestone: | ovirt-4.4.1 | Keywords: | FutureFeature, ZStream | ||||
Target Release: | --- | Flags: | lrotenbe:
testing_plan_complete+
|
||||
Hardware: | All | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Enhancement | |||||
Doc Text: |
When a host is running in FIPS mode, VNC must use SASL authorization instead of regular passwords because of a weak algorithm inherent to the VNC protocol. The current release facilitates using SASL by providing an Ansible role, ovirt-host-setup-vnc-sasl, which you can run manually on FIPS-enabled hosts. This role does the following:
* Creates an empty SASL password database.
* Prepares the SASL config file for qemu.
* Changes the libvirt config file for qemu.
|
Story Points: | --- | ||||
Clone Of: | |||||||
: | 1695567 (view as bug list) | Environment: | |||||
Last Closed: | 2020-08-04 13:26:06 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | Virt | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1640357, 1695567 | ||||||
Attachments: |
|
Description
nijin ashok
2018-06-27 05:43:07 UTC
would require securing VNC first, and change authentication method form OTP to something else Alternatively, we can ditch VNC for FIPS hosts Re-targeting to 4.3.1 since it is missing a patch, an acked blocker flag, or both WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ] For more info please contact: rhv-devops Verification failed on: ovirt-engine-4.3.3.1-0.1.el7.noarch vdsm-4.30.12-1.el7ev.x86_64 Steps: 1. Enabled FIPS on the host # yum -y install prelink dracut-fips # prelink -u -a # dracut -f # df /boot Take the Filesystem value (for example /dev/vda1 or /dev/sda1) # blkid $filesystem for example: # blkid /dev/sda1 Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14 Edit /etc/default/grub fips=1 and boot=$uuid: GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....." Regenerate grub, BIOS host: # grub2-mkconfig -o /boot/grub2/grub.cfg # reboot 2. Check FIPS enabled: # sysctl crypto.fips_enabled crypto.fips_enabled = 1 # cat /proc/sys/crypto/fips_enabled 1 3. Run the new ansible playbook: Copy ssh-key: # ssh-copy-id -i <key_path> <user>@<host> Edit /etc/ansible/hosts Add: <host> ansible_ssh_private_key_file=<path> Run: # ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml 3. Edit a VM to VNC console. 4. Run the VM on the FIPS enabled host. Results: Run VM failed. Engine log: 2019-04-02 15:30:58,045+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_DOWN_ERROR(119), VM golden_env_mixed_virtio_0 is down with error . Exit message: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future 2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S ASL authentication methods as an alternative. 2019-04-02 15:30:58,045+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] add VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'(golden_env_mixed_virtio_0) to rerun treatment 2019-04-02 15:30:58,051+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-11) [] Rerun VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'. Called from VDS 'host_mixed_2' 2019-04-02 15:30:58,060+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3309) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM golden_env_mixed_virtio_0 on Host host_mixed_2. VDSM: 2019-04-02 15:30:57,721+0300 ERROR (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') The vm start process failed (vm:937) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 866, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2864, in _run dom.createWithFlags(flags) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NU MA mappings is obsoleted and will be removed in future 2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S ASL authentication methods as an alternative 2019-04-02 15:30:57,724+0300 INFO (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Changed state to Down: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z q emu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future 2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S ASL authentication methods as an alternative (code=1) (vm:1675) 2019-04-02 15:30:57,727+0300 INFO (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Stopping connection (guestagent:455) Additional information: I suspect that we miss vdsm patch on 4.3 branch: https://gerrit.ovirt.org/#/c/97381/ (In reply to Liran Rotenberg from comment #7) > I suspect that we miss vdsm patch on 4.3 branch: > https://gerrit.ovirt.org/#/c/97381/ indeed. too late for 4.3.3 unfortunately Oh, shoot! It will need both VDSM and Core patches backported. Verification failed on: ovirt-engine-4.4.0-0.0.master.20190501171039.git320c7fe.el7.noarch vdsm-4.40.0-238.gitaf61100.el7.x86_64 Steps: 1. Enabled FIPS on the host # yum -y install prelink dracut-fips # prelink -u -a # dracut -f # df /boot Take the Filesystem value (for example /dev/vda1 or /dev/sda1) # blkid $filesystem for example: # blkid /dev/sda1 Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14 Edit /etc/default/grub fips=1 and boot=$uuid: GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....." Regenerate grub, BIOS host: # grub2-mkconfig -o /boot/grub2/grub.cfg # reboot 2. Check FIPS enabled: # sysctl crypto.fips_enabled crypto.fips_enabled = 1 # cat /proc/sys/crypto/fips_enabled 1 3. Run the new ansible playbook: Copy ssh-key: # ssh-copy-id -i <key_path> <user>@<host> Edit /etc/ansible/hosts Add: <host> ansible_ssh_private_key_file=<path> Run: # ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml 3. Edit a VM to VNC console. 4. Run the VM on the FIPS enabled host. Results: Run VM failed. Engine log: 2019-05-16 13:40:31,667+03 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-281937) [a660560a-0514-4baa-aab7-a1d3899f713d] EVENT_ID: USER_STARTED_VM(153), VM golden_env_mixed_virtio_0 was started by admin@internal-authz (Host: host_mixed_3). 2019-05-16 13:40:34,233+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-1) [] VM '1b4d96d9-a556-4d4c-84c8-a68221255ece' was reported as Down on VDS 'be687e98-06de-4bf7-b88b-87254b49af87'(host_mixed_3) 2019-05-16 13:40:34,234+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-1) [] START, DestroyVDSCommand(HostName = host_mixed_3, DestroyVmVDSCommandParameters:{hostId='be687e98-06de-4bf7-b88b-87254b49af87', vmId='1b4d96d9-a556-4d4c-84c8-a68221255ece', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 6c6a1f67 2019-05-16 13:40:34,532+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-1) [] FINISH, DestroyVDSCommand, return: , log id: 6c6a1f67 2019-05-16 13:40:34,532+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-1) [] VM '1b4d96d9-a556-4d4c-84c8-a68221255ece'(golden_env_mixed_virtio_0) moved from 'WaitForLaunch' --> 'Down' 2019-05-16 13:40:34,563+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-1) [] EVENT_ID: VM_DOWN_ERROR(119), VM golden_env_mixed_virtio_0 is down with error. Exit message: internal error: qemu unexpectedly closed the monitor: 2019-05-16T10:40:33.537988Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future 2019-05-16T10:40:33.577415Z qemu-kvm: -vnc 10.35.30.3:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or SASL authentication methods as an alternative. 2019-05-16 13:40:34,564+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-1) [] add VM '1b4d96d9-a556-4d4c-84c8-a68221255ece'(golden_env_mixed_virtio_0) to rerun treatment 2019-05-16 13:40:34,569+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-1) [] Rerun VM '1b4d96d9-a556-4d4c-84c8-a68221255ece'. Called from VDS 'host_mixed_3' 2019-05-16 13:40:34,576+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-281940) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM golden_env_mixed_virtio_0 on Host host_mixed_3. VDSM log: 2019-05-16 13:40:34,228+0300 ERROR (vm/1b4d96d9) [virt.vm] (vmId='1b4d96d9-a556-4d4c-84c8-a68221255ece') The vm start process failed (vm:933) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 867, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2880, in _run dom.createWithFlags(flags) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-05-16T10:40:33.537988Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NU MA mappings is obsoleted and will be removed in future 2019-05-16T10:40:33.577415Z qemu-kvm: -vnc 10.35.30.3:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S ASL authentication methods as an alternative 2019-05-16 13:40:34,228+0300 INFO (vm/1b4d96d9) [virt.vm] (vmId='1b4d96d9-a556-4d4c-84c8-a68221255ece') Changed state to Down: internal error: qemu unexpectedly closed the monitor: 2019-05-16T10:40:33.537988Z q emu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future 2019-05-16T10:40:33.577415Z qemu-kvm: -vnc 10.35.30.3:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S ASL authentication methods as an alternative (code=1) (vm:1690) Additional info: I saw vnc_sasl=1 added to the host after running the ansible, in /etc/libvirt/qemu.conf. Logs? Created attachment 1569506 [details]
logs
The vdsm does not send FIPS information back to the engine. The engine uses Host's kernel parameters to tell whether FIPS mode is on or off. In order to correctly construct XML for libvirt, the "FIPS enabled" option on the Host option panes must be checked. Verified on: ovirt-engine-4.4.0-0.0.master.20190519192123.gitd51360f.el7.noarch vdsm-4.30.15-1.el7.x86_64 Steps: 1. Enabled FIPS on the host # yum -y install prelink dracut-fips # prelink -u -a # dracut -f # df /boot Take the Filesystem value (for example /dev/vda1 or /dev/sda1) # blkid $filesystem for example: # blkid /dev/sda1 Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14 Edit /etc/default/grub fips=1 and boot=$uuid: GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....." Regenerate grub, BIOS host: # grub2-mkconfig -o /boot/grub2/grub.cfg # reboot 2. Check FIPS enabled: # sysctl crypto.fips_enabled crypto.fips_enabled = 1 # cat /proc/sys/crypto/fips_enabled 1 3. Set FIPS enbaled in the engine (accordingly, it possible not to add fips=1 to the kernel and redeploy+reboot the host after this step) Compute->Hosts->Edit host->Kernel->FIPS mode 4. Run the new ansible playbook: Copy ssh-key: # ssh-copy-id -i <key_path> <user>@<host> Edit /etc/ansible/hosts Add: <host> ansible_ssh_private_key_file=<path> Run: # ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml 5. Edit a VM to VNC console. 6. Run the VM on the FIPS enabled host. Results: Run VM succeed. Additional information: The host must be set as VNC Encrypted. (In reply to Liran Rotenberg from comment #20) I don't understand this, could you please clarify? I discovered that a host having 'fips=1' configured outside of RHVM won't start a VNC-based VM. The host **must** use RHVM GUI (Host Edit screens) to set 'fips=1' > 3. Set FIPS enbaled in the engine (accordingly, it possible not to add > fips=1 to the kernel and redeploy+reboot the host after this step) > Compute->Hosts->Edit host->Kernel->FIPS mode Can you describe how this is done? I saw a checkbox for enabling VNC encryption in the Edit Cluster screens, but my host's capabilities still report vncEncryped=False > Additional information: > The host must be set as VNC Encrypted. (In reply to John Call from comment #21) > (In reply to Liran Rotenberg from comment #20) > I don't understand this, could you please clarify? I discovered that a host > having 'fips=1' configured outside of RHVM won't start a VNC-based VM. The > host **must** use RHVM GUI (Host Edit screens) to set 'fips=1' > > 3. Set FIPS enbaled in the engine (accordingly, it possible not to add > > fips=1 to the kernel and redeploy+reboot the host after this step) > > Compute->Hosts->Edit host->Kernel->FIPS mode > The first thing you need to have is VNC encryption. The related bug and steps are in here: https://bugzilla.redhat.com/show_bug.cgi?id=1597085#c20 In small details, vnc_tls=1 need to be set in qemu.conf (but not only, therefore the safest way is to re-install the host). This is done when the cluster is set to VNC encryption on. (Compute->Cluster->Console->Enable VNC Encryption) When it is set, you need to redeploy the host/s in the cluster, the re-installation of the host will set the parameters. On top of that, you can have the host in fips mode, using the above ansible role and have the VMs with VNC on FIPS enabled hypervisor. > Can you describe how this is done? I saw a checkbox for enabling VNC > encryption in the Edit Cluster screens, but my host's capabilities still > report vncEncryped=False > > Additional information: > > The host must be set as VNC Encrypted. WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{}', ] For more info please contact: rhv-devops While FIPS console works now, I still cannot use it via novnc and websocket proxy service. It works fine without FIPS. failed: Error during WebSocket handshake: Unexpected response code: 503 checkConnection @ html-console-common.js:26 (In reply to Rob Sanders from comment #35) > While FIPS console works now, I still cannot use it via novnc and websocket > proxy service. It works fine without FIPS. > > failed: Error during WebSocket handshake: Unexpected response code: 503 > checkConnection @ html-console-common.js:26 it's verified working so if it doesn't work for you please open a new bug and/or ask first on mailing list with details/logs/etc Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RHV RHEL Host (ovirt-host) 4.4), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:3246 |