Hide Forgot
+++ This bug is a downstream clone. The original bug is: +++ +++ bug 1595536 +++ ====================================================================== Description of problem: A VM with a VNC console will fail to start on a host which is having fips enabled. It will fail with the error below. 2018-06-27 10:22:12,054+0530 ERROR (vm/e876d0c5) [virt.vm] (vmId='e876d0c5-6fa0-45e0-8a10-e44012a74f94') The vm start process failed (vm:943) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 872, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2872, in _run dom.createWithFlags(flags) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirtError: internal error: process exited while connecting to monitor: 2018-06-27T04:52:11.931930Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future 2018-06-27T04:52:11.973768Z qemu-kvm: -vnc 10.65.177.137:0,password: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or SASL authentication methods as an alternative If the host is operating in "FIPS mode", the VM will be created with "-enable-fips" which will disable the VNC password authentication. So the VM will fail to start with the error above. The VM with a spice console will work fine. Version-Release number of selected component (if applicable): vdsm-4.20.27.2-1.el7ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. Create a FIPS compliant host. cat /proc/sys/crypto/fips_enabled 1 2. Start a VM with VNC graphics console on this host. 3. This will fail with the error as mentioned above. Actual results: Not possible to start a VM with VNC console on a FIPS compliant host. Expected results: It should be possible to start a VM with VNC console on a FIPS compliant host. Additional info: (Originally by Nijin Ashok)
would require securing VNC first, and change authentication method form OTP to something else Alternatively, we can ditch VNC for FIPS hosts (Originally by michal.skrivanek)
Re-targeting to 4.3.1 since it is missing a patch, an acked blocker flag, or both (Originally by Ryan Barry)
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ] For more info please contact: rhv-devops (Originally by rhv-bugzilla-bot)
Verification failed on: ovirt-engine-4.3.3.1-0.1.el7.noarch vdsm-4.30.12-1.el7ev.x86_64 Steps: 1. Enabled FIPS on the host # yum -y install prelink dracut-fips # prelink -u -a # dracut -f # df /boot Take the Filesystem value (for example /dev/vda1 or /dev/sda1) # blkid $filesystem for example: # blkid /dev/sda1 Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14 Edit /etc/default/grub fips=1 and boot=$uuid: GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....." Regenerate grub, BIOS host: # grub2-mkconfig -o /boot/grub2/grub.cfg # reboot 2. Check FIPS enabled: # sysctl crypto.fips_enabled crypto.fips_enabled = 1 # cat /proc/sys/crypto/fips_enabled 1 3. Run the new ansible playbook: Copy ssh-key: # ssh-copy-id -i <key_path> <user>@<host> Edit /etc/ansible/hosts Add: <host> ansible_ssh_private_key_file=<path> Run: # ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml 3. Edit a VM to VNC console. 4. Run the VM on the FIPS enabled host. Results: Run VM failed. Engine log: 2019-04-02 15:30:58,045+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_DOWN_ERROR(119), VM golden_env_mixed_virtio_0 is down with error . Exit message: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future 2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S ASL authentication methods as an alternative. 2019-04-02 15:30:58,045+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] add VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'(golden_env_mixed_virtio_0) to rerun treatment 2019-04-02 15:30:58,051+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-11) [] Rerun VM 'd77718bc-fe6d-472c-86ba-b88c5978d9a8'. Called from VDS 'host_mixed_2' 2019-04-02 15:30:58,060+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3309) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM golden_env_mixed_virtio_0 on Host host_mixed_2. VDSM: 2019-04-02 15:30:57,721+0300 ERROR (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') The vm start process failed (vm:937) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 866, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2864, in _run dom.createWithFlags(flags) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NU MA mappings is obsoleted and will be removed in future 2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S ASL authentication methods as an alternative 2019-04-02 15:30:57,724+0300 INFO (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Changed state to Down: internal error: qemu unexpectedly closed the monitor: 2019-04-02T12:30:56.851807Z q emu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future 2019-04-02T12:30:56.878941Z qemu-kvm: -vnc 10.35.30.6:0,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or S ASL authentication methods as an alternative (code=1) (vm:1675) 2019-04-02 15:30:57,727+0300 INFO (vm/d77718bc) [virt.vm] (vmId='d77718bc-fe6d-472c-86ba-b88c5978d9a8') Stopping connection (guestagent:455) Additional information: I suspect that we miss vdsm patch on 4.3 branch: https://gerrit.ovirt.org/#/c/97381/ (Originally by Liran Rotenberg)
(In reply to Liran Rotenberg from comment #7) > I suspect that we miss vdsm patch on 4.3 branch: > https://gerrit.ovirt.org/#/c/97381/ indeed. too late for 4.3.3 unfortunately (Originally by michal.skrivanek)
Verified on: ovirt-engine-4.3.4-0.1.el7.noarch vdsm-4.30.15-1.el7ev.x86_64 Steps: 1. Enabled FIPS on the host # yum -y install prelink dracut-fips # prelink -u -a # dracut -f # df /boot Take the Filesystem value (for example /dev/vda1 or /dev/sda1) # blkid $filesystem for example: # blkid /dev/sda1 Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14 Edit /etc/default/grub fips=1 and boot=$uuid: GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....." Regenerate grub, BIOS host: # grub2-mkconfig -o /boot/grub2/grub.cfg # reboot 2. Check FIPS enabled: # sysctl crypto.fips_enabled crypto.fips_enabled = 1 # cat /proc/sys/crypto/fips_enabled 1 3. Set FIPS enbaled in the engine (accordingly, it possible not to add fips=1 to the kernel and redeploy+reboot the host after this step) Compute->Hosts->Edit host->Kernel->FIPS mode 4. Run the new ansible playbook: Copy ssh-key: # ssh-copy-id -i <key_path> <user>@<host> Edit /etc/ansible/hosts Add: <host> ansible_ssh_private_key_file=<path> Run: # ansible-playbook -l <host> /usr/share/ovirt-engine/playbooks/ovirt-vnc-sasl.yml 5. Edit a VM to VNC console. 6. Run the VM on the FIPS enabled host. Results: Run VM succeed. Additional information: The host must be set as VNC Encrypted.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1567
It should be noted that the hypervisors must have FIPS enabled via the RHVM GUI. Specifically, HOST -> Edit -> Kernel -> Reset -> FIPS mode. It is not sufficient to deploy the RHV-H host and choose a Security Profile that includes FIPS --e.g. "VPP - Protection Profile for Virtualization v. 1.0 for Red Hat Enterprise Linux Hypervisor (RHELH)"