Bug 1712481

it should be a cluster level policy and "misbehaving" hosts should be non-operational

For the Hosted Engine VM, when VNC is set, starting on a non-FIPS host, migrating fails as above.
Trying to set global maintenance mode, shutting the VM down and starting it on the FIPS enabled host fails.

2019-05-27 09:48:15,143+0300 ERROR (vm/71e1cf2d) [virt.vm] (vmId='71e1cf2d-9f24-471f-871d-0c2b3fbcb70e') The vm start process failed (vm:933)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 867, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2880, in _run
    dom.createWithFlags(flags)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-05-27T06:48:14.341127Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2019-05-27T06:48:14.373663Z qemu-kvm: -vnc 10.35.30.6:2,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or SASL authentication methods as an alternative
2019-05-27 09:48:15,144+0300 INFO  (vm/71e1cf2d) [virt.vm] (vmId='71e1cf2d-9f24-471f-871d-0c2b3fbcb70e') Changed state to Down: internal error: qemu unexpectedly closed the monitor: 2019-05-27T06:48:14.341127Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2019-05-27T06:48:14.373663Z qemu-kvm: -vnc 10.35.30.6:2,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or SASL authentication methods as an alternative (code=1) (vm:1690)
2019-05-27 09:48:15,147+0300 INFO  (vm/71e1cf2d) [virt.vm] (vmId='71e1cf2d-9f24-471f-871d-0c2b3fbcb70e') Stopping connection (guestagent:455)

Created attachment 1573769 [details]
SHE trying to start on FIPS after being on non-FIPS

Still reproducible?

yes, still reproducible,

1. Install host with RHEL 7.7 and FIPS enabled.
2. Add the host to RHV 4.3.8.0-0.1.el7 (another host is connected to the same env. with FIPS disabled)
3. Then try migration as following:

---- migration non-FIPS --> FIPS (log from source host):

2019-12-17 13:44:32,850+0200 ERROR (migsrc/9aec7b74) [virt.vm] (vmId='9aec7b74-1c75-475b-bb9a-a97aabfb8105') Failed to migrate (migration:450)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run
    time.time(), migrationParams, machineParams
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 507, in _startUnderlyingMigration
    self._perform_with_downtime_thread(duri, muri)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 584, in _perform_with_downtime_thread
    self._perform_migration(duri, muri)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration
    self._migration_flags)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1779, in migrateToURI3
    if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-12-17T11:44:31.606368Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2019-12-17T11:44:31.631296Z qemu-kvm: -vnc 10.35.30.5:5,password,tls,x509=/etc/pki/vdsm/libvirt-vnc,sasl: Failed to start VNC server: VNC password auth disabled due to FIPS mode, consider using the VeNCrypt or SASL authentication methods as an alternative
2019-12-17 13:44:32,896+0200 INFO  (jsonrpc/6) [api.virt] START getMigrationStatus() from=::ffff:10.35.30.101,41166, vmId=9aec7b74-1c75-475b-bb9a-a97aabfb8105 (api:48)
2019-12-17 13:44:32,896+0200 INFO  (jsonrpc/6) [api.virt] FINISH getMigrationStatus return={'status': {'message': 'Done', 'code': 0}, 'migrationStats': {'status': {'message': 'Fatal error during migration', 'code': 12}, 'progress': 0}} from=::ffff:10.35.30.101,41166, vmId=9aec7b74-1c75-475b-bb9a-a97aabfb8105 (api:54)

---- migration FIPS --> non-FIPS (log from destination host):

2019-12-17 13:47:38,277+0200 ERROR (jsonrpc/0) [api] FINISH create error=Error creating the requested VM (api:131)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 234, in create
    "VNC is allowed to start VNC with no password auth and "
CannotCreateVM: Error creating the requested VM
2019-12-17 13:47:38,278+0200 INFO  (jsonrpc/0) [api.virt] FINISH create return={'status': {'message': 'Error creating the requested VM', 'code': 9}} from=::ffff:10.35.30.5,60548, vmId=ad67a8c2-7fc4-4470-a036-724cb85f42f8 (api:54)

Fix to bug #1712325 might have fixed it.

(In reply to Tomasz Barański from comment #6)
> Fix to bug #1712325 might have fixed it.

Nope, still reproducible,

Verified with: 
- ovirt-engine-4.4.0-0.33.master.el8ev.noarch
- vdsm-4.40.13-1.el8ev.x86_64
- HOST with RHEL8.2

Verified with:
ovirt-engine-4.4.5.6-0.11.el8ev.noarch

Steps:
1. Create cluster with undefined, enabled or disabled fips mode using REST API.
<cluster>
  <name>cluster1</name>
  <fips_mode>undefined</fips_mode>
  <data_center id="1"/>
</cluster>
<cluster>
  <name>cluster2</name>
  <fips_mode>enabled</fips_mode>
  <data_center id="1"/>
</cluster>
<cluster>
  <name>cluster3</name>
  <fips_mode>disabled</fips_mode>
  <data_center id="1"/>
</cluster>
2. Get clusters info using REST API, check fips mode in response content.
3. Update cluster fips mode from undefined to enabled using REST API.
4. Create a new cluster4 with fips mode "Auto Detect" on UI.
5. Add a fips enabled host1 to cluster4, check cluster4 fips mode, and host1 status.
6. Add a fips disabled host2 to cluster4, check cluster4 fips mode, and host2 status.
7. Update cluster4 fips mode to Disabled on UI.
8. Move all hosts in cluster4 into maintenance, then update cluster4 fips mode to Disabled on UI.
9. Activate all hosts in cluster4, check cluster4 fips mode, and hosts status.

Results:
1. Could create cluster with undefined, enabled or disabled fips mode using REST API.
2. Fips mode in get response is correct.
3. Could update cluster fips mode from undefined to enabled using REST API.
4. Could create cluster with "Auto Detect" fips mode on UI.
5. cluster4 fips moded changes from "Auto Detect" to "Enabled" after the fips enabled host1 added to it.
6. When add the fips disabled host2 to cluster4, host2 is nonoperational, cluster4's fips mode is still "Enabled".
7. It fails to update cluster4 fips mode when hosts are not in maintenance mode, saying "Cannot update Cluster FIPS mode unless all Hosts attached to this Cluster are in Maintenance."
8. Could update cluster4 fips mode to "Disabled" when all hosts attached to it are moved to maintenance.
9. When activate all hosts in cluster4 after fips mode updated to "Disabled", the fips mode disabled host2 is up, while the fips mode enabled host1 is nonoperational.

All tests work as expected, so move this bug to verified.

This bugzilla is included in oVirt 4.4.5 release, published on March 18th 2021.

Since the problem described in this bug report should be resolved in oVirt 4.4.5 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.