Bug 2026770

Summary:	host deployment fails on fips-enabled host
Product:	[oVirt] ovirt-ansible-collection	Reporter:	Michal Skrivanek <michal.skrivanek>
Component:	hosted-engine-setup	Assignee:	Asaf Rachmani <arachman>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Nikolai Sednev <nsednev>
Severity:	high	Docs Contact:
Priority:	medium
Version:	unspecified	CC:	arachman, bugs, delfassy, mnecas, sbonazzo
Target Milestone:	ovirt-4.5.0	Keywords:	Triaged, ZStream
Target Release:	2.0.0	Flags:	nsednev: needinfo- pm-rhel: ovirt-4.5?
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ovirt-hosted-engine-setup-2.6.1 ovirt-ansible-collection-2.0.0_beta6	Doc Type:	Bug Fix
Doc Text:	Cause: SPICE doesn't support in CL 4.7/el9stream Consequence: Hosted-Engine deployment fails when fips is enabled since the default graphic protocol on FIPS host is SPICE Fix: Remove SPICE and use VNC graphic protocol Result: Hosted-Engine deployment succeeds	Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-04-20 06:33:59 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2069658
Bug Blocks:

Description Michal Skrivanek 2021-11-25 19:00:58 UTC

we're dropping SPICE console support in CL 4.7/el9stream

HE VM is using SPICE+VNC, in case of FIPS setup it's only SPICE (https://github.com/oVirt/ovirt-ansible-collection/blob/master/roles/hosted_engine_setup/tasks/create_target_vm/01_create_target_hosted_engine_vm.yml#L141)

This needs to be changed to VNC. There should be no reason for SPICE in either case, it's there probably from the times when we didn't have FIPS-enabled VNC.

Comment 2 Nikolai Sednev 2022-03-28 14:58:35 UTC

Do we have an exact instructions on how to enable the FIPS on already provisioned host, so I will be able to verify the FIPS enabled parameter during HE deployment on FIPS enabled RHEL8.6 host?

Comment 3 Nikolai Sednev 2022-03-28 14:59:42 UTC

My currently provisioned hosts has not been set for FIPS:
serval14 ~]# sysctl crypto.fips_enabled
crypto.fips_enabled = 0
cat /proc/sys/crypto/fips_enabled
0

Comment 4 Nikolai Sednev 2022-03-28 15:25:38 UTC

I followed the https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening#switching-the-system-to-fips-mode_using-the-system-wide-cryptographic-policies:

Enabled the FIPS on host:
serval15 ~]# fips-mode-setup --enable && reboot
serval15 ~]# fips-mode-setup --check
FIPS mode is enabled.
serval15 ~]# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
serval15 ~]# cat /proc/sys/crypto/fips_enabled
1
serval15 ~]# update-crypto-policies --show 
FIPS

Looks like that's it for enabling the FIPS on the host.
I tried to deploy HE using latest ovirt-hosted-engine-setup-2.6.2-1.el8ev.noarch and ovirt-ansible-collection-2.0.0-0.6.BETA.el8ev.noarch and failed with:

serval14 ~]# hosted-engine --deploy --4 --ansible-extra-vars=he_pause_before_engine_setup=true
***L:ERROR Internal error: No module named 'ansible.constants'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 141, in execute
    self.context.loadPlugins()
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 803, in loadPlugins
    self._loadPluginGroups(plugindir, needgroups, loadedgroups)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 112, in _loadPluginGroups
    self._loadPlugins(path, path, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 69, in _loadPlugins
    self._loadPlugins(base, d, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 100, in _loadPlugins
    os.path.basename(path),
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 110, in loadModule
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module>
    from . import misc
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 32, in <module>
    from ovirt_hosted_engine_setup import ansible_utils
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 35, in <module>
    from ansible.constants import AnsibleCallback
ModuleNotFoundError: No module named 'ansible.constants'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/__main__.py", line 88, in main
    installer.execute()
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 147, in execute
    sys.exc_info()[2],
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 85, in raiseExceptionInformation
    raise info[1].with_traceback(info[2])
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 141, in execute
    self.context.loadPlugins()
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 803, in loadPlugins
    self._loadPluginGroups(plugindir, needgroups, loadedgroups)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 112, in _loadPluginGroups
    self._loadPlugins(path, path, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 69, in _loadPlugins
    self._loadPlugins(base, d, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 100, in _loadPlugins
    os.path.basename(path),
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 110, in loadModule
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module>
    from . import misc
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 32, in <module>
    from ovirt_hosted_engine_setup import ansible_utils
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 35, in <module>
    from ansible.constants import AnsibleCallback
otopi.main.PluginLoadException: No module named 'ansible.constants'

Comment 5 Nikolai Sednev 2022-03-28 23:49:32 UTC

Just the same happens on regular host with disabled FIPS functionality:
serval16 ~]# update-crypto-policies --show 
DEFAULT
[root@serval16 ~]# hosted-engine --deploy --4 --ansible-extra-vars=he_pause_before_engine_setup=true
***L:ERROR Internal error: No module named 'ansible.constants'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 141, in execute
    self.context.loadPlugins()
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 803, in loadPlugins
    self._loadPluginGroups(plugindir, needgroups, loadedgroups)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 112, in _loadPluginGroups
    self._loadPlugins(path, path, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 69, in _loadPlugins
    self._loadPlugins(base, d, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 100, in _loadPlugins
    os.path.basename(path),
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 110, in loadModule
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module>
    from . import misc
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 32, in <module>
    from ovirt_hosted_engine_setup import ansible_utils
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 35, in <module>
    from ansible.constants import AnsibleCallback
ModuleNotFoundError: No module named 'ansible.constants'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/__main__.py", line 88, in main
    installer.execute()
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 147, in execute
    sys.exc_info()[2],
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 85, in raiseExceptionInformation
    raise info[1].with_traceback(info[2])
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 141, in execute
    self.context.loadPlugins()
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 803, in loadPlugins
    self._loadPluginGroups(plugindir, needgroups, loadedgroups)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 112, in _loadPluginGroups
    self._loadPlugins(path, path, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 69, in _loadPlugins
    self._loadPlugins(base, d, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 100, in _loadPlugins
    os.path.basename(path),
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 110, in loadModule
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module>
    from . import misc
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 32, in <module>
    from ovirt_hosted_engine_setup import ansible_utils
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 35, in <module>
    from ansible.constants import AnsibleCallback
otopi.main.PluginLoadException: No module named 'ansible.constants'

Comment 6 Nikolai Sednev 2022-03-28 23:56:11 UTC

serval16 ~]# ansible --version
ansible [core 2.12.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 2.10.3
  libyaml = True

Comment 7 Asaf Rachmani 2022-03-29 09:46:40 UTC

This issue was introduced in ovirt-hosted-engine-setup-2.6.2-1, the fix for it has been recently merged and is planned to be included in the upcoming release, please avoid using this version.

Comment 8 Nikolai Sednev 2022-03-29 11:55:53 UTC

ovirt-hosted-engine-setup-2.6.3-1.el8ev.noarch deployment also fails:


serval14 ~]# hosted-engine --deploy --4 --ansible-extra-vars=he_pause_before_engine_setup=true[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
          Continuing will configure this host for serving as hypervisor and will create a local VM with a running engine.
          The locally running engine will be used to configure a new storage domain and create a VM there.
          At the end the disk of the local VM will be moved to the shared storage.
          Are you sure you want to continue? (Yes, No)[Yes]: 
          It has been detected that this program is executed through an SSH connection without using tmux.
          Continuing with the installation may lead to broken installation if the network connection fails.
          It is highly recommended to abort the installation and run it inside a tmux session using command "tmux".
          Do you want to continue anyway? (Yes, No)[No]: yes
          Configuration files: 
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20220329144634-3rvy5v.log
          Version: otopi-1.10.0 (otopi-1.10.0-1.el8ev)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup (late)
[ INFO  ] Stage: Environment customization
         
          --== STORAGE CONFIGURATION ==--
         
         
          --== HOST NETWORK CONFIGURATION ==--
         
          Please indicate the gateway IP address [10.35.235.254]: 
[ INFO  ] Checking available network interfaces:
[ ERROR ] b"ERROR! couldn't resolve module/action 'firewalld'. This often indicates a misspelling, missing collection, or incorrect module path.\n"
[ ERROR ] b'\n'
[ ERROR ] b"The error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/bootstrap_local_vm/05_add_host.yml': line 12, column 5, but may\n"
[ ERROR ] b'be elsewhere in the file depending on the exact syntax problem.\n'
[ ERROR ] b'\n'
[ ERROR ] b'The offending line appears to be:\n'
[ ERROR ] b'\n'
[ ERROR ] b'    delay: 20\n'
[ ERROR ] b'  - name: Open a port on firewalld\n'
[ ERROR ] b'    ^ here\n'
[ ERROR ] Failed to execute stage 'Environment customization': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ ERROR ] b"ERROR! couldn't resolve module/action 'firewalld'. This often indicates a misspelling, missing collection, or incorrect module path.\n"
[ ERROR ] b'\n'
[ ERROR ] b"The error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/bootstrap_local_vm/05_add_host.yml': line 12, column 5, but may\n"
[ ERROR ] b'be elsewhere in the file depending on the exact syntax problem.\n'
[ ERROR ] b'\n'
[ ERROR ] b'The offending line appears to be:\n'
[ ERROR ] b'\n'
[ ERROR ] b'    delay: 20\n'
[ ERROR ] b'  - name: Open a port on firewalld\n'
[ ERROR ] b'    ^ here\n'
[ ERROR ] Failed to execute stage 'Clean up': Failed executing ansible-playbook
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20220329145306.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20220329144634-3rvy5v.log


ansible-macros-2021.1.2-2.1.el8ev.noarch
ovirt-ansible-collection-2.0.0-0.6.BETA.el8ev.noarch
ansible-core-2.12.2-3.1.el8.x86_64
ovirt-hosted-engine-setup-2.6.3-1.el8ev.noarch
ovirt-hosted-engine-ha-2.5.0-1.el8ev.noarch

Comment 9 Asaf Rachmani 2022-03-29 12:05:30 UTC

Martin, how do we provide the non-community Ansible's modules?
As we discussed for HE, ansible.posix and ansible.netcommon are required.

Comment 10 Martin Necas 2022-03-29 12:16:43 UTC

For the upstream I have done builds in cbs for ansible-posix [1], ansible-utils [2], ansible-netcommon [3] collections. 
As for the downstream we have now a few issues and waiting till the tickets get resolved.


It is possible that they were not installed because the requirement for them was added in ovirt-ansible-collection-2.0.0-0.8.BETA and you have 0.6.BETA.

Why do you have installed ansible-macros?


[1] https://cbs.centos.org/koji/packageinfo?packageID=8469
[2] https://cbs.centos.org/koji/packageinfo?packageID=8492
[3] https://cbs.centos.org/koji/packageinfo?packageID=8471
[4] https://github.com/oVirt/ovirt-ansible-collection/pull/457

Comment 11 Nikolai Sednev 2022-04-04 17:57:39 UTC

serval15 ~]#
fips-mode-setup --enable && reboot
fips-mode-setup --check && sysctl crypto.fips_enabled && update-crypto-policies --show && cat /proc/sys/crypto/fips_enabled
/sys/crypto/fips_enabled
FIPS mode is enabled.
crypto.fips_enabled = 1
FIPS
1

Works fine on:
ansible-core-2.12.2-3.1.el8.x86_64
ovirt-hosted-engine-setup-2.6.3-1.el8ev.noarch
ovirt-hosted-engine-ha-2.5.0-1.el8ev.noarch
ovirt-ansible-collection-2.0.0-0.9.BETA.el8ev.noarch
Red Hat Enterprise Linux release 8.6 Beta (Ootpa)
Linux 4.18.0-372.5.1.el8.x86_64 #1 SMP Mon Mar 28 11:03:39 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Engine Software Version:4.5.0.1-605.90f87fe14688.14.el8ev

Successfully deployed HE over NFS, on FIPS enabled host.

Comment 12 Sandro Bonazzola 2022-04-20 06:33:59 UTC

This bugzilla is included in oVirt 4.5.0 release, published on April 20th 2022.

Since the problem described in this bug report should be resolved in oVirt 4.5.0 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.