Bug 2026770 - host deployment fails on fips-enabled host
Summary: host deployment fails on fips-enabled host
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-ansible-collection
Classification: oVirt
Component: hosted-engine-setup
Version: unspecified
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ovirt-4.5.0
: 2.0.0
Assignee: Asaf Rachmani
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On: 2069658
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-25 19:00 UTC by Michal Skrivanek
Modified: 2022-04-20 06:33 UTC (History)
5 users (show)

Fixed In Version: ovirt-hosted-engine-setup-2.6.1 ovirt-ansible-collection-2.0.0_beta6
Doc Type: Bug Fix
Doc Text:
Cause: SPICE doesn't support in CL 4.7/el9stream Consequence: Hosted-Engine deployment fails when fips is enabled since the default graphic protocol on FIPS host is SPICE Fix: Remove SPICE and use VNC graphic protocol Result: Hosted-Engine deployment succeeds
Clone Of:
Environment:
Last Closed: 2022-04-20 06:33:59 UTC
oVirt Team: Integration
Embargoed:
nsednev: needinfo-
pm-rhel: ovirt-4.5?


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github oVirt ovirt-ansible-collection pull 394 0 None Merged roles: hosted_engine_setup: Remove SPICE from graphic protocols 2021-12-16 09:24:49 UTC
Github oVirt ovirt-hosted-engine-setup pull 11 0 None open SPICE: Remove SPICE from graphic protocols 2022-01-04 13:01:34 UTC
Red Hat Issue Tracker RHV-44091 0 None None None 2021-11-25 19:03:01 UTC

Description Michal Skrivanek 2021-11-25 19:00:58 UTC
we're dropping SPICE console support in CL 4.7/el9stream

HE VM is using SPICE+VNC, in case of FIPS setup it's only SPICE (https://github.com/oVirt/ovirt-ansible-collection/blob/master/roles/hosted_engine_setup/tasks/create_target_vm/01_create_target_hosted_engine_vm.yml#L141)

This needs to be changed to VNC. There should be no reason for SPICE in either case, it's there probably from the times when we didn't have FIPS-enabled VNC.

Comment 2 Nikolai Sednev 2022-03-28 14:58:35 UTC
Do we have an exact instructions on how to enable the FIPS on already provisioned host, so I will be able to verify the FIPS enabled parameter during HE deployment on FIPS enabled RHEL8.6 host?

Comment 3 Nikolai Sednev 2022-03-28 14:59:42 UTC
My currently provisioned hosts has not been set for FIPS:
serval14 ~]# sysctl crypto.fips_enabled
crypto.fips_enabled = 0
cat /proc/sys/crypto/fips_enabled
0

Comment 4 Nikolai Sednev 2022-03-28 15:25:38 UTC
I followed the https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening#switching-the-system-to-fips-mode_using-the-system-wide-cryptographic-policies:

Enabled the FIPS on host:
serval15 ~]# fips-mode-setup --enable && reboot
serval15 ~]# fips-mode-setup --check
FIPS mode is enabled.
serval15 ~]# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
serval15 ~]# cat /proc/sys/crypto/fips_enabled
1
serval15 ~]# update-crypto-policies --show 
FIPS

Looks like that's it for enabling the FIPS on the host.
I tried to deploy HE using latest ovirt-hosted-engine-setup-2.6.2-1.el8ev.noarch and ovirt-ansible-collection-2.0.0-0.6.BETA.el8ev.noarch and failed with:

serval14 ~]# hosted-engine --deploy --4 --ansible-extra-vars=he_pause_before_engine_setup=true
***L:ERROR Internal error: No module named 'ansible.constants'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 141, in execute
    self.context.loadPlugins()
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 803, in loadPlugins
    self._loadPluginGroups(plugindir, needgroups, loadedgroups)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 112, in _loadPluginGroups
    self._loadPlugins(path, path, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 69, in _loadPlugins
    self._loadPlugins(base, d, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 100, in _loadPlugins
    os.path.basename(path),
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 110, in loadModule
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module>
    from . import misc
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 32, in <module>
    from ovirt_hosted_engine_setup import ansible_utils
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 35, in <module>
    from ansible.constants import AnsibleCallback
ModuleNotFoundError: No module named 'ansible.constants'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/__main__.py", line 88, in main
    installer.execute()
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 147, in execute
    sys.exc_info()[2],
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 85, in raiseExceptionInformation
    raise info[1].with_traceback(info[2])
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 141, in execute
    self.context.loadPlugins()
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 803, in loadPlugins
    self._loadPluginGroups(plugindir, needgroups, loadedgroups)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 112, in _loadPluginGroups
    self._loadPlugins(path, path, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 69, in _loadPlugins
    self._loadPlugins(base, d, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 100, in _loadPlugins
    os.path.basename(path),
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 110, in loadModule
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module>
    from . import misc
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 32, in <module>
    from ovirt_hosted_engine_setup import ansible_utils
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 35, in <module>
    from ansible.constants import AnsibleCallback
otopi.main.PluginLoadException: No module named 'ansible.constants'

Comment 5 Nikolai Sednev 2022-03-28 23:49:32 UTC
Just the same happens on regular host with disabled FIPS functionality:
serval16 ~]# update-crypto-policies --show 
DEFAULT
[root@serval16 ~]# hosted-engine --deploy --4 --ansible-extra-vars=he_pause_before_engine_setup=true
***L:ERROR Internal error: No module named 'ansible.constants'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 141, in execute
    self.context.loadPlugins()
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 803, in loadPlugins
    self._loadPluginGroups(plugindir, needgroups, loadedgroups)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 112, in _loadPluginGroups
    self._loadPlugins(path, path, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 69, in _loadPlugins
    self._loadPlugins(base, d, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 100, in _loadPlugins
    os.path.basename(path),
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 110, in loadModule
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module>
    from . import misc
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 32, in <module>
    from ovirt_hosted_engine_setup import ansible_utils
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 35, in <module>
    from ansible.constants import AnsibleCallback
ModuleNotFoundError: No module named 'ansible.constants'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/otopi/__main__.py", line 88, in main
    installer.execute()
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 147, in execute
    sys.exc_info()[2],
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 85, in raiseExceptionInformation
    raise info[1].with_traceback(info[2])
  File "/usr/lib/python3.6/site-packages/otopi/main.py", line 141, in execute
    self.context.loadPlugins()
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 803, in loadPlugins
    self._loadPluginGroups(plugindir, needgroups, loadedgroups)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 112, in _loadPluginGroups
    self._loadPlugins(path, path, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 69, in _loadPlugins
    self._loadPlugins(base, d, groupname)
  File "/usr/lib/python3.6/site-packages/otopi/context.py", line 100, in _loadPlugins
    os.path.basename(path),
  File "/usr/lib/python3.6/site-packages/otopi/util.py", line 110, in loadModule
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module>
    from . import misc
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 32, in <module>
    from ovirt_hosted_engine_setup import ansible_utils
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 35, in <module>
    from ansible.constants import AnsibleCallback
otopi.main.PluginLoadException: No module named 'ansible.constants'

Comment 6 Nikolai Sednev 2022-03-28 23:56:11 UTC
serval16 ~]# ansible --version
ansible [core 2.12.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 2.10.3
  libyaml = True

Comment 7 Asaf Rachmani 2022-03-29 09:46:40 UTC
This issue was introduced in ovirt-hosted-engine-setup-2.6.2-1, the fix for it has been recently merged and is planned to be included in the upcoming release, please avoid using this version.

Comment 8 Nikolai Sednev 2022-03-29 11:55:53 UTC
ovirt-hosted-engine-setup-2.6.3-1.el8ev.noarch deployment also fails:


serval14 ~]# hosted-engine --deploy --4 --ansible-extra-vars=he_pause_before_engine_setup=true[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
          Continuing will configure this host for serving as hypervisor and will create a local VM with a running engine.
          The locally running engine will be used to configure a new storage domain and create a VM there.
          At the end the disk of the local VM will be moved to the shared storage.
          Are you sure you want to continue? (Yes, No)[Yes]: 
          It has been detected that this program is executed through an SSH connection without using tmux.
          Continuing with the installation may lead to broken installation if the network connection fails.
          It is highly recommended to abort the installation and run it inside a tmux session using command "tmux".
          Do you want to continue anyway? (Yes, No)[No]: yes
          Configuration files: 
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20220329144634-3rvy5v.log
          Version: otopi-1.10.0 (otopi-1.10.0-1.el8ev)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup (late)
[ INFO  ] Stage: Environment customization
         
          --== STORAGE CONFIGURATION ==--
         
         
          --== HOST NETWORK CONFIGURATION ==--
         
          Please indicate the gateway IP address [10.35.235.254]: 
[ INFO  ] Checking available network interfaces:
[ ERROR ] b"ERROR! couldn't resolve module/action 'firewalld'. This often indicates a misspelling, missing collection, or incorrect module path.\n"
[ ERROR ] b'\n'
[ ERROR ] b"The error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/bootstrap_local_vm/05_add_host.yml': line 12, column 5, but may\n"
[ ERROR ] b'be elsewhere in the file depending on the exact syntax problem.\n'
[ ERROR ] b'\n'
[ ERROR ] b'The offending line appears to be:\n'
[ ERROR ] b'\n'
[ ERROR ] b'    delay: 20\n'
[ ERROR ] b'  - name: Open a port on firewalld\n'
[ ERROR ] b'    ^ here\n'
[ ERROR ] Failed to execute stage 'Environment customization': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ ERROR ] b"ERROR! couldn't resolve module/action 'firewalld'. This often indicates a misspelling, missing collection, or incorrect module path.\n"
[ ERROR ] b'\n'
[ ERROR ] b"The error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/bootstrap_local_vm/05_add_host.yml': line 12, column 5, but may\n"
[ ERROR ] b'be elsewhere in the file depending on the exact syntax problem.\n'
[ ERROR ] b'\n'
[ ERROR ] b'The offending line appears to be:\n'
[ ERROR ] b'\n'
[ ERROR ] b'    delay: 20\n'
[ ERROR ] b'  - name: Open a port on firewalld\n'
[ ERROR ] b'    ^ here\n'
[ ERROR ] Failed to execute stage 'Clean up': Failed executing ansible-playbook
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20220329145306.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20220329144634-3rvy5v.log


ansible-macros-2021.1.2-2.1.el8ev.noarch
ovirt-ansible-collection-2.0.0-0.6.BETA.el8ev.noarch
ansible-core-2.12.2-3.1.el8.x86_64
ovirt-hosted-engine-setup-2.6.3-1.el8ev.noarch
ovirt-hosted-engine-ha-2.5.0-1.el8ev.noarch

Comment 9 Asaf Rachmani 2022-03-29 12:05:30 UTC
Martin, how do we provide the non-community Ansible's modules?
As we discussed for HE, ansible.posix and ansible.netcommon are required.

Comment 10 Martin Necas 2022-03-29 12:16:43 UTC
For the upstream I have done builds in cbs for ansible-posix [1], ansible-utils [2], ansible-netcommon [3] collections. 
As for the downstream we have now a few issues and waiting till the tickets get resolved.


It is possible that they were not installed because the requirement for them was added in ovirt-ansible-collection-2.0.0-0.8.BETA and you have 0.6.BETA.

Why do you have installed ansible-macros?


[1] https://cbs.centos.org/koji/packageinfo?packageID=8469
[2] https://cbs.centos.org/koji/packageinfo?packageID=8492
[3] https://cbs.centos.org/koji/packageinfo?packageID=8471
[4] https://github.com/oVirt/ovirt-ansible-collection/pull/457

Comment 11 Nikolai Sednev 2022-04-04 17:57:39 UTC
serval15 ~]#
fips-mode-setup --enable && reboot
fips-mode-setup --check && sysctl crypto.fips_enabled && update-crypto-policies --show && cat /proc/sys/crypto/fips_enabled
/sys/crypto/fips_enabled
FIPS mode is enabled.
crypto.fips_enabled = 1
FIPS
1

Works fine on:
ansible-core-2.12.2-3.1.el8.x86_64
ovirt-hosted-engine-setup-2.6.3-1.el8ev.noarch
ovirt-hosted-engine-ha-2.5.0-1.el8ev.noarch
ovirt-ansible-collection-2.0.0-0.9.BETA.el8ev.noarch
Red Hat Enterprise Linux release 8.6 Beta (Ootpa)
Linux 4.18.0-372.5.1.el8.x86_64 #1 SMP Mon Mar 28 11:03:39 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Engine Software Version:4.5.0.1-605.90f87fe14688.14.el8ev

Successfully deployed HE over NFS, on FIPS enabled host.

Comment 12 Sandro Bonazzola 2022-04-20 06:33:59 UTC
This bugzilla is included in oVirt 4.5.0 release, published on April 20th 2022.

Since the problem described in this bug report should be resolved in oVirt 4.5.0 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.