Description of problem: While scaling out our CephStorage nodes, we encountered an error during the ironic deploy phase of the overcloud-minimal image. The deploy systematically failed on all new nodes with the backtrace below available [1] and the provisioning status moved to `deploy failed` Version-Release number of selected component (if applicable): Red Hat OpenStack 16.2.z3 ceph VMs based on RHEL 8.6 How reproducible: The problem is observed using the overcloud-minimal image Actual results: All activities planned in pre-production and production are frozen. Customer is asking for an urgent hotfix. Expected results: New nodes added and correctly configured as ceph nodes of Red Hat OpenStack 16.2 [1] Backtrace observed: Oct 13 05:08:53 host-10-244-20-27 ironic-python-agent[3154]: 2022-10-13 05:08:53.125 3154 ERROR ironic_python_agent.extensions.image [-] Error setting up bootloader. Error Command execution failed: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command. Command: mount /dev/sda3 /tmp/tmp2wce2sw2 Exit code: 32 Stdout: '' Stderr: 'mount: /tmp/tmp2wce2sw2: /dev/sda3 already mounted on /tmp/tmp2wce2sw2.\n'.: ironic_python_agent.errors.CommandExecutionError: Command execution failed: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command. Oct 13 05:08:53 host-10-244-20-27 ironic-python-agent[3154]: 2022-10-13 05:08:53.255 3154 ERROR root [-] Command execution error: Command execution failed: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command. Command: mount /dev/sda3 /tmp/tmp2wce2sw2 Exit code: 32 Stdout: '' Stderr: 'mount: /tmp/tmp2wce2sw2: /dev/sda3 already mounted on /tmp/tmp2wce2sw2.\n'.: ironic_python_agent.errors.CommandExecutionError: Command execution failed: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command. Command: mount /dev/sda3 /tmp/tmp2wce2sw2 Exit code: 32 Stdout: '' Stderr: 'mount: /tmp/tmp2wce2sw2: /dev/sda3 already mounted on /tmp/tmp2wce2sw2.\n'. 2022-10-13 05:08:53.255 3154 ERROR root Traceback (most recent call last): 2022-10-13 05:08:53.255 3154 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/image.py", line 518, in _install_grub2 2022-10-13 05:08:53.255 3154 ERROR root utils.execute('mount', root_partition, path) 2022-10-13 05:08:53.255 3154 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/utils.py", line 82, in execute 2022-10-13 05:08:53.255 3154 ERROR root return ironic_utils.execute(*cmd, **kwargs) 2022-10-13 05:08:53.255 3154 ERROR root File "/usr/lib/python3.6/site-packages/ironic_lib/utils.py", line 99, in execute 2022-10-13 05:08:53.255 3154 ERROR root result = processutils.execute(*cmd, **kwargs) 2022-10-13 05:08:53.255 3154 ERROR root File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 431, in execute 2022-10-13 05:08:53.255 3154 ERROR root cmd=sanitized_cmd) 2022-10-13 05:08:53.255 3154 ERROR root oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. 2022-10-13 05:08:53.255 3154 ERROR root Command: mount /dev/sda3 /tmp/tmp2wce2sw2 2022-10-13 05:08:53.255 3154 ERROR root Exit code: 32 2022-10-13 05:08:53.255 3154 ERROR root Stdout: '' 2022-10-13 05:08:53.255 3154 ERROR root Stderr: 'mount: /tmp/tmp2wce2sw2: /dev/sda3 already mounted on /tmp/tmp2wce2sw2.\n' 2022-10-13 05:08:53.255 3154 ERROR root 2022-10-13 05:08:53.255 3154 ERROR root During handling of the above exception, another exception occurred: 2022-10-13 05:08:53.255 3154 ERROR root 2022-10-13 05:08:53.255 3154 ERROR root Traceback (most recent call last): 2022-10-13 05:08:53.255 3154 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/base.py", line 256, in execute_command 2022-10-13 05:08:53.255 3154 ERROR root result = ext.execute(command_part, **kwargs) 2022-10-13 05:08:53.255 3154 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/base.py", line 208, in execute 2022-10-13 05:08:53.255 3154 ERROR root return cmd(**kwargs) 2022-10-13 05:08:53.255 3154 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/base.py", line 326, in wrapper 2022-10-13 05:08:53.255 3154 ERROR root result = func(self, **command_params) 2022-10-13 05:08:53.255 3154 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/image.py", line 951, in install_bootloader 2022-10-13 05:08:53.255 3154 ERROR root prep_boot_part_uuid=prep_boot_part_uuid) 2022-10-13 05:08:53.255 3154 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/image.py", line 617, in _install_grub2 2022-10-13 05:08:53.255 3154 ERROR root raise errors.CommandExecutionError(error_msg) 2022-10-13 05:08:53.255 3154 ERROR root ironic_python_agent.errors.CommandExecutionError: Command execution failed: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command. 2022-10-13 05:08:53.255 3154 ERROR root Command: mount /dev/sda3 /tmp/tmp2wce2sw2 2022-10-13 05:08:53.255 3154 ERROR root Exit code: 32 2022-10-13 05:08:53.255 3154 ERROR root Stdout: '' 2022-10-13 05:08:53.255 3154 ERROR root Stderr: 'mount: /tmp/tmp2wce2sw2: /dev/sda3 already mounted on /tmp/tmp2wce2sw2.\n'. 2022-10-13 05:08:53.255 3154 ERROR root
Tested and verified with rhosp-director-images-minimal-16.2-20221103.2.el8ost.noarch.rpm which uses openstack-ironic-python-agent-5.0.5-2.20220204025306.el8ost.noarch.rpm
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.4), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8794
*** Bug 2227529 has been marked as a duplicate of this bug. ***
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days