Description of problem: It appears a difference between the 20200130 and 20200227 versions of the ipa images are resulting in the following traceback when using UEFI: Mar 04 18:09:03 host-172-18-171-112 ironic-python-agent[9106]: 2020-03-04 18:09:03.460 9106 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): efibootmgr --version execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:372 Mar 04 18:09:03 host-172-18-171-112 ironic-python-agent[9106]: 2020-03-04 18:09:03.493 9106 DEBUG oslo_concurrency.processutils [-] 'efibootmgr --version' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:457 Mar 04 18:09:03 host-172-18-171-112 ironic-python-agent[9106]: 2020-03-04 18:09:03.523 9106 ERROR root [-] Command execution error: [Errno 2] No such file or directory: 'efibootmgr': 'efibootmgr': FileNotFoundError: [Errno 2] No such file or directory: 'efibootmgr': 'efibootmgr' 2020-03-04 18:09:03.523 9106 ERROR root Traceback (most recent call last): 2020-03-04 18:09:03.523 9106 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/base.py", line 252, in execute_command 2020-03-04 18:09:03.523 9106 ERROR root result = ext.execute(command_part, **kwargs) 2020-03-04 18:09:03.523 9106 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/base.py", line 205, in execute 2020-03-04 18:09:03.523 9106 ERROR root return cmd(**kwargs) 2020-03-04 18:09:03.523 9106 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/base.py", line 321, in wrapper 2020-03-04 18:09:03.523 9106 ERROR root result = func(self, **command_params) 2020-03-04 18:09:03.523 9106 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/image.py", line 520, in install_bootloader 2020-03-04 18:09:03.523 9106 ERROR root utils.execute('efibootmgr', '--version') 2020-03-04 18:09:03.523 9106 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/utils.py", line 74, in execute 2020-03-04 18:09:03.523 9106 ERROR root return ironic_utils.execute(*cmd, **kwargs) 2020-03-04 18:09:03.523 9106 ERROR root File "/usr/lib/python3.6/site-packages/ironic_lib/utils.py", line 99, in execute 2020-03-04 18:09:03.523 9106 ERROR root result = processutils.execute(*cmd, **kwargs) 2020-03-04 18:09:03.523 9106 ERROR root File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 391, in execute 2020-03-04 18:09:03.523 9106 ERROR root env=env_variables) 2020-03-04 18:09:03.523 9106 ERROR root File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__ 2020-03-04 18:09:03.523 9106 ERROR root restore_signals, start_new_session) 2020-03-04 18:09:03.523 9106 ERROR root File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child 2020-03-04 18:09:03.523 9106 ERROR root raise child_exception_type(errno_num, err_msg, err_filename) 2020-03-04 18:09:03.523 9106 ERROR root FileNotFoundError: [Errno 2] No such file or directory: 'efibootmgr': 'efibootmgr' 2020-03-04 18:09:03.523 9106 ERROR root Mar 04 18:09:03 host-172-18-171-112 ironic-python-agent[9106]: 2020-03-04 18:09:03.933 9106 INFO root [-] Command image.install_bootloader completed: Command name: image.install_bootloader, params: {'root_uuid': '63a12e8d-4563-453e-9df5-d62a8fc0f0ea', 'efi_system_part_uuid': '496E-F9C1', 'prep_boot_part_uuid': None}, status: FAILED, result: None. Version-Release number of selected component (if applicable): It basically looks like the efibootmgr is missing in rhosp-director-images-ipa-x86_64-16.0-20200226.1.el8ost.noarch. How reproducible: Always. use 20200130 it works if use the 20200226 version it does not. Steps to Reproduce: 1. Try introspecting a UEFI enabled host using the ipa image affected Actual results: - FAILS with traceback provided. Expected results: - SUCCESS Additional info:
efibootmgr was added as a requirement for ironic-python-agent, it was included in diskimage-builder here https://review.opendev.org/#/c/707195/. It looks like the rhosp-director-images-ipa-x86_64 you are testing was built with diskimage-builder-2.33.1-0.20200204145616.500e60d.el8ost which did not have the packages. The next compose will be built with diskimage-builder-2.33.1-0.20200226061509.18f46bd.el8ost so it should be OK then.
Can you retest with the version in FIV?
Upstream we have a fix for this on master [1] we did the cherry-pick to stable/train[2] and it will probably merge soon. [1] https://review.opendev.org/#/c/709107 [2] https://review.opendev.org/#/c/711326
Bob, Where can we obtain the rpm you want us to test? Cory
Coming in via Bug 1809939, but if the customer is hitting an IPA issue they should only use the rhosp-director-images-ipa and not update the overcloud-full/octavia images from the hotfix patch. If they use the newer IPA image with the overcloud full the customer won't hit Bug 1809939.
Good point Alex. The rhosp-director-images-ipa-x86_64-16.0-20200305.1.el8ost.noarch.rpm should only be used from the package listed in the FixedInVersion.
For anyone having issues with this and to clarify comment 7: ### Issue ### * installed the hotfix from https://bugzilla.redhat.com/show_bug.cgi?id=1810604 RPM: rhosp-director-images-16.0-20200305.1.el8ost * used all of the images provided in that file. * did this because booting with UEFI did not work as the efibootmgr binary was missing from the ironic python agent image: ~~~ Mar 04 18:09:03 host-172-18-171-112 ironic-python-agent[9106]: 2020-03-04 18:09:03.523 9106 ERROR root [-] Command execution error: [Errno 2] No such file or directory: 'efibootmgr': 'efibootmgr': FileNotFoundError: [Errno 2] No such file or directory: 'efibootmgr': 'efibootmgr' ~~~ The reason is explained here: https://bugzilla.redhat.com/show_bug.cgi?id=1810604#c1 ~~~ efibootmgr was added as a requirement for ironic-python-agent, it was included in diskimage-builder here https://review.opendev.org/#/c/707195/. It looks like the rhosp-director-images-ipa-x86_64 you are testing was built with diskimage-builder-2.33.1-0.20200204145616.500e60d.el8ost which did not have the packages. The next compose will be built with diskimage-builder-2.33.1-0.20200226061509.18f46bd.el8ost so it should be OK then. ~~~ So, as a next step, ran `yum localinstall rhosp-director-images-ipa-x86_64-16.0-20200305.1.el8ost.noarch` and `yum localinstall rhosp-director-images-x86_64-16.0-20200305.1.el8ost.noarch`: ~~~ $ grep rhosp-director-images installed-rpms rhosp-director-images-16.0-20200226.1.el8ost.noarch Thu Mar 5 17:25:45 2020 rhosp-director-images-ipa-16.0-20200226.1.el8ost.noarch Thu Mar 5 17:25:46 2020 rhosp-director-images-ipa-x86_64-16.0-20200226.1.el8ost.noarch Thu Mar 5 11:18:13 2020 rhosp-director-images-ipa-x86_64-16.0-20200305.1.el8ost.noarch Fri Mar 6 17:32:59 2020 rhosp-director-images-x86_64-16.0-20200226.1.el8ost.noarch Thu Mar 5 11:18:42 2020 rhosp-director-images-x86_64-16.0-20200305.1.el8ost.noarch Fri Mar 6 17:33:44 2020 ~~~ And then: ~~~ mkdir images cd images for i in /usr/share/rhosp-director-images/overcloud-full-latest-16.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-16.0.tar; do tar -xvf $i; done openstack overcloud image upload --image-path /home/stack/images/ --update-existing ~~~ This updated the IPA kernel and ramdisk: ~~~ (overcloud) [stack@undercloud-0 ~]$ rpm -qpl rhosp-director-images-ipa-x86_64-16.0-20200305.1.el8ost.noarch.rpm warning: rhosp-director-images-ipa-x86_64-16.0-20200305.1.el8ost.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f21541eb: NOKEY /usr/share/rhosp-director-images/ironic-python-agent-16.0-20200305.1.el8ost.x86_64.tar /usr/share/rhosp-director-images/version-16.0-20200305.1.el8ost.txt (overcloud) [stack@undercloud-0 ~]$ tar -tf /usr/share/rhosp-director-images/ironic-python-agent-16.0-20200305.1.el8ost.x86_64.tar ironic-python-agent.initramfs ironic-python-agent.kernel ~~~ And the overcloud-full images: ~~~ (overcloud) [stack@undercloud-0 ~]$ rpm -qpl rhosp-director-images-x86_64-16.0-20200305.1.el8ost.noarch.rpm warning: rhosp-director-images-x86_64-16.0-20200305.1.el8ost.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f21541eb: NOKEY /usr/share/rhosp-director-images/overcloud-full-16.0-20200305.1.el8ost.x86_64.tar /usr/share/rhosp-director-images/version-16.0-20200305.1.el8ost.txt (overcloud) [stack@undercloud-0 ~]$ tar -tf /usr/share/rhosp-director-images/overcloud-full-16.0-20200305.1.el8ost.x86_64.tar overcloud-full.qcow2 overcloud-full.initrd overcloud-full.vmlinuz overcloud-full-rpm.manifest overcloud-full-signature.manifest ~~~ However, this leads to: https://bugzilla.redhat.com/show_bug.cgi?id=1810294 which in turn leads to https://bugzilla.redhat.com/show_bug.cgi?id=1809939 According to https://bugzilla.redhat.com/show_bug.cgi?id=1809939#c21: ~~~ The issue stems from the hotfix for Bug 1810604. The provided images include the latest version of director packages and to hotfix just THT (as being requested) will also require additional matching packages (tripleo-ansible, tripleo-common, puppet-tripleo, paunch, etc). The solution for the customer would be to get the patch for Bug 1810604 applied to the latest released director content. ~~~ And looking at: https://bugzilla.redhat.com/show_bug.cgi?id=1810604#c7 ~~~ Coming in via Bug 1809939, but if the customer is hitting an IPA issue they should only use the rhosp-director-images-ipa and not update the overcloud-full/octavia images from the hotfix patch. If they use the newer IPA image with the overcloud full the customer won't hit Bug 1809939. ~~~ And https://bugzilla.redhat.com/show_bug.cgi?id=1810604#c8: ~~~ Good point Alex. The rhosp-director-images-ipa-x86_64-16.0-20200305.1.el8ost.noarch.rpm should only be used from the package listed in the FixedInVersion. ~~~ It is clear that we should only use the IPA images, not the overcloud-full.* images. ### Correct way ### Instead, the solution is to use the old overcloud images (20200226.1) with an updated ipa kernel/initrd (20200305.1): ~~~ sudo dnf remove rhosp-director-images-16.0-20200305.1.el8ost.noarch rhosp-director-images-ipa-x86_64-16.0-20200305.1.el8ost.noarch rhosp-director-images-x86_64-16.0-20200305.1.el8ost.noarch octavia-amphora-image-x86_64-16.0-20200305.1.el8ost.noarch sudo dnf reinstall rhosp-director-images rhosp-director-images-ipa octavia-amphora-image cd images for i in /usr/share/rhosp-director-images/overcloud-full-latest-16.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-16.0.tar; do tar -xvf $i; done sudo rpm2cpio /home/redhat01/rhosp-director-images-ipa-x86_64-16.0-20200305.1.el8ost.noarch.rpm | cpio -idmv tar -xf ./usr/share/rhosp-director-images/ironic-python-agent-16.0-20200305.1.el8ost.x86_64.tar . openstack overcloud image upload --image-path /home/stack/images/ --update-existing ~~~
*** Bug 1812149 has been marked as a duplicate of this bug. ***
See comment 9 for correct method to install package. Verfied that updated package solves the missing file issue.
Hello, we have the same issue with rhosp-director-images-ipa-x86_64-16.0-20200226 I would try the method described in comment 9, but I don't now how to obtain the rpm. [root@ospd-stage ~]# dnf search --all --showduplicate rhosp-director-images Updating Subscription Management repositories. /usr/lib/python3.6/site-packages/dateutil/parser/_parser.py:70: UnicodeWarning: decode() called on unicode string, see https://bugzilla.redhat.com/show_bug.cgi?id=1693751 instream = instream.decode() Last metadata expiration check: 0:00:27 ago on Fri 24 Apr 2020 09:49:37 AM CEST. ========================================================================== Name Exactly Matched: rhosp-director-images =========================================================================== rhosp-director-images-16.0-20200226.1.el8ost.noarch : Red Hat OpenStack Platform director Images rhosp-director-images-16.0-20200130.1.el8ost.noarch : Red Hat OpenStack Platform director Images rhosp-director-images-16.0-20200226.1.el8ost.noarch : Red Hat OpenStack Platform director Images ============================================================================== Name Matched: rhosp-director-images =============================================================================== rhosp-director-images-ipa-16.0-20200226.1.el8ost.noarch : x86_64 Ironic Python Agent Images rhosp-director-images-ipa-16.0-20200130.1.el8ost.noarch : x86_64 Ironic Python Agent Images rhosp-director-images-all-16.0-20200130.1.el8ost.noarch : Meta package for overcloud/Ironic images for all architectures rhosp-director-images-all-16.0-20200226.1.el8ost.noarch : Meta package for overcloud/Ironic images for all architectures rhosp-director-images-ipa-16.0-20200226.1.el8ost.noarch : x86_64 Ironic Python Agent Images rhosp-director-images-x86_64-16.0-20200226.1.el8ost.noarch : x86_64 Overcloud images rhosp-director-images-x86_64-16.0-20200130.1.el8ost.noarch : x86_64 Overcloud images rhosp-director-images-x86_64-16.0-20200226.1.el8ost.noarch : x86_64 Overcloud images rhosp-director-images-ppc64le-16.0-20200130.1.el8ost.noarch : ppc64le Overcloud images rhosp-director-images-minimal-16.0-20200130.1.el8ost.noarch : director minimal image rhosp-director-images-minimal-16.0-20200226.1.el8ost.noarch : director minimal image rhosp-director-images-ppc64le-16.0-20200226.1.el8ost.noarch : ppc64le Overcloud images rhosp-director-images-ipa-x86_64-16.0-20200130.1.el8ost.noarch : x86_64 Ironic Python Agent Images rhosp-director-images-ipa-x86_64-16.0-20200226.1.el8ost.noarch : x86_64 Ironic Python Agent Images rhosp-director-images-ipa-x86_64-16.0-20200130.1.el8ost.noarch : x86_64 Ironic Python Agent Images rhosp-director-images-ipa-x86_64-16.0-20200226.1.el8ost.noarch : x86_64 Ironic Python Agent Images rhosp-director-images-ipa-ppc64le-16.0-20200130.1.el8ost.noarch : ppc64le Ironic Python Agent Images rhosp-director-images-ipa-ppc64le-16.0-20200226.1.el8ost.noarch : ppc64le Ironic Python Agent Images
Hello, Please open a support case with Red Hat. Refer to this bug report and request that support provide you with hotfix rhosp-director-images-16.0-20200305.1.el8ost A support engineer will download the RPM and share it with you via the case. Kind regards, Andreas
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2166