Description of problem: When performing the Undercloud upgrade from OSP16.2 to OSP17.0, during the heat-ephemeral migration stage, the upgrade fails due to missing authentication: 2022-01-12 10:59:58 | 2022-01-12 10:59:58.442155 | 5254008d-a1ec-ed91-2c16-0000000001ed | FATAL | Run undercloud-upgrade-ephemeral-heat.py | undercloud-0 | error={"changed": true, "cmd": "/var/lib/tripleo-config/scripts/undercloud-upgrade-ephemeral-heat.py", "delta": "0:00:04.954994", "end": "2022-01-12 10:59:58.413851", "msg": "non-zero return code", "rc": 1, "start": "2022-01-12 10:59:53.458857", "stderr": "INFO:undercloud:Exporting network from stack qe-Cloud-0 to /home/stack/overcloud-deploy/qe-Cloud-0/tripleo-qe-Cloud-0-network-data.yaml\n/usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change. To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`.\n DeprecationWarning\nMissing value auth-url required for auth plugin password\nTraceback (most recent call last):\n File \"/var/lib/tripleo-config/scripts/undercloud-upgrade-ephemeral-heat.py\", line 406, in <module>\n main()\n File \"/var/lib/tripleo-config/scripts/undercloud-upgrade-ephemeral-heat.py\", line 373, in main\n export_networks(stack, stack_dir)\n File \"/var/lib/tripleo-config/scripts/undercloud-upgrade-ephemeral-heat.py\", line 274, in export_networks\n '--yes'])\n File \"/usr/lib64/python3.6/subprocess.py\", line 311, in check_call\n raise CalledProcessError(retcode, cmd)\nsubprocess.CalledProcessError: Command '['openstack', 'overcloud', 'network', 'extract', '--stack', 'qe-Cloud-0', '--output', '/home/stack/overcloud-deploy/qe-Cloud-0/tripleo-qe-Cloud-0-network-data.yaml', '--yes']' returned non-zero exit status 1.", "stderr_lines": ["INFO:undercloud:Exporting network from stack qe-Cloud-0 to /home/stack/overcloud-deploy/qe-Cloud-0/tripleo-qe-Cloud-0-network-data.yaml", "/usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change. To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`.", " DeprecationWarning", "Missing value auth-url required for auth plugin password", "Traceback (most recent call last):", " File \"/var/lib/tripleo-config/scripts/undercloud-upgrade-ephemeral-heat.py\", line 406, in <module>", " main()", " File \"/var/lib/tripleo-config/scripts/undercloud-upgrade-ephemeral-heat.py\", line 373, in main", " export_networks(stack, stack_dir)", " File \"/var/lib/tripleo-config/scripts/undercloud-upgrade-ephemeral-heat.py\", line 274, in export_networks", " '--yes'])", " File \"/usr/lib64/python3.6/subprocess.py\", line 311, in check_call", " raise CalledProcessError(retcode, cmd)", "subprocess.CalledProcessError: Command '['openstack', 'overcloud', 'network', 'extract', '--stack', 'qe-Cloud-0', '--output', '/home/stack/overcloud-deploy/qe-Cloud-0/tripleo-qe-Cloud-0-network-data.yaml', '--yes']' returned non-zero exit status 1."], "stdout": "", "stdout_lines": []} 2022-01-12 10:59:58 | 2022-01-12 10:59:58.443741 | 5254008d-a1ec-ed91-2c16-0000000001ed | TIMING | Run undercloud-upgrade-ephemeral-heat.py | undercloud-0 | 0:00:56.736860 | 5.18s Logs: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-upgrades-ffu-17.0-from-16.2-latest_cdn-3cont_2comp_3ceph_1ipa-ipv4-ovn_dvr/10/undercloud-0/home/stack/ffu_undercloud_upgrade.log.gz Command used: #!/bin/bash # # Perform undercloud upgrade and related steps set -eu UPGRADE_ANSWER="" if openstack undercloud upgrade --help | grep -qe "--yes"; then UPGRADE_ANSWER="--yes" fi set -o pipefail openstack undercloud upgrade ${UPGRADE_ANSWER} Job logs: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-upgrades-ffu-17.0-from-16.2-latest_cdn-3cont_2comp_3ceph_1ipa-ipv4-ovn_dvr/10/ Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
The patches that add an entry for ephemeral heat to clouds.yaml are not relevant here. The upgrade script needs to talk the undercloud's installed Heat instance (before it is migrated) to migrate this data. Sourcing stackrc is the correct fix here, and we probably need to add a check for that in the openstack undercloud upgrade command before we start the Heat upgrade.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577