Description of problem: While upgrading from RHOS13 z2 to RHOS14 with infrared, undercloud upgrade fails: TASK [Online data migration for Ironic] *************************************************************************************************************************************************************************** fatal: [undercloud-0 -> 192.168.24.1]: FAILED! => {"changed": true, "cmd": ["docker", "exec", "ironic_api", "ironic-dbsync", "--config-file", "/etc/ironic/ironic.conf", "online_data_migrations"], "delta": "0:00:00.066630", "end": "2019-01-09 08:00:26.470229", "msg": "non-zero return code", "rc": 1, "start": "2019-01-09 08:00:26.403599", "stderr": "Error response from daemon: No such container: ironic_api", "stderr_lines": ["Error response from daemon: No such container: ironic_api"], "stdout": "", "stdout_lines": []} Version-Release number of selected component (if applicable): RHOS13 puddle: 2018-08-22.2 RHOS14: 2019-01-08.1 How reproducible: 100% Steps to Reproduce: 1. Install RHOS13 with enable_telemetry=true in undercloud.conf and custom_templates: parameter_defaults: GnocchiBackend: file 2. Follow the upgrade scenario for infrared: https://gitlab.cee.redhat.com/yprokule/RHOS-13-14-upgrade : export MIRROR=tlv export PUDDLE=2019-01-08.1 infrared tripleo-undercloud \ --upgrade yes \ --mirror ${MIRROR} \ --build ${PUDDLE} \ --ansible-args="tags=upgrade_repos,undercloud_version_discovery,undercloud_containers" \ --registry-mirror rhos-qe-mirror-$MIRROR.usersys.redhat.com:5000 It passes. 3. Add container_images_file = /home/stack/containers-prepare-parameter.yaml under [DEFAULT] section of undercloud.conf Add docker_insecure_registries = rhos-qe-mirror-qeos.usersys.redhat.com:5000,brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888,registry.access.redhat.com 4. Run openstack undercloud upgrade -y 2>&1 Actual results: TASK [Online data migration for Nova] ***************************************************************************************************************************************************************************** skipping: [undercloud-0] TASK [Online data migration for Ironic] *************************************************************************************************************************************************************************** fatal: [undercloud-0 -> 192.168.24.1]: FAILED! => {"changed": true, "cmd": ["docker", "exec", "ironic_api", "ironic-dbsync", "--config-file", "/etc/ironic/ironic.conf", "online_data_migrations"], "delta": "0:00:00.066630", "end": "2019-01-09 08:00:26.470229", "msg": "non-zero return code", "rc": 1, "start": "2019-01-09 08:00:26.403599", "stderr": "Error response from daemon: No such container: ironic_api", "stderr_lines": ["Error response from daemon: No such container: ironic_api"], "stdout": "", "stdout_lines": []} Expected results: Upgrade passes Additional info: attached
Created attachment 1519508 [details] /home/stack from undercloud
From install-undercloud.log, you are having problems just downloading images. This isn't Ironic specific. WARNING:tripleo_common.image.image_uploader:docker pull failed: Get https://rhos-qe-mirror-tlv.usersys.redhat.com:5000/v1/_ping: http: server gave HTTP response to HTTPS client Traceback (most recent call last): File "/usr/bin/tripleo-container-image-prepare", line 100, in <module> env, roles_data, cleanup=args.cleanup) File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 193, in container_images_prepare_multi uploader.upload() File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 132, in upload uploader.run_tasks() File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 514, in run_tasks result = self.upload_image(*first) File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 257, in upload_image DockerImageUploader._pull(dockerc, repo, tag=source_tag) File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 241, in wrapped_f return self.call(f, *args, **kw) File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 330, in call start_time=start_time) File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 297, in iter raise retry_exc.reraise() File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 136, in reraise raise self.last_attempt.result() File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result return self.__get_result() File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 333, in call result = fn(*args, **kwargs) File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 295, in _pull raise ImageUploaderException('Could not pull image %s' % image) tripleo_common.image.exception.ImageUploaderException: Could not pull image rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp14/openstack-tempest
Created attachment 1520468 [details] undercloud files after failed upgrade
This bug is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1647956, the fix for it is already merged waiting to be submitted: https://code.engineering.redhat.com/gerrit/#/c/158245/5 Feel free to re-open if needed. *** This bug has been marked as a duplicate of bug 1647956 ***
*** Bug 1658375 has been marked as a duplicate of this bug. ***