Bug 1891582
| Summary: | [15 to 16.1] Major upgrade requires the undercloud's hostname in overcloud's /etc/hosts | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jose Luis Franco <jfrancoa> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Jose Luis Franco <jfrancoa> |
| Status: | CLOSED ERRATA | QA Contact: | Jose Luis Franco <jfrancoa> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 16.1 (Train) | CC: | fpantano, mburns |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-11.3.2-1.20200914170164.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-12-15 18:37:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This Bugzilla would block the 15 to 16.1 tested workflow as we could not continue with the upgrade. And there is an increasing interest of some of our customers to get from OSP15 to 16.1 now that 15 is EOL. Task now succeeds:
2020-11-18 20:15:10 | TASK [Pull latest cinder_volume images] ****************************************
2020-11-18 20:15:10 | Wednesday 18 November 2020 20:15:04 +0000 (0:00:00.139) 0:03:53.194 ****
2020-11-18 20:15:10 | changed: [controller-0] => {"changed": true, "cmd": ["podman", "pull", "undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-cinder-volume:16.1_20201111.1"], "delta": "0:00:04.895436", "end": "2020-11-18 20:15:10.195796", "rc": 0, "start": "2020-11-18 20:15:05.300360", "stderr": "Trying to pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-cinder-volume:16.1_20201111.1...\nGetting image source signatures\nCopying blob sha256:04ef0e69dcba4088b009d14d21e86cda00ddbf8e84a4d9746c5d8ec9d61803af\nCopying blob sha256:b565332d1d45150165f73227d2ec4e6ef0127e7b55fac73a5131468b25bc4bfb\nCopying blob sha256:ee6fdc0d5879c79553923d86c318bd053c868071414336bc60c44aa8c64e6890\nCopying blob sha256:8e537a66c8a27678f0243606aec7e8fd5eea8f20aefa7d78186ffdf0e678d6d6\nCopying blob sha256:f50127793b42ca73b3981e29d153c41206ba405975aa320e61740a5d3243ed01\nCopying blob sha256:cda0363e3d9ad18b31b8268158f3a3423696a0be2a8e6399ba305bd56cdc2514\nCopying config sha256:51828f9bd4810008be4659f28224d7ca2abaab3be47533f04cdcfdc3c6f72fdb\nWriting manifest to image destination\nStoring signatures", "stderr_lines": ["Trying to pull undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-cinder-volume:16.1_20201111.1...", "Getting image source signatures", "Copying blob sha256:04ef0e69dcba4088b009d14d21e86cda00ddbf8e84a4d9746c5d8ec9d61803af", "Copying blob sha256:b565332d1d45150165f73227d2ec4e6ef0127e7b55fac73a5131468b25bc4bfb", "Copying blob sha256:ee6fdc0d5879c79553923d86c318bd053c868071414336bc60c44aa8c64e6890", "Copying blob sha256:8e537a66c8a27678f0243606aec7e8fd5eea8f20aefa7d78186ffdf0e678d6d6", "Copying blob sha256:f50127793b42ca73b3981e29d153c41206ba405975aa320e61740a5d3243ed01", "Copying blob sha256:cda0363e3d9ad18b31b8268158f3a3423696a0be2a8e6399ba305bd56cdc2514", "Copying config sha256:51828f9bd4810008be4659f28224d7ca2abaab3be47533f04cdcfdc3c6f72fdb", "Writing manifest to image destination", "Storing signatures"], "stdout": "51828f9bd4810008be4659f28224d7ca2abaab3be47533f04cdcfdc3c6f72fdb", "stdout_lines": ["51828f9bd4810008be4659f28224d7ca2abaab3be47533f04cdcfdc3c6f72fdb"]}
After the fix tasks get run:
2020-11-18 20:14:32 | TASK [Make sure the Undercloud hostname is included in /etc/hosts] *************
2020-11-18 20:14:32 | Wednesday 18 November 2020 20:14:00 +0000 (0:00:00.668) 0:02:48.936 ****
2020-11-18 20:14:32 | changed: [controller-0] => {"backup": "", "changed": true, "msg": "line added"}
2020-11-18 20:14:32 |
2020-11-18 20:14:32 | TASK [Set container_registry_insecure_registries fact.] ************************
2020-11-18 20:14:32 | Wednesday 18 November 2020 20:14:01 +0000 (0:00:00.832) 0:02:49.768 ****
2020-11-18 20:14:32 | ok: [controller-0] => {"ansible_facts": {"container_registry_insecure_registries": ["undercloud-0.ctlplane.redhat.local:8787"]}, "changed": false}
Job Logs: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/upgrades/view/update/job/DFG-upgrades-updates-15-to-16.1-from-latest_cdn-HA-ipv4/7/
RPM:
openstack-tripleo-heat-templates-11.3.2-1.20200914170172.el8ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.3 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:5413 |
Description of problem: When performing the major upgrade from OSP15 to OSP16.1 using the minor update's workflow the procedure works at the time of upgrading the Controllers with: TASK [Get container cinder_volume image] *************************************** Monday 26 October 2020 12:20:58 -0400 (0:00:00.246) 0:21:08.923 ******** ok: [controller-0] => {"ansible_facts": {"cinder_volume_image": "undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume:16.1-48", "cinder_volume_image_latest": "cluster.common.tag/openstack-cinder-volume:pcmklatest"}, "changed": false} TASK [Pull latest cinder_volume images] **************************************** Monday 26 October 2020 12:20:58 -0400 (0:00:00.125) 0:21:09.049 ******** fatal: [controller-0]: FAILED! => {"changed": true, "cmd": ["podman", "pull", "undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume:16.1-48"], "delta" : "0:00:10.146390", "end": "2020-10-26 16:21:08.593937", "msg": "non-zero return code", "rc": 125, "start": "2020-10-26 16:20:58.447547", "stderr": "Trying to pull undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume:16.1-48...\n Get https://undercloud-0.ctlplane.redhat.local:8787/v2/: dial tcp: lookup undercloud-0.ctlpl ane.redhat.local: no such host\nError: error pulling image \"undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume:16.1-48\": unable to pull undercloud -0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume:16.1-48: unable to pull image: Error initializing source docker://undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume:16.1-48: error pinging docker registry undercloud-0.ctlplane.redhat.local:8787: Get https://undercloud-0.ctlplane.redhat.local:8787/v2/: d ial tcp: lookup undercloud-0.ctlplane.redhat.local: no such host", "stderr_lines": ["Trying to pull undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volu me:16.1-48...", " Get https://undercloud-0.ctlplane.redhat.local:8787/v2/: dial tcp: lookup undercloud-0.ctlplane.redhat.local: no such host", "Error: error pulling image \"undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume:16.1-48\": unable to pull undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-v olume:16.1-48: unable to pull image: Error initializing source docker://undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume:16.1-48: error pinging do cker registry undercloud-0.ctlplane.redhat.local:8787: Get https://undercloud-0.ctlplane.redhat.local:8787/v2/: dial tcp: lookup undercloud-0.ctlplane.redhat.local: no such host"], "stdout": "", "stdout_lines": []} NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* controller-0 : ok=46 changed=12 unreachable=0 failed=1 skipped=109 rescued=0 ignored=0 controller-1 : ok=4 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 controller-2 : ok=4 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Monday 26 October 2020 12:21:08 -0400 (0:00:10.426) 0:21:19.475 ******** =============================================================================== The reason is, because the hostname's registry isn't identified by the IP anymore but by the whole FQDN. This was fixed already on the upgrade_tasks: https://github.com/openstack/tripleo-heat-templates/commit/49fc109bb9ad7163730f7d08e7f8a5217fe01c68 But as we don't use the upgrade_tasks for this workflow, but the update_tasks we need to include them there too. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Deploy OSP15 2. Upgrade the Undercloud to OSP16.1 3. Upgrade the Overcloud to OSP16.1 Follow the steps described in: https://gitlab.cee.redhat.com/osp16/osp-upgrade-16 Actual results: Controller's upgrade fails as the overcloud node can't reach the configured Undercloud's registry. Expected results: Controller's upgrade succeeds. Additional info: