Bug 1840009
| Summary: | Allow OSP13 minor upgrade when overcloud is in FAILED state | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eduard Barrera <ebarrera> |
| Component: | openstack-heat | Assignee: | Rabi Mishra <ramishra> |
| Status: | CLOSED ERRATA | QA Contact: | nlevinki <nlevinki> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 13.0 (Queens) | CC: | drosenfe, emacchi, jschluet, mburns, ramishra, sbaker, shardy |
| Target Milestone: | --- | Keywords: | Triaged, ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-heat-10.0.3-14.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-28 18:30:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (openstack-heat bug fix advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4395 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |
Description of problem: IHAC customer wishing to upgrade the undercloud from Z5 to Z11 but it fails because they have th overcloud stack in faulty state $ openstack stack list +--------------------------------------+------------+----------------------------------+---------------+----------------------+----------------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+------------+----------------------------------+---------------+----------------------+----------------------+ | e3c49acb-b96e-4c6a-bfa0-23b640835dd6 | us-east1 | 0428af0771db4e0395577876a18eafdc | UPDATE_FAILED | 2019-05-20T22:31:52Z | 2020-02-01T10:06:40Z | +--------------------------------------+------------+----------------------------------+---------------+----------------------+----------------------+ $ openstack undercloud upgrade Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager rhel-7-server-extras-rpms | 2.0 kB 00:00:00 rhel-7-server-openstack-13-rpms | 2.3 kB 00:00:00 rhel-7-server-rh-common-rpms | 2.1 kB 00:00:00 rhel-7-server-rhceph-3-tools-rpms | 2.3 kB 00:00:00 rhel-7-server-rpms | 2.0 kB 00:00:00 rhel-ha-for-rhel-7-server-rpms | 2.0 kB 00:00:00 No packages marked for update Uploading Enabled Repositories Report Loaded plugins: product-id, subscription-manager 2020-05-15 09:09:14,720 INFO: Check the stack status to be able to migrate heat data. 2020-05-15 09:09:18,143 ERROR: Can not upgrade undercloud with FAILED overcloud <======================== Command '['instack-pre-upgrade-undercloud']' returned non-zero exit status 1 (undercloud) [stack@director ~]$ My suggestion was to run the overcloud deploy command to return the status of the overcloud status to a healty status, but it seems complex in their environment: The change we tried to make in last deployment was to update overcloud_images.yaml with ovn & neutron hotfix images. We tried to apply the hotfix container images through deployment process, which was not successful. So we ended up using paunch to replace ovn/neutron containers with hotfix images manually. If we re-run the deploy now, we probably have to switch back to old version of ovn/neutron containers via paunch first, run deploy, and apply hotfix image via paunch again. We have had a few outages during above process, so we'd like to avoid it if all possible. According to customer the stack is faulty because they ran an overcloud update but they subscription expired. Question: Can we use openstack undercloud upgrade --force or a manual database hack to change the status of the overcloud stack to continue with the upgrade ? $ openstack undercloud upgrade --help ... --force No to be used in normal update/upgrade! This helps getting out of an error loop when the overcloud is in error and it needs new code to work again. Version-Release number of selected component (if applicable): OSP13z5 to OSP13z11