1846591 – openstack overcloud upgrade converge doesn't work after upgrade

Bug 1846591 - openstack overcloud upgrade converge doesn't work after upgrade

Summary: openstack overcloud upgrade converge doesn't work after upgrade

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-tripleoclient
Sub Component:
Version:	16.1 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	medium
Target Milestone:	rc
Target Release:	16.1 (Train on RHEL 8.2)
Assignee:	Jose Luis Franco
QA Contact:	Arik Chernetsky
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-06-12 03:27 UTC by jpateteg
Modified:	2020-07-29 07:53 UTC (History)
CC List:	12 users (show)
Fixed In Version:	python-tripleoclient-12.3.2-0.20200615103426.6f877f6.el8ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-07-29 07:53:11 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
OpenStack gerrit	731619	0	None	MERGED	Refactor upgrade prepare and converge.	2020-12-16 14:58:44 UTC
Red Hat Product Errata	RHBA-2020:3148	0	None	None	None	2020-07-29 07:53:32 UTC

Description jpateteg 2020-06-12 03:27:57 UTC

Description of problem:

After upgrading from OSP13 to OSP16.1 during the hackfest, could successfully upgrade 3xcontrollers + 3compute nodes. 

When trying to sync the platform with the command:

$ openstack overcloud upgrade converge \
    --stack STACK NAME \
    --templates \
    -e <ENVIRONMENT FILE>

I receive an error and no synchronization occurs.

Version-Release number of selected component (if applicable):
OSP16.1

How reproducible:
Everytime the command "openstack overcloud upgrade converge " is executed (with parameters or without parameters I receive the following error:

(undercloud) [stack@director ~]$ openstack overcloud upgrade converge
2020-06-11 22:58:39.657 142621 ERROR osc_lib.shell [-] Exception raised: module 'tripleoclient.v1.overcloud_upgrade' has no attribute 'UpgradeConvergeOvercloud'
2020-06-11 22:58:39.657 142621 INFO osc_lib.shell [-] END return value: 1

Steps to Reproduce:
1. Upgrade from OSP13 to OSP16.1 (Using the framework and workarounds created for the HackFest)
2. execute the openstack overcloud upgrade converge (with or without parameters)
3.

Actual results:

2020-06-11 22:58:39.657 142621 ERROR osc_lib.shell [-] Exception raised: module 'tripleoclient.v1.overcloud_upgrade' has no attribute 'UpgradeConvergeOvercloud'
2020-06-11 22:58:39.657 142621 INFO osc_lib.shell [-] END return value: 1

Expected results:

Synchronization should start and complete

Additional info:

Containers running:

[heat-admin@overcloud-home-lab-compute-2 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                                                                   COMMAND      CREATED      STATUS          PORTS  NAMES
aa25549c6eb7  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-compute:16.1_20200515.3               kolla_start  2 hours ago  Up 2 hours ago         nova_compute
c4d2effa284c  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-openvswitch-agent:16.1_20200515.3  kolla_start  2 hours ago  Up 2 hours ago         neutron_ovs_agent
51a9cd9c869b  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-compute:16.1_20200515.3               kolla_start  2 hours ago  Up 2 hours ago         nova_migration_target
c0e6ce8b0070  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-cron:16.1_20200515.3                       kolla_start  2 hours ago  Up 2 hours ago         logrotate_crond
309814561f51  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-iscsid:16.1_20200515.3                     kolla_start  2 hours ago  Up 2 hours ago         iscsid
ffad71293cbc  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-libvirt:16.1_20200515.3               kolla_start  2 hours ago  Up 2 hours ago         nova_libvirt
bac0788de542  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-libvirt:16.1_20200515.3               kolla_start  2 hours ago  Up 2 hours ago         nova_virtlogd
[heat-admin@overcloud-home-lab-compute-2 ~]$ logout
Connection to 172.16.0.192 closed.
[stack@director ~]$ ssh heat-admin.0.191
This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

Last login: Thu Jun 11 21:29:27 2020 from 172.16.0.18
[heat-admin@overcloud-home-lab-compute-1 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                                                                   COMMAND      CREATED         STATUS             PORTS  NAMES
1f21800100b3  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-compute:16.1_20200515.3               kolla_start  23 minutes ago  Up 23 minutes ago         nova_compute
b0b7527ccb37  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-openvswitch-agent:16.1_20200515.3  kolla_start  26 minutes ago  Up 26 minutes ago         neutron_ovs_agent
73d62238d494  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-compute:16.1_20200515.3               kolla_start  26 minutes ago  Up 26 minutes ago         nova_migration_target
69489cd0b83a  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-cron:16.1_20200515.3                       kolla_start  26 minutes ago  Up 26 minutes ago         logrotate_crond
ab7dd437da7b  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-iscsid:16.1_20200515.3                     kolla_start  30 minutes ago  Up 30 minutes ago         iscsid
ef73f9c1f530  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-libvirt:16.1_20200515.3               kolla_start  30 minutes ago  Up 30 minutes ago         nova_libvirt
2a97cbfe307d  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-libvirt:16.1_20200515.3               kolla_start  30 minutes ago  Up 30 minutes ago         nova_virtlogd
[heat-admin@overcloud-home-lab-compute-1 ~]$ logout
Connection to 172.16.0.191 closed.
[stack@director ~]$ ssh heat-admin.0.190
This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

Last login: Thu Jun 11 22:12:24 2020 from 172.16.0.18
[heat-admin@overcloud-home-lab-compute-0 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                                                                   COMMAND      CREATED      STATUS          PORTS  NAMES
020869d5dbda  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-compute:16.1_20200515.3               kolla_start  3 hours ago  Up 3 hours ago         nova_compute
1caa18b88859  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-openvswitch-agent:16.1_20200515.3  kolla_start  3 hours ago  Up 3 hours ago         neutron_ovs_agent
1b349f79a206  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-compute:16.1_20200515.3               kolla_start  3 hours ago  Up 3 hours ago         nova_migration_target
520dd34fc2c2  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-cron:16.1_20200515.3                       kolla_start  3 hours ago  Up 3 hours ago         logrotate_crond
828133997fa2  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-iscsid:16.1_20200515.3                     kolla_start  3 hours ago  Up 3 hours ago         iscsid
4dac7aa71392  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-libvirt:16.1_20200515.3               kolla_start  3 hours ago  Up 3 hours ago         nova_libvirt
312f52cace4b  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-libvirt:16.1_20200515.3               kolla_start  3 hours ago  Up 3 hours ago         nova_virtlogd
[heat-admin@overcloud-home-lab-compute-0 ~]$ logout
Connection to 172.16.0.190 closed.
[stack@director ~]$ ssh heat-admin.0.187
This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

Last login: Thu Jun 11 22:12:01 2020 from 172.16.0.18
[heat-admin@overcloud-home-lab-controller-0 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                                                                   COMMAND               CREATED       STATUS           PORTS  NAMES
57c6b07e9dde  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-proxy-server:16.1_20200515.3         kolla_start           5 hours ago   Up 5 hours ago          swift_proxy
14223f0c4f81  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-object:16.1_20200515.3               kolla_start           5 hours ago   Up 5 hours ago          swift_rsync
c5f1e9fc5aa3  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-object:16.1_20200515.3               kolla_start           5 hours ago   Up 5 hours ago          swift_object_updater
a78a6305bc87  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-object:16.1_20200515.3               kolla_start           5 hours ago   Up 5 hours ago          swift_object_server
318d83054677  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-object:16.1_20200515.3               kolla_start           5 hours ago   Up 5 hours ago          swift_object_replicator
66165b7a680e  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-proxy-server:16.1_20200515.3         kolla_start           5 hours ago   Up 5 hours ago          swift_object_expirer
fbca716331bd  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-object:16.1_20200515.3               kolla_start           5 hours ago   Up 5 hours ago          swift_object_auditor
48e78998507b  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-container:16.1_20200515.3            kolla_start           5 hours ago   Up 5 hours ago          swift_container_updater
b67e2aa715bc  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-container:16.1_20200515.3            kolla_start           5 hours ago   Up 5 hours ago          swift_container_server
56333bb91713  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-container:16.1_20200515.3            kolla_start           5 hours ago   Up 5 hours ago          swift_container_replicator
f96fdb7e5fa4  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-container:16.1_20200515.3            kolla_start           5 hours ago   Up 5 hours ago          swift_container_auditor
45c18df8b1e7  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-account:16.1_20200515.3              kolla_start           5 hours ago   Up 5 hours ago          swift_account_server
2dd650683756  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-account:16.1_20200515.3              kolla_start           5 hours ago   Up 5 hours ago          swift_account_replicator
1824dadfce7e  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-account:16.1_20200515.3              kolla_start           5 hours ago   Up 5 hours ago          swift_account_reaper
c9cf32a03f0a  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-swift-account:16.1_20200515.3              kolla_start           5 hours ago   Up 5 hours ago          swift_account_auditor
a54031490355  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-mariadb:16.1_20200515.3                    /bin/bash /usr/lo...  5 hours ago   Up 5 hours ago          galera-bundle-podman-0
e310b02a884c  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-l3-agent:16.1_20200515.3           /bin/bash -c HAPR...  11 hours ago  Up 11 hours ago         neutron-haproxy-qrouter-0a514eba-fa2a-4879-8a12-8b0f34186f85
e4e1e247432a  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-l3-agent:16.1_20200515.3           /usr/sbin/keepali...  11 hours ago  Up 11 hours ago         neutron-keepalived-qrouter-0a514eba-fa2a-4879-8a12-8b0f34186f85
cf3bac109e8c  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-dhcp-agent:16.1_20200515.3         /usr/sbin/dnsmasq...  13 hours ago  Up 13 hours ago         neutron-dnsmasq-qdhcp-753d07de-f2b9-46e6-a5fb-28e81c67e004
9ddbb2074c71  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-dhcp-agent:16.1_20200515.3         /usr/sbin/dnsmasq...  13 hours ago  Up 13 hours ago         neutron-dnsmasq-qdhcp-18b5215b-96d0-4109-b2d3-936b20da0834
e8ae8dceb0d8  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-cinder-volume:16.1_20200515.3              /bin/bash /usr/lo...  13 hours ago  Up 13 hours ago         openstack-cinder-volume-podman-0
2d2f20c2d37d  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-haproxy:16.1_20200515.3                    /bin/bash /usr/lo...  13 hours ago  Up 13 hours ago         haproxy-bundle-podman-0
5afb6103dd08  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-rabbitmq:16.1_20200515.3                   /bin/bash /usr/lo...  13 hours ago  Up 13 hours ago         rabbitmq-bundle-podman-0
29cae7c62c39  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-openvswitch-agent:16.1_20200515.3  kolla_start           32 hours ago  Up 11 hours ago         neutron_ovs_agent
21cdcaa00d1b  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-metadata-agent:16.1_20200515.3     kolla_start           32 hours ago  Up 13 hours ago         neutron_metadata_agent
1afd3a4661c2  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-l3-agent:16.1_20200515.3           kolla_start           32 hours ago  Up 12 hours ago         neutron_l3_agent
e375ce4c87b1  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-dhcp-agent:16.1_20200515.3         kolla_start           32 hours ago  Up 13 hours ago         neutron_dhcp
f4ecea377b45  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200515.3                   kolla_start           32 hours ago  Up 13 hours ago         nova_api_cron
82d3f0a07a9a  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200515.3                   kolla_start           32 hours ago  Up 13 hours ago         nova_metadata
bd5dbac6ec49  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200515.3                   kolla_start           32 hours ago  Up 13 hours ago         nova_api
ea68da4d1db9  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-glance-api:16.1_20200515.3                 kolla_start           32 hours ago  Up 13 hours ago         glance_api
32a1cdcf1aaf  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-placement-api:16.1_20200515.3              kolla_start           32 hours ago  Up 13 hours ago         placement_api
014866af33da  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-novncproxy:16.1_20200515.3            kolla_start           32 hours ago  Up 13 hours ago         nova_vnc_proxy
abb73cea4786  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-scheduler:16.1_20200515.3             kolla_start           32 hours ago  Up 11 hours ago         nova_scheduler
4310ffdc0b6e  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-nova-conductor:16.1_20200515.3             kolla_start           32 hours ago  Up 13 hours ago         nova_conductor
52c3267fe45e  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-neutron-server:16.1_20200515.3             kolla_start           32 hours ago  Up 13 hours ago         neutron_api
dad8305fe135  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-cron:16.1_20200515.3                       kolla_start           32 hours ago  Up 13 hours ago         logrotate_crond
9298d3b0b959  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-heat-engine:16.1_20200515.3                kolla_start           32 hours ago  Up 13 hours ago         heat_engine
770cbdac2bc2  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-heat-api:16.1_20200515.3                   kolla_start           32 hours ago  Up 13 hours ago         heat_api_cron
b578d3d70fe4  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-heat-api-cfn:16.1_20200515.3               kolla_start           32 hours ago  Up 13 hours ago         heat_api_cfn
7e7cd10a0dcb  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-heat-api:16.1_20200515.3                   kolla_start           32 hours ago  Up 13 hours ago         heat_api
cd216f28a497  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-cinder-scheduler:16.1_20200515.3           kolla_start           32 hours ago  Up 13 hours ago         cinder_scheduler
2143882c488b  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-cinder-api:16.1_20200515.3                 kolla_start           32 hours ago  Up 13 hours ago         cinder_api_cron
4a0bfb0dfeca  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-cinder-api:16.1_20200515.3                 kolla_start           32 hours ago  Up 13 hours ago         cinder_api
d1116adf7721  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-keystone:16.1_20200515.3                   kolla_start           32 hours ago  Up 13 hours ago         keystone
c967850561c7  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-iscsid:16.1_20200515.3                     kolla_start           32 hours ago  Up 13 hours ago         iscsid
3ca7e5f71c6a  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-horizon:16.1_20200515.3                    kolla_start           32 hours ago  Up 13 hours ago         horizon
9fab5e98278b  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-mariadb:16.1_20200515.3                    kolla_start           32 hours ago  Up 13 hours ago         clustercheck
3588fb036169  director.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-memcached:16.1_20200515.3                  kolla_start           32 hours ago  Up 13 hours ago         memcached

Comment 1 Jose Luis Franco 2020-06-15 12:56:35 UTC

Hello,

So, the problem here is related to this patch: https://review.opendev.org/#/c/731619/, which is included in the workarounds file used for the hackfest: https://gitlab.cee.redhat.com/osp16/ffwd2/-/blob/hackfest-US/infrared/workarounds/ffwd2_workarounds_unsubscribed.yaml#L199-203

This patch includes two changes, one in the python-tripleoclient and another in the naming class for the converge. However, this change doesn't get properly applied via the workarounds system and what happens is that your python code changed the class name to UpgradeConverge but your entry-points.txt file still points at UpgradeConvergeOvercloud for the overcloud upgrade converge command.

I'll remove the class name change from the patch (it can be done as a follow up thing) so this doesn't occur again.

If you want to get your converge working simply rename UpgradeConvergeOvercloud into UpgradeConverge in the /usr/lib/python3.6/site-packages/python_tripleoclient-12.3.2.dev75-py3.6.egg-info/entry_points.txt

Comment 2 Derek Li 2020-06-15 15:21:20 UTC

I tried the workaround ("rename UpgradeConvergeOvercloud into UpgradeConverge") and it worked & resumed my stack converge.

Comment 18 errata-xmlrpc 2020-07-29 07:53:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3148

Note You need to log in before you can comment on or make changes to this bug.