Bug 2051832

Summary: [OSP13 -> OSP16.2] External upgrade tasks do not run on any nodes
Product: Red Hat OpenStack Reporter: Jose Luis Franco <jfrancoa>
Component: python-tripleoclientAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED ERRATA QA Contact: Jason Grosso <jgrosso>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 16.2 (Train)CC: drosenfe, hbrock, jpretori, jslagle, lbezdick, mburns, sathlang, spower, supadhya, tvignaud
Target Milestone: z2Keywords: Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-tripleoclient-12.6.1-2.20220114104841.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-23 22:13:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jose Luis Franco 2022-02-08 07:02:27 UTC
Description of problem:

Currently, the FFU upgrade from OSP13 to OSP16.2 relies quite intensively on the external upgrade tasks, however a recent change merged in python-tripleoclient has cuased a regression in the functionality:

https://review.opendev.org/c/openstack/python-tripleoclient/+/784416

Now, when running some of the external upgrade tasks, the ansible code is not able to interpret the inventory, as the used inventory file contains the location of the inventory to be using:

Inventory in a 16.2 mistral's config download:
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-upgrades-ffu-16.2-from-13-latest_cdn-3cont_2comp_1ceph-ipv4-stf/78/undercloud-0/var/lib/mistral/3af96284-de18-4888-aece-b771bed00a5b/inventory.yaml.gz

This causes the execution of commands like transfer_data or continers_image_prepare to run successfully but not execute any task on any node:

2022-02-07 12:50:31 | [Mon Feb  7 12:50:31 UTC 2022] Start preparing container images for overcloud upgrade
2022-02-07 12:50:33 | 2022-02-07 12:50:33.256 155027 INFO osc_lib.shell [-] command: overcloud external-upgrade run -> tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun (auth=True)[00m
2022-02-07 12:50:33 | 2022-02-07 12:50:33.257 155027 INFO osc_lib.clientmanager [-] Using auth plugin: password[00m
2022-02-07 12:50:33 | 2022-02-07 12:50:33.258 155027 DEBUG osc_lib.clientmanager [-] Using parameters {'auth_url': 'https://192.168.24.2:13000', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'} setup_auth /usr/lib/python3.6/site-packages/osc_lib/clientmanager.py:157[00m
2022-02-07 12:50:33 | 2022-02-07 12:50:33.260 155027 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python3.6/site-packages/osc_lib/clientmanager.py:201[00m
2022-02-07 12:50:37 | 2022-02-07 12:50:37.836 155027 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.[00m
2022-02-07 12:50:45 | [WARNING]: Could not match supplied host pattern, ignoring: Undercloud
2022-02-07 12:50:45 | 
2022-02-07 12:50:45 | PLAY [Clear cached facts] ******************************************************
2022-02-07 12:50:45 | 
2022-02-07 12:50:45 | PLAY [Gather facts from undercloud] ********************************************
2022-02-07 12:50:45 | skipping: no hosts matched
2022-02-07 12:50:45 | [WARNING]: Could not match supplied host pattern, ignoring: overcloud
.....
2022-02-07 12:50:45 | PLAY RECAP *********************************************************************
2022-02-07 12:50:45 | /home/stack/tripleo-ansible-inventory.yaml : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0  

http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-upgrades-ffu-16.2-from-13-latest_cdn-3cont_2comp_2net-ipv4-ovn_vlan-provider-network/102/undercloud-0/home/stack/overcloud_upgrade_prepare_containers.log.gz

This bug breaks the FFU procedure for OSP16.2 completely.

CI job: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/upgrades/view/ffu/job/DFG-upgrades-ffu-16.2-from-13-latest_cdn-3cont_2comp_1ceph-ipv4-stf/78/

Version-Release number of selected component (if applicable):


How reproducible:
Always. Run any FFU job from OSP13 to OSP16.2

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Lukas Bezdicka 2022-02-08 20:15:06 UTC
Patch 828282 does not address the issue as stable/train overcloud_external_upgrade has different codebase. I'm going for full revert of change https://review.opendev.org/c/openstack/python-tripleoclient/+/828234

Comment 3 Sofer Athlan-Guyot 2022-02-16 17:39:56 UTC
This affect update as well, thanks for the revert and +1 for the blocker.

Comment 12 errata-xmlrpc 2022-03-23 22:13:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1001