Bug 1902849

Summary: osp13-osp16.1 ffu fails on clusters previously upgraded from osp8, osp10
Product: Red Hat OpenStack Reporter: Jeremy <jmelvin>
Component: rhosp-directorAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED WONTFIX QA Contact: Jason Grosso <jgrosso>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: jpretori, lbezdick, mburns, morazi, sathlang, vgrosu
Target Milestone: z16Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-16 11:32:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jeremy 2020-11-30 19:35:36 UTC
Description of problem:
on step 19.2 of the upgrade documentation -- > https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/framework_for_upgrades_13_to_16.1/upgrading-an-overcloud-with-external-ceph-deployments_upgrading-overcloud-external-ceph#running-the-overcloud-upgrade-preparation-upgrading-overcloud-external-ceph

The upgrade command which is failing is -->  openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit xxxx

Failure message :


2020-11-24 12:55:22,143 p=100846 u=mistral n=ansible | TASK [tripleo-podman : Stop a container] ***************************************
2020-11-24 12:55:22,143 p=100846 u=mistral n=ansible | Tuesday 24 November 2020  12:55:22 -0500 (0:00:00.326)       0:00:32.456 ******
2020-11-24 12:55:22,805 p=100846 u=mistral n=ansible | failed: [xxxxx] (item=985976509ce8) => {"ansible_loop_var": "item", "changed": false, "item": "985976509ce8", "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on xxxxx's Python /usr/bin/python. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` or `pip install docker-py` (Python 2.6). The error was: No module named docker"}
2020-11-24 12:55:23,159 p=100846 u=mistral n=ansible | failed: [xxxx] (item=06ce45f22ce4) => {"ansible_loop_var": "item", "changed": false, "item": "06ce45f22ce4", "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on xxxxx's Python /usr/bin/python. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` or `pip install docker-py` (Python 2.6). The error was: No module named docker"}


Version-Release number of selected component (if applicable):
osp13

How reproducible:
unknown

Steps to Reproduce:
1. attempt ffu on systems that were upgraded from osp10 timeframe to osp13
2.
3.

Actual results:
update fails

Expected results:
update works.

Additional info:

I found that the package python-docker contains the module needed it seems. This rpm is in the osp13 image, however the rpm isn't on osp10 systems. So this problem occurs when older environments are upgraded. 


[root@undercloud-0 docker]# grep -ir "import docker" /usr/lib/python2.7/site-packages/docker
/usr/lib/python2.7/site-packages/docker/__init__.py:from .client import DockerClient, from_env
/usr/lib/python2.7/site-packages/docker/api/client.py:        >>> import docker
...
...
[root@undercloud-0 docker]# rpm -q --whatprovides /usr/lib/python2.7/site-packages/docker/__init__.py
python-docker-2.4.2-2.el7.noarch

Comment 2 Sofer Athlan-Guyot 2020-12-07 14:16:24 UTC
Hi,

one way is to make sure we get it installed during osp13 update. Let's add the necessary snippet there.

Comment 4 Sofer Athlan-Guyot 2021-03-05 16:45:34 UTC
Hi Jeremy,

sorry for the long delay. Here is an upstream patch, I'll just need to do sanity check on it, but according to your description that should do the trick.

Comment 6 Sofer Athlan-Guyot 2021-03-15 13:57:09 UTC
Adding it to the undercloud as well.

Comment 10 Vlada Grosu 2021-06-18 11:55:42 UTC
I've added a note to the Framework for Upgrades (13 to 16.1) guide to point to the KB solution to avoid this BZ:

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/framework_for_upgrades_13_to_16.1/index?lb_target=production#known-issues-that-might-block-an-upgrade

Thank you

Comment 13 Lukas Bezdicka 2021-08-16 11:32:16 UTC
The code side of fix would have to go into OSP13 and it still does not guarantee that users will actually update to this version. It's better to document the issue and just make sure users with so long living environments will apply the workaround.