Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1584608

Summary:	Openshift-on-OpenStack install playbook fails installing python-docker package
Product:	OpenShift Container Platform	Reporter:	Jon Uriarte <juriarte>
Component:	Installer	Assignee:	Tomas Sedovic <tsedovic>
Status:	CLOSED ERRATA	QA Contact:	Jon Uriarte <juriarte>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	3.10.0	CC:	aos-bugs, jokerman, mmccomas
Target Milestone:	---
Target Release:	3.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-07-30 19:16:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jon Uriarte 2018-05-31 09:56:54 UTC

Description of problem:

Since 'python-docker' package installation was added to prerequisites playbook [1], the Openshift-on-OpenStack install playbook fails, as it imports prerequisites playbook before OCP repositories are configured in Openshift RHEL 7.5 nodes. When prerequisites playbook tries to install 'python-docker' it fails because the package is not found.

The python-docker package is shipped in OCP repositories (and not in RHEL 7.5 repositories), and the OCP repositories are configured in Openshift nodes after prerequisites playbook is executed.

[1] https://github.com/openshift/openshift-ansible/commit/d3b5b493994ee2cf7358a000d800d77bd0521082#diff-a07a29ccd9bae4a306dfbdccaec5a961

Version-Release number of the following components:
[cloud-user@ansible-host ~]$ rpm -q openshift-ansible
openshift-ansible-3.10.0-0.53.0.git.0.53fe016.el7.noarch

[cloud-user@ansible-host ~]$ rpm -q ansible
ansible-2.4.4.0-1.el7ae.noarch

[cloud-user@ansible-host ~]$ ansible --version
ansible 2.4.4.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

How reproducible: always

Steps to Reproduce:
1. Get OCP openshift-ansible downstream rpm
2. Configure OSP and OCP inventory files
3. Run:
ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.yml

ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml

ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory red-hat-ca.yml

ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/repos.yml

ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.yml
   
Actual results:

TASK [Ensure openshift-ansible installer package deps are installed] **********************************************************************************************************************************************
ok: [infra-node-0.openshift.example.com] => (item=iproute)
ok: [master-0.openshift.example.com] => (item=iproute)
ok: [app-node-0.openshift.example.com] => (item=iproute)
ok: [app-node-1.openshift.example.com] => (item=iproute)
ok: [app-node-0.openshift.example.com] => (item=dbus-python)
ok: [infra-node-0.openshift.example.com] => (item=dbus-python)
ok: [master-0.openshift.example.com] => (item=dbus-python)
ok: [app-node-1.openshift.example.com] => (item=dbus-python)
ok: [master-0.openshift.example.com] => (item=PyYAML)
ok: [app-node-0.openshift.example.com] => (item=PyYAML)
ok: [infra-node-0.openshift.example.com] => (item=PyYAML)
ok: [app-node-1.openshift.example.com] => (item=PyYAML)
ok: [app-node-0.openshift.example.com] => (item=python-ipaddress)
ok: [app-node-1.openshift.example.com] => (item=python-ipaddress)
ok: [master-0.openshift.example.com] => (item=python-ipaddress)
ok: [infra-node-0.openshift.example.com] => (item=python-ipaddress)
ok: [master-0.openshift.example.com] => (item=libsemanage-python)
ok: [infra-node-0.openshift.example.com] => (item=libsemanage-python)
ok: [app-node-0.openshift.example.com] => (item=libsemanage-python)
ok: [app-node-1.openshift.example.com] => (item=libsemanage-python)
ok: [infra-node-0.openshift.example.com] => (item=yum-utils)
ok: [app-node-0.openshift.example.com] => (item=yum-utils)
ok: [app-node-1.openshift.example.com] => (item=yum-utils)
ok: [master-0.openshift.example.com] => (item=yum-utils)
FAILED - RETRYING: Ensure openshift-ansible installer package deps are installed (2 retries left).
FAILED - RETRYING: Ensure openshift-ansible installer package deps are installed (2 retries left).
FAILED - RETRYING: Ensure openshift-ansible installer package deps are installed (2 retries left).
FAILED - RETRYING: Ensure openshift-ansible installer package deps are installed (2 retries left).
FAILED - RETRYING: Ensure openshift-ansible installer package deps are installed (1 retries left).
FAILED - RETRYING: Ensure openshift-ansible installer package deps are installed (1 retries left).
FAILED - RETRYING: Ensure openshift-ansible installer package deps are installed (1 retries left).
FAILED - RETRYING: Ensure openshift-ansible installer package deps are installed (1 retries left).
failed: [app-node-0.openshift.example.com] (item=python-docker) => {"attempts": 2, "changed": false, "failed": true, "item": "python-docker", "msg": "No package matching 'python-docker' found available, installe
d or updated", "rc": 126, "results": ["No package matching 'python-docker' found available, installed or updated"]}
failed: [infra-node-0.openshift.example.com] (item=python-docker) => {"attempts": 2, "changed": false, "failed": true, "item": "python-docker", "msg": "No package matching 'python-docker' found available, instal
led or updated", "rc": 126, "results": ["No package matching 'python-docker' found available, installed or updated"]}
failed: [app-node-1.openshift.example.com] (item=python-docker) => {"attempts": 2, "changed": false, "failed": true, "item": "python-docker", "msg": "No package matching 'python-docker' found available, installe
d or updated", "rc": 126, "results": ["No package matching 'python-docker' found available, installed or updated"]}
failed: [master-0.openshift.example.com] (item=python-docker) => {"attempts": 2, "changed": false, "failed": true, "item": "python-docker", "msg": "No package matching 'python-docker' found available, installed 
or updated", "rc": 126, "results": ["No package matching 'python-docker' found available, installed or updated"]}

NO MORE HOSTS LEFT ************************************************************************************************************************************************************************************************
 [WARNING]: Could not create retry file '/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.retry'.         [Errno 13] Permission denied: u'/usr/share/ansible/openshift-
ansible/playbooks/openstack/openshift-cluster/install.retry'

PLAY RECAP ********************************************************************************************************************************************************************************************************
app-node-0.openshift.example.com : ok=19   changed=2    unreachable=0    failed=1   
app-node-1.openshift.example.com : ok=18   changed=2    unreachable=0    failed=1   
infra-node-0.openshift.example.com : ok=18   changed=2    unreachable=0    failed=1   
localhost                  : ok=13   changed=0    unreachable=0    failed=0   
master-0.openshift.example.com : ok=22   changed=2    unreachable=0    failed=1   

  1. Hosts:    app-node-0.openshift.example.com, app-node-1.openshift.example.com, infra-node-0.openshift.example.com, master-0.openshift.example.com
     Play:     Install packages necessary for installer
     Task:     Ensure openshift-ansible installer package deps are installed
     Message:  All items completed

Expected results:
Successful run.

Additional info:
As a workaround the package name in https://github.com/openshift/openshift-ansible/blob/master/playbooks/init/base_packages.yml#L42 can be changed, 'python-docker' by 'python-docker-py', which exists in RHEL 7.5 repositories.

Comment 1 Scott Dodson 2018-05-31 12:41:28 UTC

This package comes from the OCP channels, are those not enabled?

Comment 2 Tomas Sedovic 2018-05-31 13:00:37 UTC

If the repos are set up using `openshift_additional_repos`, that get's set up after the dependencies are installed.

The error happens here, calling out to playbooks/init/base_packages.yml:

https://github.com/openshift/openshift-ansible/blob/master/playbooks/prerequisites.yml#L4

But the additional repos are set up (playbooks/init/repos.yml) a few lines after:

https://github.com/openshift/openshift-ansible/blob/master/playbooks/prerequisites.yml#L11

I'm wondering if we can just change the order here.

Comment 3 Tomas Sedovic 2018-05-31 14:28:52 UTC

Jon, would you check whether that this fixes your issue?

https://github.com/openshift/openshift-ansible/pull/8583

Comment 4 Jon Uriarte 2018-05-31 16:05:53 UTC

Tried it manually but the additional repos are not set as 'oo_all_hosts' is not defined when importing init/repos.yml playbook.

 2018-05-31 11:48:51,557 p=14992 u=cloud-user |  PLAY [Setup yum repositories for all hosts]
 **************************************************************************************************
 2018-05-31 11:48:51,557 p=14992 u=cloud-user |  skipping: no hosts matched 

'Setup yum repositories for all hosts' playbook is skipped and the install playbook fails at the same stage (when trying to install python-docker package)

    "item": "python-docker",
    "msg": "No package matching 'python-docker' found available, installed or updated",
    "rc": 126,
    "results": [
        "No package matching 'python-docker' found available, installed or updated"
    ] 

  1. Hosts:    app-node-0.openshift.example.com, app-node-1.openshift.example.com, infra-node-0.openshift.example.com, master-0.openshift.example.com
     Play:     Install packages necessary for installer
     Task:     Ensure openshift-ansible installer package deps are installed
     Message:  All items completed

Comment 5 Jon Uriarte 2018-06-01 10:14:39 UTC

Tried manually the latest changes in the PR and it works now.
The 'python-docker' package is correctly installed from the OCP repository, which is installed in a previous stage.

[openshift@master-0 ~]$ yum list python-docker
Loaded plugins: search-disabled-repos
Installed Packages     python-docker.noarch    2.4.2-1.3.el7    @ose-devel

Comment 8 Jon Uriarte 2018-06-05 14:40:19 UTC

Verified in openshift-ansible-3.10.0-0.58.0, the installer finishes successfully and the python-docker package is installed in the Openshift nodes correctly from the OCP repository.

[openshift@master-0 ~]$ yum list python-docker
Loaded plugins: search-disabled-repos
Installed Packages
python-docker.noarch               2.4.2-1.3.el7             @ose-devel

[openshift@master-0 ~]$ cat /etc/yum.repos.d/ose-devel.repo 
[ose-devel]
baseurl = http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/3.10/latest/x86_64/os/
enabled = 1
gpgcheck = 0
name = ose-devel
sslverify = 1

Comment 10 errata-xmlrpc 2018-07-30 19:16:51 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816