Bug 1564076

Summary: prerequiste playbook fails when using openshift_docker_disable_push_dockerhub=True
Product: OpenShift Container Platform Reporter: Paul Dwyer <pdwyer>
Component: InstallerAssignee: Michael Gugino <mgugino>
Status: CLOSED ERRATA QA Contact: Gan Huang <ghuang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: aos-bugs, jokerman, mgugino, mmccomas, snalawad
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-17 06:43:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Paul Dwyer 2018-04-05 09:54:02 UTC
Description of problem:
The following is set in ansible inventory
openshift_docker_disable_push_dockerhub=True 

When running the pre-requisite playbook it fails as docker does not restart

Looks like when the above variable is set the following option is added to /etc/sysconfig/docker

--confirm-def-push=true

Docker is complaining that this is not a supported flag

Version-Release number of the following components:
rpm -q openshift-ansible
openshift-ansible-3.9.14-1.git.3.c62bc34.el7.noarch

rpm -q ansible
ansible-2.4.3.0-1.el7ae.noarch

rpm -q docker
docker-1.13.1-53.git774336d.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create inventory with openshift_docker_disable_push_dockerhub=True
2. Run the prequisites.yml playbook
3. restart container runtime fails

Actual results:
restarting of container runtime fails for each node
fatal: [x.x.x.x]: FAILED! => {"attempts": 3, "changed": false, "msg": "Unable to start service docker: Job for docker.service failed because the control process exited with error code. See \"systemctl status docker.service\" and \"journalctl -xe\" for details.\n"}

Expected results:
Docker to restart successfully and playbook to complete without error

Additional info:
The same ansible variable has been used in inventories to install previous OpenShift versions without issue

Comment 3 Scott Dodson 2018-04-05 12:48:29 UTC
Can we get `journalctl --no-pager -u docker.service` so we see why docker has failed to start?

Comment 5 Michael Gugino 2018-04-06 14:38:47 UTC
This seems to be a RHEL specific RPM patch to docker according to this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1245472

Either the source of docker was not RHEL or we have removed this custom flag.  I will investigate if this flag is still available.

Comment 7 Michael Gugino 2018-04-06 17:42:57 UTC
This option appears to have been removed from docker years ago.

$ git branch --contains 599c714f31360a2c95fa0a838e9ca7df3222e9e4 -r
  origin/rhel7-1.9


That commit added the confirm-push functionality.  It is only present in 1.9 from current docker source, github.com/projectatomic/docker

That functionality never made it into 1.10 or later.  Looks like we should remove this option entirely.  I'm not sure if the functionality was replicated by some other means.

Comment 8 Michael Gugino 2018-04-09 15:00:29 UTC
PR Created: https://github.com/openshift/openshift-ansible/pull/7858

Comment 9 Michael Gugino 2018-04-10 15:51:07 UTC
As Scott noted in the PR, this functionality is not obsolete, it has been moved into a plugin for docker named docker-rhel-push-plugin.x86_64 

I verified we are still shipping this in system-containerized docker.  We should add the necessary package to container_runtime install playbook to allow users to continue with functionality.

Comment 10 Michael Gugino 2018-04-10 17:03:39 UTC
PR Created in Master: https://github.com/openshift/openshift-ansible/pull/7884

Comment 11 Michael Gugino 2018-04-10 20:36:31 UTC
3.7 Backport Created: https://github.com/openshift/openshift-ansible/pull/7892

Comment 12 Michael Gugino 2018-04-10 22:24:33 UTC
Master PR Merged.

3.9 Backport Created: https://github.com/openshift/openshift-ansible/pull/7898

Comment 14 Michael Gugino 2018-04-11 16:13:59 UTC
Looks like the method of activating or deactivating this plugin has changed as well.  I'll need to investigate how to appropriately use the plugin after the package is installed.

Comment 15 Michael Gugino 2018-04-11 19:57:13 UTC
Okay, so I have looked into this some more.

rhel-push-plugin is enabled by default with docker in RHEL.  This prevents pushing to docker.io if the source of an image is RH content.

We don't have any options to disable this plugin, and the plugin doesn't take any arguments.  It's either enabled (by us, in docker.service unit), or it's not enabled (eg, CentOS).

I think the correct course of action is indeed to remove that boolean.  There is no more 'prompt/confirm' functionality, that has been removed.

Comment 16 Michael Gugino 2018-04-12 19:29:24 UTC
Actual fix that removes the variable: https://github.com/openshift/openshift-ansible/pull/7858

Merged in master.

3.9 Backport tbd

Comment 17 Michael Gugino 2018-04-13 14:15:42 UTC
3.9 Backport created: https://github.com/openshift/openshift-ansible/pull/7949

3.7 Backport created: https://github.com/openshift/openshift-ansible/pull/7951

Comment 18 Scott Dodson 2018-04-13 14:35:34 UTC
(In reply to Michael Gugino from comment #17)
> 3.9 Backport created:
> https://github.com/openshift/openshift-ansible/pull/7949
> 
> 3.7 Backport created:
> https://github.com/openshift/openshift-ansible/pull/7951

We should really be creating clones of these bugs whenever we backport so that QE can verify them independently. This bug we'll limit to 3.9. Once QE has verified the fix there we'll complete the backports to older releases and clone this bug for testing.

The 3.9 PR is merged, moving to MODIFIED

Comment 20 Gan Huang 2018-04-18 09:57:07 UTC
Verified in openshift-ansible-3.9.22-1.git.7.92620c6.el7.noarch.rpm

The parameter is removed from installer.

The installation is able to proceed and succeeded with the variable set.

No new test case needed for this functionality.

Comment 22 Gan Huang 2018-04-20 03:27:47 UTC
Fix is in openshift-ansible-3.9.24-1.git.0.d0289ea.el7

Per comment 20, moving to verified.

Comment 25 errata-xmlrpc 2018-05-17 06:43:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566