Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1598169

Summary:

Failed to query Docker API while running deploy_cluster.yml, even though docker service is running in the nodes

Product:

OpenShift Container Platform

Reporter:

Ashmitha Ambastha <asambast>

Component:

Installer

Assignee:

Scott Dodson <sdodson>

Status:

CLOSED INSUFFICIENT_DATA

QA Contact:

Johnny Liu <jialiu>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

3.10.0

CC:

aos-bugs, asambast, bsivasub, jokerman, maszulik, mgugino, mmccomas, sandmc12, sdodson

Target Milestone:

---

Keywords:

Reopened, Unconfirmed

Target Release:

3.10.z

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2018-11-29 17:03:31 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Inventory file used for this deployment	none

Description Ashmitha Ambastha 2018-07-04 14:12:43 UTC

Created attachment 1456514 [details]
Inventory file used for this deployment

Description of problem:
While running deploy_cluster.yml to deploy OCP (v3.10.10.-1)+CNS+Logging+Metrics, the playbook fails at the Health Check play with the following error. 


---------snippet of error faced while running deploy_cluster.yml--------------

CHECK [memory_availability : dhcp47-62.lab.eng.blr.redhat.com] **************************************************************************************************************
fatal: [dhcp47-62.lab.eng.blr.redhat.com]: FAILED! => {"changed": false, "checks": {"disk_availability": {}, "docker_image_availability": {"skipped": true, "skipped_reason": "Disabled by user request"}, "docker_storage": {"failed": true, "failures": [["OpenShiftCheckException", "Failed to query Docker API. Is docker running on this host?"]], "msg": "Failed to query Docker API. Is docker running on this host?"}, "memory_availability": {}, "package_availability": {"changed": false, "invocation": {"module_args": {"packages": ["PyYAML", "atomic-openshift", "atomic-openshift-node", "atomic-openshift-sdn-ovs", "bind", "ceph-common", "dnsmasq", "docker", "firewalld", "flannel", "glusterfs-fuse", "iptables", "iptables-services", "iscsi-initiator-utils", "libselinux-python", "nfs-utils", "ntp", "openssl", "pyparted", "python-httplib2", "yum-utils"]}}}, "package_version": {"changed": false, "invocation": {"module_args": {"package_list": [{"check_multi": true, "name": "atomic-openshift", "version": "3.10"}, {"check_multi": true, "name": "atomic-openshift-master", "version": "3.10"}, {"check_multi": true, "name": "atomic-openshift-node", "version": "3.10"}], "package_mgr": "yum"}}}}, "failed": true, "msg": "One or more checks failed", "playbook_context": "install"}

CHECK [docker_image_availability : dhcp46-172.lab.eng.blr.redhat.com] *******************************************************************************************************

CHECK [memory_availability : dhcp46-172.lab.eng.blr.redhat.com] *************************************************************************************************************
fatal: [dhcp46-172.lab.eng.blr.redhat.com]: FAILED! => {"changed": false, "checks": {"disk_availability": {}, "docker_image_availability": {"skipped": true, "skipped_reason": "Disabled by user request"}, "docker_storage": {"failed": true, "failures": [["OpenShiftCheckException", "Failed to query Docker API. Is docker running on this host?"]], "msg": "Failed to query Docker API. Is docker running on this host?"}, "memory_availability": {}, "package_availability": {"changed": false, "invocation": {"module_args": {"packages": ["PyYAML", "atomic-openshift", "atomic-openshift-node", "atomic-openshift-sdn-ovs", "bind", "ceph-common", "dnsmasq", "docker", "firewalld", "flannel", "glusterfs-fuse", "iptables", "iptables-services", "iscsi-initiator-utils", "libselinux-python", "nfs-utils", "ntp", "openssl", "pyparted", "python-httplib2", "yum-utils"]}}}, "package_version": {"changed": false, "invocation": {"module_args": {"package_list": [{"check_multi": true, "name": "atomic-openshift", "version": "3.10"}, {"check_multi": true, "name": "atomic-openshift-master", "version": "3.10"}, {"check_multi": true, "name": "atomic-openshift-node", "version": "3.10"}], "package_mgr": "yum"}}}}, "failed": true, "msg": "One or more checks failed", "playbook_context": "install"}

CHECK [disk_availability : dhcp46-17.lab.eng.blr.redhat.com] ****************************************************************************************************************

CHECK [package_availability : dhcp46-17.lab.eng.blr.redhat.com] *************************************************************************************************************

CHECK [package_version : dhcp46-17.lab.eng.blr.redhat.com] ******************************************************************************************************************

CHECK [docker_image_availability : dhcp46-17.lab.eng.blr.redhat.com] ********************************************************************************************************

CHECK [memory_availability : dhcp46-17.lab.eng.blr.redhat.com] **************************************************************************************************************
changed: [dhcp46-17.lab.eng.blr.redhat.com]

NO MORE HOSTS LEFT **********************************************************************************************************************************************************
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry

PLAY RECAP ******************************************************************************************************************************************************************
dhcp46-17.lab.eng.blr.redhat.com : ok=26   changed=1    unreachable=0    failed=0   
dhcp46-172.lab.eng.blr.redhat.com : ok=25   changed=0    unreachable=0    failed=1   
dhcp46-177.lab.eng.blr.redhat.com : ok=26   changed=1    unreachable=0    failed=0   
dhcp46-30.lab.eng.blr.redhat.com : ok=56   changed=1    unreachable=0    failed=0   
dhcp47-62.lab.eng.blr.redhat.com : ok=25   changed=0    unreachable=0    failed=1   
localhost                  : ok=14   changed=0    unreachable=0    failed=0   


INSTALLER STATUS ************************************************************************************************************************************************************
Initialization  : Complete (0:00:44)
Health Check    : In Progress (0:03:23)
	This phase can be restarted by running: playbooks/openshift-checks/pre-install.yml


Failure summary:


  1. Hosts:    dhcp46-172.lab.eng.blr.redhat.com, dhcp47-62.lab.eng.blr.redhat.com
     Play:     OpenShift Health Checks
     Task:     Run health checks (install) - EL
     Message:  One or more checks failed
     Details:  check "docker_storage":
               Failed to query Docker API. Is docker running on this host?

The execution of "/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml" includes checks designed to fail early if the requirements of the playbook are not met. One or more of these checks failed. To disregard these results,explicitly disable checks by setting an Ansible variable:
   openshift_disable_check=docker_storage
Failing check names are shown in the failure details above. Some checks may be configurable by variables if your requirements are different from the defaults; consult check documentation.
Variables can be set in the inventory or passed on the command line using the -e flag to ansible-playbook.

------------------- end of snippet----------------------------

------------------- Snippet - status of docker service from the nodes where it gives the false failure-----------------

[root@dhcp47-62 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─custom.conf
   Active: active (running) since Wed 2018-07-04 11:11:26 IST; 5h 31min ago
     Docs: http://docs.docker.com
 Main PID: 8263 (dockerd-current)
   CGroup: /system.slice/docker.service
           ├─ 8263 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/...
           └─13485 /usr/bin/docker-containerd-current -l unix:///var/run/docker/lib...

Jul 04 16:40:18 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Jul 04 16:40:19 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Jul 04 16:40:19 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Jul 04 16:40:19 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Jul 04 16:40:19 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Jul 04 16:40:20 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Jul 04 16:40:20 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Jul 04 16:41:07 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Jul 04 16:42:40 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Jul 04 16:42:40 dhcp47-62.lab.eng.blr.redhat.com dockerd-current[8263]: time="2018-...
Hint: Some lines were ellipsized, use -l to show in full.


[root@dhcp46-172 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─custom.conf
   Active: active (running) since Wed 2018-07-04 11:11:25 IST; 5h 31min ago
     Docs: http://docs.docker.com

------------------end of snippet----------------------

Version-Release number of selected component (if applicable): v3.10.10-1

How reproducible: twice

Steps to Reproduce:
1. Run prerequisite.yml with the inventory file attached
2. Once prerequisite.yml passes, run the deploy_cluster.yml playbook with the same inventory file.

Actual results: The playbook failed at the Health Check play. 

Expected results: The playbook should not fail at this play. As docker service is running on the nodes. 


Additional info: As the error message mentions to add openshift_disable_check=docker_storage to the inventory file as a work around.

Comment 1 Maciej Szulik 2018-07-05 09:35:19 UTC

Look like this is installer problem, moving accordingly.

Comment 2 Scott Dodson 2018-07-05 12:09:47 UTC

Is this reproducable? If you run deploy_cluster.yml again after verifying that docker is running does it happen again? If you run `docker info` what do you get?

Comment 3 Ashmitha Ambastha 2018-07-11 05:21:44 UTC

Hi Scott, 

I haven't seen this issue multiple times. I saw it twice that very day and used the work around mentioned. I'll try to reproduce the issue today & update the bug with the details.

Comment 4 Ashmitha Ambastha 2018-07-12 09:26:12 UTC

Hi Scott, 

When we run deploy_cluster.yml again after verifying that docker service is running, we still faced this issue. I raised the bug after the second try. So, I'm sure of this. I had not checked the results of running `docker info`.
I'll update with the details if I face this issue again.

Comment 5 Ashmitha Ambastha 2018-07-12 10:08:43 UTC

Scott, 

You can close this bug. I didn't realize you had closed this already. I'll raise another one if I face this issue again.

Comment 6 Buvanesh Kumar 2018-11-06 06:07:09 UTC

Scott,

I can be able to reproduce this error.

[root@docker-master openshift-ansible]# systemctl status docker 
\u25cf docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           \u2514\u2500custom.conf
   Active: active (running) since Tue 2018-11-06 00:29:55 EST; 33min ago
     Docs: http://docs.docker.com
 Main PID: 18235 (dockerd-current)
    Tasks: 24
   CGroup: /system.slice/docker.service
           \u251c\u250018235 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/li...
           \u2514\u250018242 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/container...

Nov 06 00:29:54 docker-master dockerd-current[18235]: time="2018-11-06T00:29:54.214496750-05:00" level=info msg="34ee61453761b418 became leader at term 6"
Nov 06 00:29:54 docker-master dockerd-current[18235]: time="2018-11-06T00:29:54.214507452-05:00" level=info msg="raft.node: 34ee61453761b418 elected leader 34ee61453761b418 at term 6"
Nov 06 00:29:55 docker-master dockerd-current[18235]: time="2018-11-06T00:29:55.287819771-05:00" level=info msg="Initializing Libnetwork Agent Listen-Addr=0.0.0.0 Local-addr=10.76.98.134 Adv-addr...emote-addr ="
Nov 06 00:29:55 docker-master dockerd-current[18235]: time="2018-11-06T00:29:55.307720077-05:00" level=info msg="Daemon has completed initialization"
Nov 06 00:29:55 docker-master dockerd-current[18235]: time="2018-11-06T00:29:55.307751510-05:00" level=info msg="Docker daemon" commit="8633870/1.13.1" graphdriver=overlay2 version=1.13.1
Nov 06 00:29:55 docker-master dockerd-current[18235]: time="2018-11-06T00:29:55.308470373-05:00" level=info msg="Initializing Libnetwork Agent Listen-Addr=0.0.0.0 Local-addr=10.76.98.134 Adv-addr...emote-addr ="
Nov 06 00:29:55 docker-master dockerd-current[18235]: time="2018-11-06T00:29:55.308529028-05:00" level=info msg="Gossip cluster hostname docker-master-172a5ded2d9c"
Nov 06 00:29:55 docker-master dockerd-current[18235]: time="2018-11-06T00:29:55.311675774-05:00" level=info msg="API listen on /var/run/docker.sock"
Nov 06 00:29:55 docker-master systemd[1]: Started Docker Application Container Engine.
Nov 06 00:29:55 docker-master dockerd-current[18235]: time="2018-11-06T00:29:55-05:00" level=info msg="Firewalld running: false"


[root@docker-master openshift-ansible]# docker info
Containers: 6
 Running: 0
 Paused: 0
 Stopped: 6
Images: 15
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: tgocqakvmn2t3ridxv4nmovg2
 Is Manager: true
 ClusterID: 1fohnsqjbeqxszqpndix8ullj
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.76.98.134
 Manager Addresses:
  10.76.98.134:2377
Runtimes: runc docker-runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 5eda6f6fd0c2884c2c8e78a6e7119e8d0ecedb77 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-693.2.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 4
Total Memory: 7.454 GiB
Name: docker-master
ID: WYGB:VC74:X3UM:JWS5:NESB:VYDL:MEX7:D7NI:TCJP:OAX3:W5ZS:WA45
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

=====================================================================

[root@docker-master openshift-ansible]# ansible-playbook -i inventory/hosts.localhost playbooks/deploy_cluster.yml

PLAY [Initialization Checkpoint Start] ****************************************************************************************************************************************************************************

TASK [Set install initialization 'In Progress'] *******************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:23 -0500 (0:00:00.073)       0:00:00.073 ****** 
ok: [localhost]

PLAY [Populate config host groups] ********************************************************************************************************************************************************************************

TASK [Load group name mapping variables] **************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.258)       0:00:00.331 ****** 
ok: [localhost]

TASK [Evaluate groups - g_nfs_hosts is single host] ***************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.046)       0:00:00.378 ****** 
skipping: [localhost]

TASK [Evaluate oo_all_hosts] **************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.054)       0:00:00.432 ****** 
ok: [localhost] => (item=localhost)

TASK [Evaluate oo_masters] ****************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.079)       0:00:00.511 ****** 
ok: [localhost] => (item=localhost)

TASK [Evaluate oo_first_master] ***********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.053)       0:00:00.565 ****** 
ok: [localhost]

TASK [Evaluate oo_new_etcd_to_config] *****************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.039)       0:00:00.605 ****** 

TASK [Evaluate oo_masters_to_config] ******************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.029)       0:00:00.634 ****** 
ok: [localhost] => (item=localhost)

TASK [Evaluate oo_etcd_to_config] *********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.056)       0:00:00.691 ****** 
ok: [localhost] => (item=localhost)

TASK [Evaluate oo_first_etcd] *************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.079)       0:00:00.770 ****** 
ok: [localhost]

TASK [Evaluate oo_etcd_hosts_to_upgrade] **************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.063)       0:00:00.834 ****** 
ok: [localhost] => (item=localhost)

TASK [Evaluate oo_etcd_hosts_to_backup] ***************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.097)       0:00:00.932 ****** 
ok: [localhost] => (item=localhost)

TASK [Evaluate oo_nodes_to_config] ********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.074)       0:00:01.006 ****** 
ok: [localhost] => (item=localhost)

TASK [Evaluate oo_lb_to_config] ***********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.147)       0:00:01.153 ****** 

TASK [Evaluate oo_nfs_to_config] **********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:24 -0500 (0:00:00.061)       0:00:01.214 ****** 

TASK [Evaluate oo_glusterfs_to_config] ****************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:25 -0500 (0:00:00.114)       0:00:01.329 ****** 

TASK [Evaluate oo_etcd_to_migrate] ********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:25 -0500 (0:00:00.077)       0:00:01.407 ****** 
ok: [localhost] => (item=localhost)
 [WARNING]: Could not match supplied host pattern, ignoring: oo_lb_to_config

 [WARNING]: Could not match supplied host pattern, ignoring: oo_nfs_to_config


PLAY [Ensure that all non-node hosts are accessible] **************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:25 -0500 (0:00:00.214)       0:00:01.622 ****** 
ok: [localhost]

PLAY [Initialize basic host facts] ********************************************************************************************************************************************************************************

TASK [openshift_sanitize_inventory : include_tasks] ***************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:26 -0500 (0:00:00.912)       0:00:02.534 ****** 
included: /root/openshift-ansible/roles/openshift_sanitize_inventory/tasks/deprecations.yml for localhost

TASK [openshift_sanitize_inventory : Check for usage of deprecated variables] *************************************************************************************************************************************
Tuesday 06 November 2018  01:03:26 -0500 (0:00:00.165)       0:00:02.700 ****** 
ok: [localhost]

TASK [openshift_sanitize_inventory : debug] ***********************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:26 -0500 (0:00:00.164)       0:00:02.864 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : set_stats] *******************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:26 -0500 (0:00:00.060)       0:00:02.925 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : set_fact] ********************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:26 -0500 (0:00:00.097)       0:00:03.022 ****** 
ok: [localhost]

TASK [openshift_sanitize_inventory : Standardize on latest variable names] ****************************************************************************************************************************************
Tuesday 06 November 2018  01:03:26 -0500 (0:00:00.116)       0:00:03.140 ****** 
ok: [localhost]

TASK [openshift_sanitize_inventory : Normalize openshift_release] *************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:26 -0500 (0:00:00.076)       0:00:03.216 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Abort when openshift_release is invalid] *************************************************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.077)       0:00:03.293 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : include_tasks] ***************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.071)       0:00:03.365 ****** 
included: /root/openshift-ansible/roles/openshift_sanitize_inventory/tasks/unsupported.yml for localhost

TASK [openshift_sanitize_inventory : set_fact] ********************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.112)       0:00:03.478 ****** 

TASK [openshift_sanitize_inventory : Ensure that dynamic provisioning is set if using dynamic storage] ************************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.097)       0:00:03.575 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Ensure the hosted registry's GlusterFS storage is configured correctly] ******************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.072)       0:00:03.648 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Ensure the hosted registry's GlusterFS storage is configured correctly] ******************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.073)       0:00:03.722 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Check for deprecated prometheus/grafana install] *****************************************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.086)       0:00:03.808 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Ensure clusterid is set along with the cloudprovider] ************************************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.064)       0:00:03.873 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Ensure ansible_service_broker_remove and ansible_service_broker_install are mutually exclusive] ******************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.064)       0:00:03.937 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Ensure template_service_broker_remove and template_service_broker_install are mutually exclusive] ****************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.068)       0:00:04.006 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Ensure that all requires vsphere configuration variables are set] ************************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.075)       0:00:04.082 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : ensure provider configuration variables are defined] *************************************************************************************************************************
Tuesday 06 November 2018  01:03:27 -0500 (0:00:00.075)       0:00:04.157 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Ensure removed web console extension variables are not set] ******************************************************************************************************************
Tuesday 06 November 2018  01:03:28 -0500 (0:00:00.078)       0:00:04.235 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : Ensure that web console port matches API server port] ************************************************************************************************************************
Tuesday 06 November 2018  01:03:28 -0500 (0:00:00.085)       0:00:04.321 ****** 
skipping: [localhost]

TASK [openshift_sanitize_inventory : At least one master is schedulable] ******************************************************************************************************************************************
Tuesday 06 November 2018  01:03:28 -0500 (0:00:00.109)       0:00:04.430 ****** 
skipping: [localhost]

TASK [Detect OS Variant from /etc/os-release] *********************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:28 -0500 (0:00:00.127)       0:00:04.558 ****** 
skipping: [localhost]

TASK [set openshift_deployment_type if unset] *********************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:28 -0500 (0:00:00.104)       0:00:04.663 ****** 
skipping: [localhost]

PLAY [Retrieve existing master configs and validate] **************************************************************************************************************************************************************

TASK [openshift_control_plane : stat] *****************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:28 -0500 (0:00:00.072)       0:00:04.735 ****** 
ok: [localhost]

TASK [openshift_control_plane : slurp] ****************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:28 -0500 (0:00:00.332)       0:00:05.067 ****** 
skipping: [localhost]

TASK [openshift_control_plane : set_fact] *************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:28 -0500 (0:00:00.096)       0:00:05.164 ****** 
skipping: [localhost]

TASK [openshift_control_plane : Check for file paths outside of /etc/origin/master in master's config] ************************************************************************************************************
Tuesday 06 November 2018  01:03:28 -0500 (0:00:00.066)       0:00:05.242 ****** 
skipping: [localhost]

TASK [openshift_control_plane : set_fact] *************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.083)       0:00:05.325 ****** 
skipping: [localhost]

TASK [set_fact] ***************************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.049)       0:00:05.374 ****** 
skipping: [localhost]

TASK [set_fact] ***************************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.073)       0:00:05.448 ****** 
skipping: [localhost]

PLAY [Initialize special first-master variables] ******************************************************************************************************************************************************************

TASK [set_fact] ***************************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.063)       0:00:05.511 ****** 
skipping: [localhost]

TASK [set_fact] ***************************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.073)       0:00:05.585 ****** 
ok: [localhost]

PLAY [Disable web console if required] ****************************************************************************************************************************************************************************

TASK [set_fact] ***************************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.089)       0:00:05.674 ****** 
skipping: [localhost]

PLAY [Setup yum repositories for all hosts] ***********************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Install packages necessary for installer] *******************************************************************************************************************************************************************

TASK [Determine if chrony is installed] ***************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.101)       0:00:05.775 ****** 
skipping: [localhost]

TASK [Install ntp package] ****************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.073)       0:00:05.849 ****** 
skipping: [localhost]

TASK [Start and enable ntpd/chronyd] ******************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.064)       0:00:05.913 ****** 
skipping: [localhost]

TASK [Ensure openshift-ansible installer package deps are installed] **********************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.074)       0:00:05.988 ****** 
skipping: [localhost]

PLAY [Initialize cluster facts] ***********************************************************************************************************************************************************************************

TASK [get openshift_current_version] ******************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:29 -0500 (0:00:00.102)       0:00:06.090 ****** 
ok: [localhost]

TASK [set_fact openshift_portal_net if present on masters] ********************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:30 -0500 (0:00:00.324)       0:00:06.415 ****** 
ok: [localhost]

TASK [Gather Cluster facts] ***************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:30 -0500 (0:00:00.166)       0:00:06.581 ****** 
ok: [localhost]

TASK [Set fact of no_proxy_internal_hostnames] ********************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:31 -0500 (0:00:01.127)       0:00:07.709 ****** 
skipping: [localhost]

TASK [Initialize openshift.node.sdn_mtu] **************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:31 -0500 (0:00:00.063)       0:00:07.772 ****** 
ok: [localhost]

TASK [set_fact l_kubelet_node_name] *******************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:32 -0500 (0:00:00.738)       0:00:08.511 ****** 
ok: [localhost]

PLAY [Initialize etcd host variables] *****************************************************************************************************************************************************************************

TASK [set_fact] ***************************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:32 -0500 (0:00:00.088)       0:00:08.600 ****** 
ok: [localhost]

TASK [set_fact] ***************************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:32 -0500 (0:00:00.153)       0:00:08.754 ****** 
ok: [localhost]

PLAY [Determine openshift_version to configure on first master] ***************************************************************************************************************************************************

TASK [include_role : openshift_version] ***************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:32 -0500 (0:00:00.122)       0:00:08.876 ****** 

TASK [openshift_version : Use openshift_current_version fact as version to configure if already installed] ********************************************************************************************************
Tuesday 06 November 2018  01:03:32 -0500 (0:00:00.126)       0:00:09.002 ****** 
skipping: [localhost]

TASK [openshift_version : Set openshift_version to openshift_release if undefined] ********************************************************************************************************************************
Tuesday 06 November 2018  01:03:32 -0500 (0:00:00.073)       0:00:09.076 ****** 
ok: [localhost]

TASK [openshift_version : debug] **********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:32 -0500 (0:00:00.102)       0:00:09.178 ****** 
ok: [localhost] => {
    "msg": "openshift_pkg_version was not defined. Falling back to -4.0"
}

TASK [openshift_version : set_fact] *******************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.090)       0:00:09.268 ****** 
ok: [localhost]

TASK [openshift_version : debug] **********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.078)       0:00:09.347 ****** 
ok: [localhost] => {
    "msg": "openshift_image_tag was not defined. Falling back to v4.0"
}

TASK [openshift_version : set_fact] *******************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.060)       0:00:09.408 ****** 
ok: [localhost]

TASK [openshift_version : assert openshift_release in openshift_image_tag] ****************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.076)       0:00:09.484 ****** 
ok: [localhost] => {
    "changed": false, 
    "msg": "All assertions passed"
}

TASK [openshift_version : assert openshift_release in openshift_pkg_version] **************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.093)       0:00:09.577 ****** 
ok: [localhost] => {
    "changed": false, 
    "msg": "All assertions passed"
}

TASK [openshift_version : debug] **********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.091)       0:00:09.668 ****** 
ok: [localhost] => {
    "openshift_release": "4.0"
}

TASK [openshift_version : debug] **********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.075)       0:00:09.744 ****** 
ok: [localhost] => {
    "openshift_image_tag": "v4.0"
}

TASK [openshift_version : debug] **********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.097)       0:00:09.842 ****** 
ok: [localhost] => {
    "openshift_pkg_version": "-4.0*"
}

TASK [openshift_version : debug] **********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.134)       0:00:09.977 ****** 
ok: [localhost] => {
    "openshift_version": "4.0"
}

PLAY [Set openshift_version for etcd, node, and master hosts] *****************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Verify Requirements] ****************************************************************************************************************************************************************************************

TASK [Run variable sanity checks] *********************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:33 -0500 (0:00:00.206)       0:00:10.184 ****** 
ok: [localhost]

TASK [Validate openshift_node_groups and openshift_node_group_name] ***********************************************************************************************************************************************
Tuesday 06 November 2018  01:03:36 -0500 (0:00:02.836)       0:00:13.020 ****** 
ok: [localhost]

PLAY [Verify Node NetworkManager] *********************************************************************************************************************************************************************************

TASK [Check for NetworkManager service] ***************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:36 -0500 (0:00:00.121)       0:00:13.142 ****** 
ok: [localhost]

TASK [fail] *******************************************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:37 -0500 (0:00:00.388)       0:00:13.530 ****** 
skipping: [localhost]

PLAY [Initialization Checkpoint End] ******************************************************************************************************************************************************************************

TASK [Set install initialization 'Complete'] **********************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:37 -0500 (0:00:00.088)       0:00:13.619 ****** 
ok: [localhost]

PLAY [Health Check Checkpoint Start] ******************************************************************************************************************************************************************************

TASK [Set Health Check 'In Progress'] *****************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:37 -0500 (0:00:00.144)       0:00:13.764 ****** 
ok: [localhost]

PLAY [OpenShift Health Checks] ************************************************************************************************************************************************************************************

TASK [Run health checks (install) - EL] ***************************************************************************************************************************************************************************
Tuesday 06 November 2018  01:03:37 -0500 (0:00:00.100)       0:00:13.864 ****** 

CHECK [docker_storage : localhost] ********************************************************************************************************************************************************************************

CHECK [disk_availability : localhost] *****************************************************************************************************************************************************************************

CHECK [package_availability : localhost] **************************************************************************************************************************************************************************

CHECK [package_version : localhost] *******************************************************************************************************************************************************************************

CHECK [docker_image_availability : localhost] *********************************************************************************************************************************************************************

CHECK [memory_availability : localhost] ***************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "checks": {"disk_availability": {"skipped": true, "skipped_reason": "Disabled by user request"}, "docker_image_availability": {"changed": true, "failed": true, "failures": [["OpenShiftCheckException", "One or more required container images are not available:\n    quay.io/coreos/etcd:v3.2.22\nChecked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>\n"]], "msg": "One or more required container images are not available:\n    quay.io/coreos/etcd:v3.2.22\nChecked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>\n"}, "docker_storage": {"failed": true, "failures": [["OpenShiftCheckException", "Failed to query Docker API. Is docker running on this host?"]], "msg": "Failed to query Docker API. Is docker running on this host?"}, "memory_availability": {"skipped": true, "skipped_reason": "Disabled by user request"}, "package_availability": {"changed": false, "invocation": {"module_args": {"packages": ["PyYAML", "bash-completion", "bind", "ceph-common", "dnsmasq", "docker", "firewalld", "flannel", "glusterfs-fuse", "httpd-tools", "iptables", "iptables-services", "iscsi-initiator-utils", "libselinux-python", "nfs-utils", "ntp", "openssl", "origin", "origin-clients", "origin-hyperkube", "origin-node", "pyparted", "python-httplib2", "yum-utils"]}}}, "package_version": {"changed": false, "invocation": {"module_args": {"package_list": [{"check_multi": false, "name": "origin", "version": ""}, {"check_multi": false, "name": "origin-master", "version": ""}, {"check_multi": false, "name": "origin-node", "version": ""}], "package_mgr": "yum"}}}}, "msg": "One or more checks failed", "playbook_context": "install"}

NO MORE HOSTS LEFT ************************************************************************************************************************************************************************************************

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost                  : ok=43   changed=0    unreachable=0    failed=1   


INSTALLER STATUS **************************************************************************************************************************************************************************************************
Initialization  : Complete (0:00:14)
Health Check    : In Progress (0:02:06)
	This phase can be restarted by running: playbooks/openshift-checks/pre-install.yml
Tuesday 06 November 2018  01:05:43 -0500 (0:02:06.263)       0:02:20.128 ****** 
=============================================================================== 
Run health checks (install) - EL ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 126.26s
Run variable sanity checks --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.84s
Gather Cluster facts --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.13s
Gathering Facts -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.91s
Initialize openshift.node.sdn_mtu -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.74s
Check for NetworkManager service --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.39s
openshift_control_plane : stat ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.33s
get openshift_current_version ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.32s
Set install initialization 'In Progress' ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.26s
Evaluate oo_etcd_to_migrate -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.21s
openshift_version : debug ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.21s
set_fact openshift_portal_net if present on masters -------------------------------------------------------------------------------------------------------------------------------------------------------- 0.17s
openshift_sanitize_inventory : include_tasks --------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.17s
openshift_sanitize_inventory : Check for usage of deprecated variables ------------------------------------------------------------------------------------------------------------------------------------- 0.16s
set_fact --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.15s
Evaluate oo_nodes_to_config -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.15s
Set install initialization 'Complete' ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.14s
openshift_version : debug ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.14s
openshift_sanitize_inventory : At least one master is schedulable ------------------------------------------------------------------------------------------------------------------------------------------ 0.13s
include_role : openshift_version --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.13s


Failure summary:


  1. Hosts:    localhost
     Play:     OpenShift Health Checks
     Task:     Run health checks (install) - EL
     Message:  One or more checks failed
     Details:  check "docker_image_availability":
               One or more required container images are not available:
                   quay.io/coreos/etcd:v3.2.22
               Checked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>
               
               
               check "docker_storage":
               Failed to query Docker API. Is docker running on this host?

The execution of "playbooks/deploy_cluster.yml" includes checks designed to fail early if the requirements of the playbook are not met. One or more of these checks failed. To disregard these results,explicitly disable checks by setting an Ansible variable:
   openshift_disable_check=docker_image_availability,docker_storage
Failing check names are shown in the failure details above. Some checks may be configurable by variables if your requirements are different from the defaults; consult check documentation.
Variables can be set in the inventory or passed on the command line using the -e flag to ansible-playbook.

===================================================

Comment 7 Michael Gugino 2018-11-29 17:03:31 UTC

Need -vvv log outputs and inventory.

I don't think this is a bug, this is some kind of setup problem.

Comment 8 Sandeep MC 2019-06-17 08:52:54 UTC

We are installing a Openshift cluster with three node master + 2 node cluster. We have ran the pre-reqs yml and also configured xfs partition and mounted it to /var/lib/docker and ran the deploy cluster yml, but it fails in the below section.

TASK [openshift_control_plane : Check status of control plane image pre-pull] *****************************************************************
changed: [dcaw-oc-master.ibmresiliencyservices.com]

TASK [openshift_control_plane : Check status of etcd image pre-pull] **************************************************************************
changed: [dcaw-oc-master.ibmresiliencyservices.com]

TASK [openshift_control_plane : Start and enable self-hosting node] ***************************************************************************
changed: [dcaw-oc-master.ibmresiliencyservices.com]

TASK [openshift_control_plane : Get node logs] ************************************************************************************************
skipping: [dcaw-oc-master.ibmresiliencyservices.com]

TASK [openshift_control_plane : debug] ********************************************************************************************************
skipping: [dcaw-oc-master.ibmresiliencyservices.com]

TASK [openshift_control_plane : fail] *********************************************************************************************************
skipping: [dcaw-oc-master.ibmresiliencyservices.com]

TASK [openshift_control_plane : Wait for control plane pods to appear] ************************************************************************
FAILED - RETRYING: Wait for control plane pods to appear (60 retries left).
FAILED - RETRYING: Wait for control plane pods to appear (59 retries left).
FAILED - RETRYING: Wait for control plane pods to appear (58 retries left).
FAILED - RETRYING: Wait for control plane pods to appear (57 retries left).
FAILED - RETRYING: Wait for control plane pods to appear (56 retries left).
FAILED - RETRYING: Wait for control plane pods to appear (55 retries left).
^C [ERROR]: User interrupted execution

------------------

Also we have got another error as below

1. Hosts: dcaw-oc-worker1.ibmresiliencyservices.com, dcaw-oc-worker2.ibmresiliencyservices.com
Play: OpenShift Health Checks
Task: Run health checks (install) - EL
Message: One or more checks failed
Details: check "docker_storage":
Docker storage drivers 'overlay' and 'overlay2' are only supported with
'xfs' as the backing storage, but this host's storage is type 'extfs'.

2. Hosts: dcaw-oc-master.ibmresiliencyservices.com
Play: OpenShift Health Checks
Task: Run health checks (install) - EL
Message: One or more checks failed
Details: check "docker_storage":
Failed to query Docker API. Is docker running on this host?

The execution of "/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml" includes checks designed to fail early if the requirements of the playbook are not met. One or more of these checks failed. To disregard these results,explicitly disable checks by setting an Ansible variable:
openshift_disable_check=docker_storage
Failing check names are shown in the failure details above. Some checks may be configurable by variables if your requirements are different from the defaults; consult check documentation.
Variables can be set in the inventory or passed on the command line using the -e flag to ansible-playbook.
----------

Please assist, as we are not sue how to go about the next steps.