Description of problem: When running the deploy-cluster.yml playbook, it fails during the execution of pre-install.yml at "Health Check" task. Version-Release number of the following components: rpm -q openshift-ansible openshift-ansible-3.9.40-1.git.0.188c954.el7.noarch rpm -q Ansible ansible-2.4.6.0-1.el7ae.noarch ansible --Version ansible 2.4.6.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, May 31 2018, 09:41:32) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] How reproducible: Define Hosts in Ansible inventory file and run the deploy-cluster.yml playbook. Actual results: TASK [Run health checks (install) - EL] ************************************************************************************************************************************************* An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'list' object has no attribute 'replace' fatal: [hostname1]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'list' object has no attribute 'replace' fatal: [hostname2]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'list' object has no attribute 'replace' fatal: [hostname3]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'list' object has no attribute 'replace' fatal: [hostname4]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'list' object has no attribute 'replace' fatal: [hostname5]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'list' object has no attribute 'replace' fatal: [hostname6]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'list' object has no attribute 'replace' fatal: [hostname7]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""} An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'list' object has no attribute 'replace' fatal: [hostname8]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""} NO MORE HOSTS LEFT ********************************************************************************************************************************************************************** to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry PLAY RECAP ****************************************************************************************************************************************************************************** hostname1 : ok=43 changed=0 unreachable=0 failed=1 hostname2 : ok=32 changed=0 unreachable=0 failed=1 hostname3 : ok=32 changed=0 unreachable=0 failed=1 hostname4 : ok=28 changed=0 unreachable=0 failed=1 hostname5 : ok=28 changed=0 unreachable=0 failed=1 hostname6 : ok=19 changed=0 unreachable=0 failed=1 hostname7 : ok=28 changed=0 unreachable=0 failed=1 hostname8 : ok=28 changed=0 unreachable=0 failed=1 localhost : ok=13 changed=0 unreachable=0 failed=0 INSTALLER STATUS ************************************************************************************************************************************************************************ Initialization : Complete (0:01:48) Health Check : In Progress (0:00:08) This phase can be restarted by running: playbooks/openshift-checks/pre-install.yml Failure summary: 1. Hosts: hostname1, hostname2, hostname3, hostname4, hostname5, hostname6, hostname7, hostname8 Play: OpenShift Health Checks Task: Run health checks (install) - EL Message: Unexpected failure during module execution. Expected results: Health checks pass. Additional info: Please attach logs from ansible-playbook with the -vvv flag
Is there something unique about your configuration? Can you provide an inventory file even if redacted?
This is /etc/ansible/Hosts: #openshift file [OSEv3:children] masters etcd lb glusterfs glusterfs_registry nodes #Set variables common for all OSEv3hosts [OSEv3:vars] ansible_user=root openshift_clock_enabled=true openshift_deployment_type=openshift-enterprise openshift_release=3.9 openshift_http_proxy='"http://user:password@proxy.company.tld:3128"' openshift_https_proxy='"http://user:password@proxy.company.tld:3128"' openshift_no_proxy='localhost','.company.com','.svc','svc' openshift_disable_check=docker_image_availability openshift_metrics_install_metrics=true openshift_metrics_cassandra_storage_type=dynamic openshift_logging_install_logging=true openshift_logging_es_pvc_dynamic=true openshift_master_api_port=443 openshift_master_console_port=443 openshift_master_cluster_method=native openshift_master_default_subdomain=ocpapps.company-internal.com openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}] openshift_master_htpasswd_users={'admin': 'somehash', 'developer': 'somehash'} openshift_hosted_registry_storage_kind=glusterfs [masters] vm0tho.company-internal.com vm1tho.company-internal.com vm2tho.company-internal.com [etcd] vm0tho.company-internal.com vm1tho.company-internal.com vm2tho.company-internal.com [lb] vm6tho.company-internal.com [glusterfs] #vm6tho.company-internal.com glusterfs_devices='["/dev/sdb"]' vm5tho.company-internal.com glusterfs_devices='["/dev/sda"]' vm8tho.company-internal.com glusterfs_devices='["/dev/sda"]' vm9tho.company-internal.com glusterfs_devices='["/dev/sda"]' [glusterfs_registry] #vm6tho.company-internal.com glusterfs_devices='["/dev/sdc"]' vm5tho.company-internal.com glusterfs_devices='["/dev/sdb"]' vm8tho.company-internal.com glusterfs_devices='["/dev/sdb"]' vm9tho.company-internal.com glusterfs_devices='["/dev/sdb"]' [nodes] vm0tho.company-internal.com openshift_node_labels="{'region': 'master'}" vm1tho.company-internal.com openshift_node_labels="{'region': 'master'}" vm2tho.company-internal.com openshift_node_labels="{'region': 'master'}" vm4tho.company-internal.com openshift_node_labels="{'region': 'primary'}" vm8tho.company-internal.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" vm5tho.company-internal.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" vm9tho.company-internal.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" vm8tho.company-internal.com openshift_schedulable=True vm5tho.company-internal.com openshift_schedulable=True vm9tho.company-internal.com openshift_schedulable=True
Here is the full traceback (-vvv). TASK [Run health checks (install) - EL] ************************************************************************************************************************************************* task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-checks/private/install.yml:22 The full traceback is: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 148, in run res = self._execute() File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 551, in _execute result = self._handler.run(task_vars=variables) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 50, in run known_checks = self.load_known_checks(tmp, task_vars, output_dir) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 105, in load_known_checks templar=self._templar File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_image_availability.py", line 90, in __init__ proxies.append(var.upper() + "=" + quote(self.template_var(value))) File "/usr/lib64/python2.7/pipes.py", line 278, in quote return "'" + file.replace("'", "'\"'\"'") + "'" AttributeError: 'list' object has no attribute 'replace' fatal: [vm4tho.company-internal.com]: FAILED! => { "failed": true, "msg": "Unexpected failure during module execution.", "stdout": "" } The full traceback is: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 148, in run res = self._execute() File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 551, in _execute result = self._handler.run(task_vars=variables) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 50, in run known_checks = self.load_known_checks(tmp, task_vars, output_dir) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 105, in load_known_checks templar=self._templar File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_image_availability.py", line 90, in __init__ proxies.append(var.upper() + "=" + quote(self.template_var(value))) File "/usr/lib64/python2.7/pipes.py", line 278, in quote return "'" + file.replace("'", "'\"'\"'") + "'" AttributeError: 'list' object has no attribute 'replace' fatal: [vm1tho.company-internal.com]: FAILED! => { "failed": true, "msg": "Unexpected failure during module execution.", "stdout": "" } The full traceback is: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 148, in run res = self._execute() File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 551, in _execute result = self._handler.run(task_vars=variables) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 50, in run known_checks = self.load_known_checks(tmp, task_vars, output_dir) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 105, in load_known_checks templar=self._templar File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_image_availability.py", line 90, in __init__ proxies.append(var.upper() + "=" + quote(self.template_var(value))) File "/usr/lib64/python2.7/pipes.py", line 278, in quote return "'" + file.replace("'", "'\"'\"'") + "'" AttributeError: 'list' object has no attribute 'replace' fatal: [vm8tho.company-internal.com]: FAILED! => { "failed": true, "msg": "Unexpected failure during module execution.", "stdout": "" } The full traceback is: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 148, in run res = self._execute() File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 551, in _execute result = self._handler.run(task_vars=variables) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 50, in run known_checks = self.load_known_checks(tmp, task_vars, output_dir) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 105, in load_known_checks templar=self._templar File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_image_availability.py", line 90, in __init__ proxies.append(var.upper() + "=" + quote(self.template_var(value))) File "/usr/lib64/python2.7/pipes.py", line 278, in quote return "'" + file.replace("'", "'\"'\"'") + "'" AttributeError: 'list' object has no attribute 'replace' fatal: [vm2tho.company-internal.com]: FAILED! => { "failed": true, "msg": "Unexpected failure during module execution.", "stdout": "" } The full traceback is: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 148, in run res = self._execute() File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 551, in _execute result = self._handler.run(task_vars=variables) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 50, in run known_checks = self.load_known_checks(tmp, task_vars, output_dir) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 105, in load_known_checks templar=self._templar File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_image_availability.py", line 90, in __init__ proxies.append(var.upper() + "=" + quote(self.template_var(value))) File "/usr/lib64/python2.7/pipes.py", line 278, in quote return "'" + file.replace("'", "'\"'\"'") + "'" AttributeError: 'list' object has no attribute 'replace' fatal: [vm0tho.company-internal.com]: FAILED! => { "failed": true, "msg": "Unexpected failure during module execution.", "stdout": "" } The full traceback is: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 148, in run res = self._execute() File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 551, in _execute result = self._handler.run(task_vars=variables) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 50, in run known_checks = self.load_known_checks(tmp, task_vars, output_dir) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 105, in load_known_checks templar=self._templar File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_image_availability.py", line 90, in __init__ proxies.append(var.upper() + "=" + quote(self.template_var(value))) File "/usr/lib64/python2.7/pipes.py", line 278, in quote return "'" + file.replace("'", "'\"'\"'") + "'" AttributeError: 'list' object has no attribute 'replace' fatal: [vm5tho.company-internal.com]: FAILED! => { "failed": true, "msg": "Unexpected failure during module execution.", "stdout": "" } The full traceback is: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 148, in run res = self._execute() File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 551, in _execute result = self._handler.run(task_vars=variables) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 50, in run known_checks = self.load_known_checks(tmp, task_vars, output_dir) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 105, in load_known_checks templar=self._templar File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_image_availability.py", line 90, in __init__ proxies.append(var.upper() + "=" + quote(self.template_var(value))) File "/usr/lib64/python2.7/pipes.py", line 278, in quote return "'" + file.replace("'", "'\"'\"'") + "'" AttributeError: 'list' object has no attribute 'replace' fatal: [vm9tho.company-internal.com]: FAILED! => { "failed": true, "msg": "Unexpected failure during module execution.", "stdout": "" } The full traceback is: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 148, in run res = self._execute() File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 551, in _execute result = self._handler.run(task_vars=variables) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 50, in run known_checks = self.load_known_checks(tmp, task_vars, output_dir) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 105, in load_known_checks templar=self._templar File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_image_availability.py", line 90, in __init__ proxies.append(var.upper() + "=" + quote(self.template_var(value))) File "/usr/lib64/python2.7/pipes.py", line 278, in quote return "'" + file.replace("'", "'\"'\"'") + "'" AttributeError: 'list' object has no attribute 'replace' fatal: [vm6tho.company-internal.com]: FAILED! => { "failed": true, "msg": "Unexpected failure during module execution.", "stdout": "" } NO MORE HOSTS LEFT ********************************************************************************************************************************************************************** to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry PLAY RECAP ****************************************************************************************************************************************************************************** vm0tho.company-internal.com : ok=43 changed=3 unreachable=0 failed=1 vm1tho.company-internal.com : ok=32 changed=3 unreachable=0 failed=1 vm2tho.company-internal.com : ok=32 changed=3 unreachable=0 failed=1 vm4tho.company-internal.com : ok=28 changed=3 unreachable=0 failed=1 vm5tho.company-internal.com : ok=28 changed=3 unreachable=0 failed=1 vm6tho.company-internal.com : ok=19 changed=3 unreachable=0 failed=1 vm8tho.company-internal.com : ok=28 changed=3 unreachable=0 failed=1 vm9tho.company-internal.com : ok=28 changed=3 unreachable=0 failed=1 localhost : ok=13 changed=0 unreachable=0 failed=0 INSTALLER STATUS ************************************************************************************************************************************************************************ Initialization : Complete (0:02:05) Health Check : In Progress (0:00:09) This phase can be restarted by running: playbooks/openshift-checks/pre-install.yml Failure summary: 1. Hosts: vm0tho.company-internal.com, vm1tho.company-internal.com, vm2tho.company-internal.com, vm4tho.company-internal.com, vm5tho.company-internal.com, vm6tho.company-internal.com, vm8tho.company-internal.com, vm9tho.company-internal.com Play: OpenShift Health Checks Task: Run health checks (install) - EL Message: Unexpected failure during module execution.
There appear to be no active cases related to this bug. As such we're closing this bug in order to focus on bugs that are still tied to active customer cases. Please re-open this bug if you feel it was closed in error or a new active case is attached.