Bug 1532684

Created attachment 1379141 [details]
Ansible log of install failure

Description of problem:

Installing 3.9.0-0.16.0 with openshift-ansible/playbooks/deploy_cluster.yml fails in this task with 120 retries:

TASK [openshift_web_console : Verify that the web console is running]


fatal: [ec2-54-190-196-89.us-west-2.compute.amazonaws.com]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "-k", "https://webconsole.openshift-web-console.svc/healthz"], "delta": "0:00:01.013538", "end": "2018-01-09 13:12:56.996536", "msg": "non-zero return code", "rc": 7, "start": "2018-01-09 13:12:55.982998", "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused", "stderr_lines": ["  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", "                                 Dload  Upload   Total   Spent    Left  Speed", "", "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0", "  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused"], "stdout": "", "stdout_lines": []}

Version-Release number of the following components:
rpm -q openshift-ansible

openshift-ansible-3.9.0-0.16.0.git.0.9f19afc.el7.noarch

ansible --version           
ansible 2.4.2.0                    
  config file = /etc/ansible/ansible.cfg                              
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']                                 
  ansible python module location = /usr/lib/python2.7/site-packages/ansible                                                                  
  executable location = /usr/bin/ansible                              
  python version = 2.7.5 (default, Dec  8 2017, 16:39:59) [GCC 4.8.5 20150623 (Red Hat 4.8.5-25)] 

How reproducible: Always

Steps to Reproduce:
1.  Install 3.9.0-0.16.0 with the inventory below (or similar) on AWS


Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

TASK [openshift_web_console : Verify that the web console is running] **********
Tuesday 09 January 2018  13:07:37 +0000 (0:00:00.953)       0:12:07.503 ******* 
FAILED - RETRYING: Verify that the web console is running (120 retries left).
FAILED - RETRYING: Verify that the web console is running (119 retries left).
FAILED - RETRYING: Verify that the web console is running (118 retries left).
FAILED - RETRYING: Verify that the web console is running (117 retries left).
FAILED - RETRYING: Verify that the web console is running (116 retries left).
FAILED - RETRYING: Verify that the web console is running (115 retries left).
FAILED - RETRYING: Verify that the web console is running (114 retries left).
FAILED - RETRYING: Verify that the web console is running (113 retries left).
FAILED - RETRYING: Verify that the web console is running (112 retries left).
FAILED - RETRYING: Verify that the web console is running (111 retries left).
FAILED - RETRYING: Verify that the web console is running (110 retries left).
FAILED - RETRYING: Verify that the web console is running (109 retries left).
FAILED - RETRYING: Verify that the web console is running (108 retries left).
FAILED - RETRYING: Verify that the web console is running (107 retries left).
FAILED - RETRYING: Verify that the web console is running (106 retries left).
FAILED - RETRYING: Verify that the web console is running (105 retries left).
FAILED - RETRYING: Verify that the web console is running (104 retries left).
FAILED - RETRYING: Verify that the web console is running (103 retries left).
FAILED - RETRYING: Verify that the web console is running (102 retries left).
FAILED - RETRYING: Verify that the web console is running (101 retries left).
FAILED - RETRYING: Verify that the web console is running (100 retries left).
FAILED - RETRYING: Verify that the web console is running (99 retries left).
FAILED - RETRYING: Verify that the web console is running (98 retries left).
FAILED - RETRYING: Verify that the web console is running (97 retries left).
FAILED - RETRYING: Verify that the web console is running (96 retries left).
FAILED - RETRYING: Verify that the web console is running (95 retries left).
FAILED - RETRYING: Verify that the web console is running (94 retries left).
FAILED - RETRYING: Verify that the web console is running (93 retries left).
FAILED - RETRYING: Verify that the web console is running (92 retries left).
FAILED - RETRYING: Verify that the web console is running (91 retries left).
FAILED - RETRYING: Verify that the web console is running (90 retries left).
FAILED - RETRYING: Verify that the web console is running (89 retries left).
FAILED - RETRYING: Verify that the web console is running (88 retries left).
FAILED - RETRYING: Verify that the web console is running (87 retries left).
FAILED - RETRYING: Verify that the web console is running (86 retries left).
FAILED - RETRYING: Verify that the web console is running (85 retries left).
FAILED - RETRYING: Verify that the web console is running (84 retries left).
FAILED - RETRYING: Verify that the web console is running (83 retries left).
FAILED - RETRYING: Verify that the web console is running (82 retries left).
FAILED - RETRYING: Verify that the web console is running (81 retries left).
FAILED - RETRYING: Verify that the web console is running (80 retries left).
FAILED - RETRYING: Verify that the web console is running (79 retries left).
FAILED - RETRYING: Verify that the web console is running (78 retries left).
FAILED - RETRYING: Verify that the web console is running (77 retries left).
FAILED - RETRYING: Verify that the web console is running (76 retries left).
FAILED - RETRYING: Verify that the web console is running (75 retries left).
FAILED - RETRYING: Verify that the web console is running (74 retries left).
FAILED - RETRYING: Verify that the web console is running (73 retries left).
FAILED - RETRYING: Verify that the web console is running (72 retries left).
FAILED - RETRYING: Verify that the web console is running (71 retries left).
FAILED - RETRYING: Verify that the web console is running (70 retries left).
FAILED - RETRYING: Verify that the web console is running (69 retries left).
FAILED - RETRYING: Verify that the web console is running (68 retries left).
FAILED - RETRYING: Verify that the web console is running (67 retries left).
FAILED - RETRYING: Verify that the web console is running (66 retries left).
FAILED - RETRYING: Verify that the web console is running (65 retries left).
FAILED - RETRYING: Verify that the web console is running (64 retries left).
FAILED - RETRYING: Verify that the web console is running (63 retries left).
FAILED - RETRYING: Verify that the web console is running (62 retries left).
FAILED - RETRYING: Verify that the web console is running (61 retries left).
FAILED - RETRYING: Verify that the web console is running (60 retries left).
FAILED - RETRYING: Verify that the web console is running (59 retries left).
FAILED - RETRYING: Verify that the web console is running (58 retries left).
FAILED - RETRYING: Verify that the web console is running (57 retries left).
FAILED - RETRYING: Verify that the web console is running (56 retries left).
FAILED - RETRYING: Verify that the web console is running (55 retries left).
FAILED - RETRYING: Verify that the web console is running (54 retries left).
FAILED - RETRYING: Verify that the web console is running (53 retries left).
FAILED - RETRYING: Verify that the web console is running (52 retries left).
FAILED - RETRYING: Verify that the web console is running (51 retries left).
FAILED - RETRYING: Verify that the web console is running (50 retries left).
FAILED - RETRYING: Verify that the web console is running (49 retries left).
FAILED - RETRYING: Verify that the web console is running (48 retries left).
FAILED - RETRYING: Verify that the web console is running (47 retries left).
FAILED - RETRYING: Verify that the web console is running (46 retries left).
FAILED - RETRYING: Verify that the web console is running (45 retries left).
FAILED - RETRYING: Verify that the web console is running (44 retries left).
FAILED - RETRYING: Verify that the web console is running (43 retries left).
FAILED - RETRYING: Verify that the web console is running (42 retries left).
FAILED - RETRYING: Verify that the web console is running (41 retries left).
FAILED - RETRYING: Verify that the web console is running (40 retries left).
FAILED - RETRYING: Verify that the web console is running (39 retries left).
FAILED - RETRYING: Verify that the web console is running (38 retries left).
FAILED - RETRYING: Verify that the web console is running (37 retries left).
FAILED - RETRYING: Verify that the web console is running (36 retries left).
FAILED - RETRYING: Verify that the web console is running (35 retries left).
FAILED - RETRYING: Verify that the web console is running (34 retries left).
FAILED - RETRYING: Verify that the web console is running (33 retries left).
FAILED - RETRYING: Verify that the web console is running (32 retries left).
FAILED - RETRYING: Verify that the web console is running (31 retries left).
FAILED - RETRYING: Verify that the web console is running (30 retries left).
FAILED - RETRYING: Verify that the web console is running (29 retries left).
FAILED - RETRYING: Verify that the web console is running (28 retries left).
FAILED - RETRYING: Verify that the web console is running (27 retries left).
FAILED - RETRYING: Verify that the web console is running (26 retries left).
FAILED - RETRYING: Verify that the web console is running (25 retries left).
FAILED - RETRYING: Verify that the web console is running (24 retries left).
FAILED - RETRYING: Verify that the web console is running (23 retries left).
FAILED - RETRYING: Verify that the web console is running (22 retries left).
FAILED - RETRYING: Verify that the web console is running (21 retries left).
FAILED - RETRYING: Verify that the web console is running (20 retries left).
FAILED - RETRYING: Verify that the web console is running (19 retries left).
FAILED - RETRYING: Verify that the web console is running (18 retries left).
FAILED - RETRYING: Verify that the web console is running (17 retries left).
FAILED - RETRYING: Verify that the web console is running (16 retries left).
FAILED - RETRYING: Verify that the web console is running (15 retries left).
FAILED - RETRYING: Verify that the web console is running (14 retries left).
FAILED - RETRYING: Verify that the web console is running (13 retries left).
FAILED - RETRYING: Verify that the web console is running (12 retries left).
FAILED - RETRYING: Verify that the web console is running (11 retries left).
FAILED - RETRYING: Verify that the web console is running (10 retries left).
FAILED - RETRYING: Verify that the web console is running (9 retries left).
FAILED - RETRYING: Verify that the web console is running (8 retries left).
FAILED - RETRYING: Verify that the web console is running (7 retries left).
FAILED - RETRYING: Verify that the web console is running (6 retries left).
FAILED - RETRYING: Verify that the web console is running (5 retries left).
FAILED - RETRYING: Verify that the web console is running (4 retries left).
FAILED - RETRYING: Verify that the web console is running (3 retries left).
FAILED - RETRYING: Verify that the web console is running (2 retries left).
FAILED - RETRYING: Verify that the web console is running (1 retries left).
fatal: [ec2-54-190-196-89.us-west-2.compute.amazonaws.com]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "-k", "https://webconsole.openshift-web-console.svc/healthz"], "delta": "0:00:01.013538", "end": "2018-01-09 13:12:56.996536", "msg": "non-zero return code", "rc": 7, "start": "2018-01-09 13:12:55.982998", "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused", "stderr_lines": ["  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", "                                 Dload  Upload   Total   Spent    Left  Speed", "", "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0", "  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused"], "stdout": "", "stdout_lines": []}
	to retry, use: --limit @/home/slave3/workspace/Launch Environment Flexy/private-openshift-ansible/playbooks/deploy_cluster.retry

Expected results:

Successful install

Additional info:


Inventory with some info redacted

[OSEv3:children]
masters
nodes

etcd





[OSEv3:vars]

#The following parameters is used by post-actions
iaas_name=AWS
use_rpm_playbook=true
openshift_playbook_rpm_repos=[{'id': 'aos-playbook-rpm', 'name': 'aos-playbook-rpm', 'baseurl': 'http://download.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/3.9/latest/x86_64/os', 'enabled': 1, 'gpgcheck': 0}]




update_is_images_url=registry.reg-aws.openshift.com:443











#The following parameters is used by openshift-ansible
ansible_ssh_user=root




openshift_cloudprovider_kind=aws

openshift_cloudprovider_aws_access_key=redacted

openshift_cloudprovider_aws_secret_key=redacted












openshift_master_default_subdomain_enable=true
openshift_master_default_subdomain=apps.0109-b1n.qe.rhcloud.com




openshift_auth_type=allowall

openshift_master_identity_providers=[{'name': 'allow_all', 'login': 'true', 'challenge': 'true', 'kind': 'AllowAllPasswordIdentityProvider'}]



openshift_release=v3.9
openshift_deployment_type=openshift-enterprise
openshift_cockpit_deployer_prefix=registry.reg-aws.openshift.com:443/openshift3/
oreg_url=registry.reg-aws.openshift.com:443/openshift3/ose-${component}:${version}
oreg_auth_user={{ lookup('env','REG_AUTH_USER') }}
oreg_auth_password={{ lookup('env','REG_AUTH_PASSWORD') }}
openshift_docker_additional_registries=registry.reg-aws.openshift.com:443
openshift_docker_insecure_registries=registry.reg-aws.openshift.com:443
openshift_service_catalog_image_prefix=registry.reg-aws.openshift.com:443/openshift3/ose-
ansible_service_broker_image_prefix=registry.reg-aws.openshift.com:443/openshift3/ose-
template_service_broker_prefix=registry.reg-aws.openshift.com:443/openshift3/
openshift_enable_service_catalog=true
osm_cockpit_plugins=['cockpit-kubernetes']
osm_use_cockpit=false
openshift_docker_options=--log-opt max-size=20M --log-opt max-file=5 --signature-verification=false
use_cluster_metrics=true
openshift_master_cluster_method=native
openshift_master_dynamic_provisioning_enabled=true
openshift_hosted_router_registryurl=registry.reg-aws.openshift.com:443/openshift3/ose-${component}:${version}
openshift_hosted_registry_registryurl=registry.reg-aws.openshift.com:443/openshift3/ose-${component}:latest
osm_default_node_selector=region=primary
openshift_registry_selector="region=infra,zone=default"
openshift_hosted_router_selector="region=infra,zone=default"
openshift_disable_check=disk_availability,memory_availability,package_availability,docker_image_availability,docker_storage,package_version
openshift_master_portal_net=172.24.0.0/14
openshift_portal_net=172.24.0.0/14
osm_cluster_network_cidr=172.20.0.0/14
osm_host_subnet_length=9
openshift_node_kubelet_args={"pods-per-core": ["0"], "max-pods": ["510"], "image-gc-high-threshold": ["80"], "image-gc-low-threshold": ["70"], "cloud-config":["/etc/origin/cloudprovider/aws.conf"]}
debug_level=2
openshift_set_hostname=true
openshift_override_hostname_check=true
os_sdn_network_plugin_name=redhat/openshift-ovs-multitenant
openshift_hosted_router_replicas=1
openshift_hosted_registry_storage_kind=object
openshift_hosted_registry_storage_provider=s3
openshift_hosted_registry_storage_s3_accesskey=redacted
openshift_hosted_registry_storage_s3_secretkey=redacted
openshift_hosted_registry_storage_s3_bucket=aoe-svt-test
openshift_hosted_registry_storage_s3_region=us-west-2
openshift_hosted_registry_replicas=1
openshift_metrics_install_metrics=false
openshift_metrics_image_prefix=registry.reg-aws.openshift.com:443/openshift3/
openshift_metrics_image_version=v3.9
openshift_metrics_cassandra_storage_type=dynamic
openshift_metrics_cassandra_pvc_size=25Gi
openshift_logging_install_logging=false
openshift_logging_image_prefix=registry.reg-aws.openshift.com:443/openshift3/
openshift_logging_image_version=v3.9
openshift_logging_storage_kind=dynamic
openshift_logging_es_pvc_size=50Gi
openshift_logging_es_pvc_dynamic=true
openshift_clusterid=mffiedler-39
openshift_use_system_containers=false
openshift_use_crio=false
system_images_registry=registry.reg-aws.openshift.com:443
openshift_image_tag=v3.9.0-0.16.0




[lb]


[etcd]
ec2-54-190-196-89.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-190-196-89.us-west-2.compute.amazonaws.com


[masters]
ec2-54-190-196-89.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-190-196-89.us-west-2.compute.amazonaws.com



[nodes]
ec2-54-190-196-89.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-190-196-89.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_scheduleable=false

ec2-54-218-127-22.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-218-127-22.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"

ec2-54-218-127-22.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-218-127-22.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"

ec2-54-186-253-57.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-54-186-253-57.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'primary', 'zone': 'default'}"
ec2-34-209-73-58.us-west-2.compute.amazonaws.com ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="/home/slave3/workspace/Launch Environment Flexy/private/config/keys/id_rsa_perf" openshift_public_hostname=ec2-34-209-73-58.us-west-2.compute.amazonaws.com openshift_node_labels="{'region': 'primary', 'zone': 'default'}"