Bug 1574935

Summary: Failed connect to webconsole.openshift-web-console.svc:443; Connection refused
Product: OpenShift Container Platform Reporter: Mahesh Taru <mtaru>
Component: Management ConsoleAssignee: Samuel Padgett <spadgett>
Status: CLOSED NOTABUG QA Contact: Yadan Pei <yapei>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: aos-bugs, jokerman, mmccomas, mtaru, sdodson
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-08 13:02:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Mahesh Taru 2018-05-04 11:41:29 UTC
Description of problem:

Following are the webconsole variables passed in inventory file.

openshift_web_console_prefix=satellite.example.com:5000/organization-containers-ose- 
openshift_web_console_version=latest

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Setup local docker registry or satellite6 and get docker repos and images sync on it [1].
2. Perform installation in verbose mode using following variables in inventory file.
****************************
[OSEv3:vars]
openshift_docker_additional_registries=satellite.example.com:5000
openshift_docker_insecure_registries=satellite.example.com:5000
openshift_docker_blocked_registries=registry.access.redhat.com,docker.io
oreg_url=satellite.example.com:5000/organization-containers-ose-${component}:${version}
openshift_examples_modify_imagestreams=true

openshift_cockpit_deployer_prefix=satellite.example.com:5000/organization-containers-
openshift_cockpit_deployer_version=latest
openshift_web_console_prefix=satellite.example.com:5000/organization-containers-ose-
openshift_web_console_version=latest
****************************

3. Installation completes but heath check for webconsole fails with following error.
*****************************************
2018-05-03 22:33:22,875 p=8197 u=root |  task path: /usr/share/ansible/openshift-ansible/roles/openshift_web_console/tasks/install.yml:137
2018-05-03 22:33:22,972 p=8197 u=root |  Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
2018-05-03 22:33:24,562 p=8197 u=root |  FAILED - RETRYING: Verify that the web console is running (60 retries left).Result was: {
    "attempts": 1, 
    "changed": false, 
    "cmd": [
        "curl", 
        "-k", 
        "https://webconsole.openshift-web-console.svc/healthz"
    ], 
    "delta": "0:00:01.026071", 
    "end": "2018-05-03 22:33:24.510148", 
    "failed": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "curl -k https://webconsole.openshift-web-console.svc/healthz", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "warn": false
        }
    }, 
    "msg": "non-zero return code", 
    "rc": 7, 
    "retries": 61, 
    "start": "2018-05-03 22:33:23.484077", 
    "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused", 
    "stderr_lines": [
        "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", 
        "                                 Dload  Upload   Total   Spent    Left  Speed", 
        "", 
        "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refus:
    "stderr_lines": [
        "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", 
        "                                 Dload  Upload   Total   Spent    Left  Speed", 
        "", 
        "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused"
    ], 
    "stdout": "", 
    "stdout_lines": []
}
2018-05-03 22:33:34,572 p=8197 u=root |  Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
2018-05-03 22:33:36,298 p=8197 u=root |  FAILED - RETRYING: Verify that the web console is running (59 retries left).Result was: {
*****************************************

[1] https://docs.openshift.com/container-platform/3.9/install_config/install/disconnected_install.html#disconnected-syncing-images


Actual results:
Installation logs shows error for webconsole health check.

Expected results:
Installation should complete without any error.

Additional info:

Comment 1 Samuel Padgett 2018-05-04 17:49:07 UTC
Can you include more of the ansible log? There is a lot of troubleshooting information that is printed when console install fails which will help us debug.

Comment 7 Samuel Padgett 2018-05-08 13:02:15 UTC
After the web console is installed, the installer waits to verify that the web console is ready before continuing. This is done by polling the web console service health endpoint until ready. Since the web console runs in a pod, there is some delay for the deployment to rollout and the pod to start, often because the web console image needs to be pulled. This results of each request to the service are output to the ansible log. As long as the health check succeeds in the time allowed (currently 10 minutes), the console was successfully installed. The log messages are normal and expected.

Note that we changed the way we check for readiness recently, which should reduce some of the noise in the logs: https://github.com/openshift/openshift-ansible/pull/8274

Based on the understanding that the console was installed successfully and the question is specifically on this output in the logs, this is working as intended. If the web console install did not complete and the install failed, let us know, and we can help troubleshoot the problem.