Bug 1574935 - Failed connect to webconsole.openshift-web-console.svc:443; Connection refused
Summary: Failed connect to webconsole.openshift-web-console.svc:443; Connection refused
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Management Console
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.9.z
Assignee: Samuel Padgett
QA Contact: Yadan Pei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-04 11:41 UTC by Mahesh Taru
Modified: 2018-05-08 13:02 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-08 13:02:15 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Mahesh Taru 2018-05-04 11:41:29 UTC
Description of problem:

Following are the webconsole variables passed in inventory file.

openshift_web_console_prefix=satellite.example.com:5000/organization-containers-ose- 
openshift_web_console_version=latest

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Setup local docker registry or satellite6 and get docker repos and images sync on it [1].
2. Perform installation in verbose mode using following variables in inventory file.
****************************
[OSEv3:vars]
openshift_docker_additional_registries=satellite.example.com:5000
openshift_docker_insecure_registries=satellite.example.com:5000
openshift_docker_blocked_registries=registry.access.redhat.com,docker.io
oreg_url=satellite.example.com:5000/organization-containers-ose-${component}:${version}
openshift_examples_modify_imagestreams=true

openshift_cockpit_deployer_prefix=satellite.example.com:5000/organization-containers-
openshift_cockpit_deployer_version=latest
openshift_web_console_prefix=satellite.example.com:5000/organization-containers-ose-
openshift_web_console_version=latest
****************************

3. Installation completes but heath check for webconsole fails with following error.
*****************************************
2018-05-03 22:33:22,875 p=8197 u=root |  task path: /usr/share/ansible/openshift-ansible/roles/openshift_web_console/tasks/install.yml:137
2018-05-03 22:33:22,972 p=8197 u=root |  Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
2018-05-03 22:33:24,562 p=8197 u=root |  FAILED - RETRYING: Verify that the web console is running (60 retries left).Result was: {
    "attempts": 1, 
    "changed": false, 
    "cmd": [
        "curl", 
        "-k", 
        "https://webconsole.openshift-web-console.svc/healthz"
    ], 
    "delta": "0:00:01.026071", 
    "end": "2018-05-03 22:33:24.510148", 
    "failed": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "curl -k https://webconsole.openshift-web-console.svc/healthz", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "warn": false
        }
    }, 
    "msg": "non-zero return code", 
    "rc": 7, 
    "retries": 61, 
    "start": "2018-05-03 22:33:23.484077", 
    "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused", 
    "stderr_lines": [
        "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", 
        "                                 Dload  Upload   Total   Spent    Left  Speed", 
        "", 
        "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refus:
    "stderr_lines": [
        "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", 
        "                                 Dload  Upload   Total   Spent    Left  Speed", 
        "", 
        "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused"
    ], 
    "stdout": "", 
    "stdout_lines": []
}
2018-05-03 22:33:34,572 p=8197 u=root |  Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
2018-05-03 22:33:36,298 p=8197 u=root |  FAILED - RETRYING: Verify that the web console is running (59 retries left).Result was: {
*****************************************

[1] https://docs.openshift.com/container-platform/3.9/install_config/install/disconnected_install.html#disconnected-syncing-images


Actual results:
Installation logs shows error for webconsole health check.

Expected results:
Installation should complete without any error.

Additional info:

Comment 1 Samuel Padgett 2018-05-04 17:49:07 UTC
Can you include more of the ansible log? There is a lot of troubleshooting information that is printed when console install fails which will help us debug.

Comment 7 Samuel Padgett 2018-05-08 13:02:15 UTC
After the web console is installed, the installer waits to verify that the web console is ready before continuing. This is done by polling the web console service health endpoint until ready. Since the web console runs in a pod, there is some delay for the deployment to rollout and the pod to start, often because the web console image needs to be pulled. This results of each request to the service are output to the ansible log. As long as the health check succeeds in the time allowed (currently 10 minutes), the console was successfully installed. The log messages are normal and expected.

Note that we changed the way we check for readiness recently, which should reduce some of the noise in the logs: https://github.com/openshift/openshift-ansible/pull/8274

Based on the understanding that the console was installed successfully and the question is specifically on this output in the logs, this is working as intended. If the web console install did not complete and the install failed, let us know, and we can help troubleshoot the problem.


Note You need to log in before you can comment on or make changes to this bug.