1689263 – Internal error: Unable to set dialer for kube-service-catalog

Bug 1689263 - Internal error: Unable to set dialer for kube-service-catalog

Summary: Internal error: Unable to set dialer for kube-service-catalog

Keywords:
Status:	CLOSED DUPLICATE of bug 1680342
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Service Catalog
Sub Component:
Version:	3.11.0
Hardware:	Unspecified
OS:	Linux
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	Jay Boyd
QA Contact:	Jian Zhang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-03-15 14:24 UTC by szustkowski
Modified:	2019-03-19 11:44 UTC (History)
CC List:	0 users
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-03-19 11:44:23 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description szustkowski 2019-03-15 14:24:38 UTC

Description of problem:

Possibly similar to https://bugzilla.redhat.com/show_bug.cgi?id=1669131. An installation of OpenShift 3.11 fails when 

"Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered"

The error is: 

"Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserve
r as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding

===

Version-Release number of the following components:
openshift-ansible-3.11.92-1.git.0.f2fade7.el7.noarch
ansible-2.6.14-1.el7ae.noarch

ansible 2.6.14
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/tu_ose_ansible/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

===

How reproducible:

Steps to Reproduce:
1. Try to install OpenShift 3.11 from scratch (doesnt matter if ovs-multitenant is enabled or disabled), with enabled service catalog. 
2. Wait for the installer to get to this particular task
3.

Actual results:
TASK [openshift_control_plane : Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered] *************************************************************************************************************************************************
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (30 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (30 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (30 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (29 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (29 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (29 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (28 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (28 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (28 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (27 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (27 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (27 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (26 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (26 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (26 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (25 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (25 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (25 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (24 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (24 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (24 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (23 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (23 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (23 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (22 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (22 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (22 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (21 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (21 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (21 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (20 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (20 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (20 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (19 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (19 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (19 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (18 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (18 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (18 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (17 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (17 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (17 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (16 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (16 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (16 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (15 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (15 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (15 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (14 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (14 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (14 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (13 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (13 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (13 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (12 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (12 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (12 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (11 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (11 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (11 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (10 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (10 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (10 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (9 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (9 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (8 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (9 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (8 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (7 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (8 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (7 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (6 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (7 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (6 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (5 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (6 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (5 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (4 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (5 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (4 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (3 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (4 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (3 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (2 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (3 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (2 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (1 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (2 retries left).
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (1 retries left).
fatal: [prod-ose3-master2.internal.viada.de]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["oc", "--config=/etc/origin/master/admin.kubeconfig", "get", "--raw", "/apis/servicecatalog.k8s.io/v1beta1"], "delta": "0:00:00.163678", "en
d": "2019-03-15 14:52:41.209982", "msg": "non-zero return code", "rc": 1, "start": "2019-03-15 14:52:41.046304", "stderr": "Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserve
r as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding", "stderr_lines": ["Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding"], "stdout": "", "stdout_lines": []}
FAILED - RETRYING: Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered (1 retries left).
fatal: [prod-ose3-master1.internal.viada.de]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["oc", "--config=/etc/origin/master/admin.kubeconfig", "get", "--raw", "/apis/servicecatalog.k8s.io/v1beta1"], "delta": "0:00:00.153208", "end": "2019-03-15 14:52:44.034452", "msg": "non-zero return code", "rc": 1, "start": "2019-03-15 14:52:43.881244", "stderr": "Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding", "stderr_lines": ["Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding"], "stdout": "", "stdout_lines": []}
fatal: [prod-ose3-master3.internal.viada.de]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["oc", "--config=/etc/origin/master/admin.kubeconfig", "get", "--raw", "/apis/servicecatalog.k8s.io/v1beta1"], "delta": "0:00:00.291200", "end": "2019-03-15 14:52:48.847290", "msg": "non-zero return code", "rc": 1, "start": "2019-03-15 14:52:48.556090", "stderr": "Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding", "stderr_lines": ["Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding"], "stdout": "", "stdout_lines": []}

NO MORE HOSTS LEFT ****************************************************************************************************************************************************************************************************************************
 [WARNING]: Could not create retry file '/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry'.         [Errno 13] Permission denied: u'/usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.retry'


PLAY RECAP ************************************************************************************************************************************************************************************************************************************
localhost                  : ok=11   changed=0    unreachable=0    failed=0
prod-ose3-infra1.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-infra2.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-infra3.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-master1.internal.viada.de : ok=299  changed=51   unreachable=0    failed=1   
prod-ose3-master2.internal.viada.de : ok=238  changed=46   unreachable=0    failed=1   
prod-ose3-master3.internal.viada.de : ok=238  changed=46   unreachable=0    failed=1   
prod-ose3-worker1.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-worker2.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
prod-ose3-worker3.internal.viada.de : ok=105  changed=16   unreachable=0    failed=0
INSTALLER STATUS ******************************************************************************************************************************************************************************************************************************
Initialization              : Complete (0:01:22)
Health Check                : Complete (0:01:15)
Node Bootstrap Preparation  : Complete (0:17:02)
etcd Install                : Complete (0:01:46)
Master Install              : In Progress (0:07:16)
        This phase can be restarted by running: playbooks/openshift-master/config.yml


Failure summary:


  1. Hosts:    prod-ose3-master1.internal.viada.de, prod-ose3-master2.internal.viada.de, prod-ose3-master3.internal.viada.de
     Play:     Configure masters
     Task:     Wait for /apis/servicecatalog.k8s.io/v1beta1 when registered
     Message:  non-zero return code

===

Expected results:
This task should succeed without errors.

Comment 1 Jay Boyd 2019-03-15 14:57:03 UTC

This error:

Error from server (InternalError): an error on the server (\"unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper\") has prevented the request from succeeding"

is a pretty solid indication you are hitting the same issue described in https://bugzilla.redhat.com/show_bug.cgi?id=1680342.

There is a bug in core Kubernetes (https://github.com/kubernetes/kubernetes/issues/65848) where if the verbosity level for the Kube API Server is configured >5 it will cause this error and log the error "unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper"

Can you please check the configuration and verify the log level?  And change loglevel < 5 if applicable?

Comment 2 szustkowski 2019-03-19 08:08:59 UTC

Hey Jay,

thanks for your help. This was exactly the issue. I have set the loglevel to 4 (it was 6 when i opened this bugreport), and the installation succeeded. So, it is a duplicate of #1680342. 

Best regards, Simon

Comment 3 Jay Boyd 2019-03-19 11:44:23 UTC

Excellent, thanks Simon.

*** This bug has been marked as a duplicate of bug 1680342 ***

Note You need to log in before you can comment on or make changes to this bug.