Bug 1458660 - [DOCS] NO_PROXY of etcd IPs need to be configured in master-controllers
Summary: [DOCS] NO_PROXY of etcd IPs need to be configured in master-controllers
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Documentation
Version: 3.5.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Alex Dellapenta
QA Contact: Gan Huang
Vikram Goyal
: 1459505 (view as bug list)
Depends On:
Blocks: 1557156 1466783
TreeView+ depends on / blocked
Reported: 2017-06-05 06:54 UTC by Kenjiro Nakayama
Modified: 2018-03-16 06:33 UTC (History)
11 users (show)

Clone Of:
: 1466783 1557156 (view as bug list)
Last Closed: 2017-07-12 20:49:57 UTC

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Github openshift openshift-ansible issues 4490 None None None 2017-06-19 12:45 UTC
Red Hat Bugzilla 1459505 None CLOSED atomic-openshift-master-controllers reports etcd cluster is unavailable or misconfigured; error #0: Forbidden 2019-05-30 10:41 UTC
Red Hat Knowledge Base (Solution) 3158961 None None None 2017-08-22 03:25 UTC

Internal Trackers: 1459505

Description Kenjiro Nakayama 2017-06-05 06:54:32 UTC
Description of problem:
-  According to https://docs.openshift.com/container-platform/3.5/install_config/http_proxies.html#configuring-no-proxy, etcd IPs are not necessary to be added in master-controllers of NO_PROXY. However, we experienced an issue if there are no etcd's IP in NO_PROXY of /etc/sysconfig/atomic-openshift-master-controllers

Version-Release number of selected component (if applicable):
- OCP 3.5

Steps to Reproduce:
1. Install OCP 3.5 behind the proxy

Actual results:
- master-controllers keeps restarding with below error.
  atomic-openshift-master-controllers[72302]: ; error #1: Forbidden
  atomic-openshift-master-controllers[72302]: ; error #2: Forbidden
  atomic-openshift-master-controllers[72302]: E0530 14:05:07.264674   72302 leaderlease.go:95] unable to check lease openshift.io/leases/controllers: Forbidden
  atomic-openshift-master-controllers[72302]: E0530 14:05:07.295492   72302 leaderlease.go:87] client: etcd cluster is unavailable or misconfigured; error #0: Forbidden

Expected results:
- OCP should work without NO_PROXY config for the etcd IPs. Otherwise, Docs should mention about it.
- (However, /etc/origin.)
  - https://knakayam-ose35-master1.example.com:2379

Additional info:
- put the customers' logs in private.

Comment 3 Jordan Liggitt 2017-06-06 14:52:54 UTC
I would expect the etcd hosts to be included in NO_PROXY for the openshift-master and openshift-master-controllers

Not sure if this is docs or ansible or both

Comment 4 Scott Dodson 2017-06-06 15:06:58 UTC

Reviewing the playbooks the etcd hosts should automatically be added to the NO_PROXY list, can you confirm whether or not the proxy variables were defined manually or by the installer?

Comment 5 Kenjiro Nakayama 2017-06-06 15:44:37 UTC
Hi Scott, Jordan,

NO_PROXY had etcd's hostname, but it didn't have etcd's IP and we had to add it. Here are the short summary.

  1. ansible inventory doesn't have NO_PROXY(=openshift_no_proxy) config for etcd's IP.
  2. ansible inventory also doesn't have etcd's IPs, as it defines them as etcd's hostname.
  3. After running ansible, NO_PROXY in /etc/sysconfig/atomic-openshift-master-controllers doesn't have etcd's IP.
  4. After running ansible, NO_PROXY in /etc/sysconfig/atomic-openshift-master-controllers had etcd's hostname.
  5. We had to _manually_ add NO_PROXY=<etcd *IPs*> in /etc/sysconfig/atomic-openshift-master-controllers

/etc/origin/master/master-config.yaml also had etcd's hostname, so I think IPs should not be necessary to be added. (But again, the proxy log showed the access and fixed by adding etcd's IP)

Comment 9 Scott Dodson 2017-06-06 15:49:57 UTC
I think the etcd client prefers to connect to the IP address even if you've specified the endpoints using a hostname. We should make sure that the ip address of the etcd endpoints is added via the playbooks too.

Comment 10 Kenjiro Nakayama 2017-06-06 15:54:50 UTC
Sounds great. Thank you Scott.

Comment 11 Seth Jennings 2017-06-19 17:56:39 UTC
*** Bug 1459505 has been marked as a duplicate of this bug. ***

Comment 12 Brenton Leanhardt 2017-06-30 12:20:07 UTC
Scott, should this bug be cloned to the Installer?

Comment 13 Scott Dodson 2017-06-30 12:44:17 UTC
(In reply to Brenton Leanhardt from comment #12)
> Scott, should this bug be cloned to the Installer?

Yeah, I think we have one already but on the off chance that we don't I'll go ahead and clone it and set it 3.6.1.

Comment 14 Alex Dellapenta 2017-07-06 20:51:10 UTC
See https://github.com/openshift/openshift-docs/pull/4720 for description of changes and links to preview docs.

Comment 15 Gan Huang 2017-07-11 02:39:46 UTC
Looks good to me! Sorry for the late response.

Comment 16 Alex Dellapenta 2017-07-12 20:49:57 UTC
Published changes described in revhistory here:


Note You need to log in before you can comment on or make changes to this bug.