Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1589734

Summary: node hostname is not added into NO_PROXY list in /etc/origin/master/master.env after upgrade
Product: OpenShift Container Platform Reporter: Johnny Liu <jialiu>
Component: Cluster Version OperatorAssignee: Vadim Rutkovsky <vrutkovs>
Status: CLOSED CURRENTRELEASE QA Contact: Johnny Liu <jialiu>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.10.0CC: aos-bugs, jialiu, jokerman, mmccomas, vlaad, vrutkovs
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-08 14:13:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
upgrade log with inventory file embeded none

Description Johnny Liu 2018-06-11 09:46:13 UTC 
Created attachment 1449951 [details]
upgrade log with inventory file embeded

Description of problem:
See the following details.

Version-Release number of the following components:
openshift-ansible-3.10.0-0.64.0.git.20.48df973.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. Set up a 3.9 cluster behind proxy, make sure "oc logs" is working well.
2. upgrade the cluster to 3.10
3.

Actual results:
After upgrade, check NO_PROXY setting in /etc/origin/master/master.env
[root@qe-jialiu392-master-etcd-nfs-1 ~]# cat /etc/origin/master/master.env |grep NO_PROXY
NO_PROXY=.centralci.eng.rdu2.redhat.com,.cluster.local,.svc,169.254.169.254,172.30.0.1,192.168.100.15,qe-jialiu392-master-etcd-nfs-1,172.30.0.0/16,10.128.0.0/14

[root@qe-jialiu392-master-etcd-nfs-1 ~]# oc -n default logs router-2-5x64d
Error from server: Get https://qe-jialiu392-node-registry-router-1:10250/containerLogs/default/router-2-5x64d/router: Forbidden

[root@qe-jialiu392-master-etcd-nfs-1 ~]# ping qe-jialiu392-node-registry-router-1
PING qe-jialiu392-node-registry-router-1.openshift-snvl2.internal (192.168.100.12) 56(84) bytes of data.
64 bytes from 192.168.100.12 (192.168.100.12): icmp_seq=1 ttl=64 time=0.435 ms


Expected results:
node hostname should be added into NO_PROXY list in /etc/origin/master/master.env

Additional info:
After the above failure, add "qe-jialiu392-node-registry-router-1" into NO_PROXY   list, restart master static pod, "oc logs" works well.

3.10 fresh installation have no such issue.
Comment 1 Vadim Rutkovsky 2018-06-11 16:27:31 UTC 
Node hostnames should be added to noproxy list, which is composed of hostnames in oo_nodes_to_config, oo_masters_to_config and oo_etcd_to_config.

I see `openshift_no_proxy_internal_hostnames` is being set on master during node upgrade, was this change not propagated to master.env?

If this is reproducible could you attach a dump of ansible facts - /etc/ansible/facts.d on hosts - to check if openshift.common.no_proxy was set.
Comment 4 Vadim Rutkovsky 2018-06-15 14:33:35 UTC 
Turns out we've excluded facts collection from nodes during upgrade, but it should be required for 3.10

Created https://github.com/openshift/openshift-ansible/pull/8782
Comment 5 Vadim Rutkovsky 2018-06-21 07:53:33 UTC 
Fix is available in openshift-ansible-3.10.2-1
Comment 6 Johnny Liu 2018-06-22 03:33:36 UTC 
Verified this bug with openshift-ansible-3.10.3-1.git.200.e950396.el7.noarch, and PASS.


After upgrade, node hostname is added into NO_PROXY list.
# cat /etc/origin/master/master.env 

# Proxy configuration
# See https://docs.openshift.com/enterprise/latest/install_config/install/advanced_install.html#configuring-global-proxy
HTTP_PROXY=http://file.rdu.redhat.com:3128
HTTPS_PROXY=http://file.rdu.redhat.com:3128
NO_PROXY=.centralci.eng.rdu2.redhat.com,.cluster.local,.lab.sjc.redhat.com,.svc,10.14.89.4,169.254.169.254,172.30.0.1,192.168.100.9,qe-jialiu39-master-etcd-nfs-1,qe-jialiu39-node-registry-router-1,172.30.0.0/16,10.128.0.0/14

DEBUG_LOGLEVEL=5


`oc log` is running well.
# oc logs registry-console-2-jpkh4
INFO: cockpit-ws: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert