Bug 1762618 - [proxy] report-progress.sh on bootstrap need external api server to create bootstrap configmap resource
Summary: [proxy] report-progress.sh on bootstrap need external api server to create bo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.3.0
Assignee: Jeremiah Stuever
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks: 1766066 1770223
TreeView+ depends on / blocked
 
Reported: 2019-10-17 04:42 UTC by Johnny Liu
Modified: 2020-01-23 11:08 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1766066 (view as bug list)
Environment:
Last Closed: 2020-01-23 11:07:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 2647 0 'None' closed Bug 1762618: pkg/asset/ignition: bootstrap kubeconfig to use api-int 2021-02-09 08:32:56 UTC
Red Hat Product Errata RHBA-2020:0062 0 None None None 2020-01-23 11:08:19 UTC

Description Johnny Liu 2019-10-17 04:42:03 UTC
Description of problem:

Version-Release number of the following components:
4.3.0-0.nightly-2019-10-16-010826

How reproducible:
Always

Steps to Reproduce:
1. Drop internet gateway for private subnets in VPC to create a disconnected env
2. Set up a proxy in public subnets, the proxy could be connected both external and internal network.
3. In proxy, use whitelist to control which traffic could get through, NOT adding api url into the list. such as:
acl whitelist dstdomain ec2.us-east-2.amazonaws.com iam.amazonaws.com .s3.us-east-2.amazonaws.com .apps.jialiu-42dis8.qe.devcluster.openshift.com ec2-18-191-189-164.us-east-2.compute.amazonaws.com .github.com .rubygems.org 
http_access allow whitelist
4. Enable proxy setting in install-config.yaml
5. Trigger a UPI install on aws

Actual results:
$ ./openshift-install wait-for bootstrap-complete --dir '/home/installer2/workspace/Launch Environment Flexy/workdir/install-dir'
level=info msg="Waiting up to 30m0s for the Kubernetes API at https://api.jialiu-42dis8.qe.devcluster.openshift.com:6443..."
level=info msg="API v1.16.0-beta.2+453eff1 up"
level=info msg="Waiting up to 30m0s for bootstrapping to complete..."
level=info msg="Use the following commands to gather logs from the cluster"
level=info msg="openshift-install gather bootstrap --help"
level=fatal msg="failed to wait for bootstrapping to complete: timed out waiting for the condition"

Expected results:
Installation get passed

Additional info:
Log into bootstrap node, bootkube service is completed successfully.
$ journalctl -b -f -u bootkube.service
-- Logs begin at Wed 2019-10-16 09:26:30 UTC. --
Oct 16 09:38:35 ip-10-0-61-231 bootkube.sh[1610]: Skipped "secret-control-plane-client-signer.yaml" secrets.v1./kube-control-plane-signer -n openshift-kube-apiserver-operator as it already exists
Oct 16 09:38:35 ip-10-0-61-231 bootkube.sh[1610]: Skipped "secret-csr-signer-signer.yaml" secrets.v1./csr-signer-signer -n openshift-kube-controller-manager-operator as it already exists
Oct 16 09:38:36 ip-10-0-61-231 bootkube.sh[1610]: Skipped "secret-initial-kube-controller-manager-service-account-private-key.yaml" secrets.v1./initial-service-account-private-key -n openshift-config as it already exists
Oct 16 09:38:36 ip-10-0-61-231 bootkube.sh[1610]: Skipped "secret-kube-apiserver-to-kubelet-signer.yaml" secrets.v1./kube-apiserver-to-kubelet-signer -n openshift-kube-apiserver-operator as it already exists
Oct 16 09:38:37 ip-10-0-61-231 bootkube.sh[1610]: Skipped "secret-loadbalancer-serving-signer.yaml" secrets.v1./loadbalancer-serving-signer -n openshift-kube-apiserver-operator as it already exists
Oct 16 09:38:37 ip-10-0-61-231 bootkube.sh[1610]: Skipped "secret-localhost-serving-signer.yaml" secrets.v1./localhost-serving-signer -n openshift-kube-apiserver-operator as it already exists
Oct 16 09:38:37 ip-10-0-61-231 bootkube.sh[1610]: Skipped "secret-service-network-serving-signer.yaml" secrets.v1./service-network-serving-signer -n openshift-kube-apiserver-operator as it already exists
Oct 16 09:38:38 ip-10-0-61-231 bootkube.sh[1610]: Skipped "user-ca-bundle-config.yaml" configmaps.v1./user-ca-bundle -n openshift-config as it already exists
Oct 16 09:38:38 ip-10-0-61-231 bootkube.sh[1610]: Tearing down temporary bootstrap control plane...
Oct 16 09:38:38 ip-10-0-61-231 bootkube.sh[1610]: bootkube.service complete

But report-progress.sh is reporting some error.
Oct 16 11:37:12 ip-10-0-61-231 report-progress.sh[1611]: error: unable to recognize "STDIN": Get https://api.jialiu-42dis8.qe.devcluster.openshift.com:6443/api?timeout=32s: Forbidden
Oct 16 11:37:18 ip-10-0-61-231 report-progress.sh[1611]: error: unable to recognize "STDIN": Get https://api.jialiu-42dis8.qe.devcluster.openshift.com:6443/api?timeout=32s: Forbidden
Oct 16 11:37:23 ip-10-0-61-231 report-progress.sh[1611]: error: unable to recognize "STDIN": Get https://api.jialiu-42dis8.qe.devcluster.openshift.com:6443/api?timeout=32s: Forbidden
Oct 16 11:37:28 ip-10-0-61-231 report-progress.sh[1611]: error: unable to recognize "STDIN": Get https://api.jialiu-42dis8.qe.devcluster.openshift.com:6443/api?timeout=32s: Forbidden


# env|grep -i proxy
HTTP_PROXY=http://ec2-18-191-189-164.us-east-2.compute.amazonaws.com:3128
NO_PROXY=.cluster.local,.svc,.us-east-2.compute.internal,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.jialiu-42dis8.qe.devcluster.openshift.com,etcd-0.jialiu-42dis8.qe.devcluster.openshift.com,etcd-1.jialiu-42dis8.qe.devcluster.openshift.com,etcd-2.jialiu-42dis8.qe.devcluster.openshift.com,localhost,test.no-proxy.com
HTTPS_PROXY=http://ec2-18-191-189-164.us-east-2.compute.amazonaws.com:3128

Check report-progress.sh code:
# cat /usr/local/bin/report-progress.sh
#!/usr/bin/env bash

KUBECONFIG="${1}"

wait_for_existance() {
	while [ ! -e "${1}" ]
	do
		sleep 5
	done
}

echo "Waiting for bootstrap to complete..."
wait_for_existance /opt/openshift/.bootkube.done

echo "Reporting install progress..."
while ! oc --config="$KUBECONFIG" create -f - <<-EOF
	apiVersion: v1
	kind: ConfigMap
	metadata:
	  name: bootstrap
	  namespace: kube-system
	data:
	  status: complete
EOF
do
	sleep 5
done


The script is calling oc command against external api server to create some resource. But the api server is not in NO_PROXY list.


This issue is a regression issue, which is caused by https://github.com/openshift/installer/pull/2425

Comment 1 Scott Dodson 2019-10-21 18:56:29 UTC
That PR was merged in order to resolve https://bugzilla.redhat.com/show_bug.cgi?id=1762618

My opinion is that customer proxy configuration should include external api in its whitelist.

Comment 2 Daneyon Hansen 2019-10-21 19:10:46 UTC
What is the reason for report-progress.sh to use the api-server's external name instead of internal name?

Comment 3 Johnny Liu 2019-10-22 01:07:08 UTC
> My opinion is that customer proxy configuration should include external api in its whitelist.
I think including external api in its whitelist is some kind of workaround.
The potential reasonable fix is run oc command against internal api in cluster itself instead of external api.

> What is the reason for report-progress.sh to use the api-server's external
> name instead of internal name?

The default kubeconfig for oc command is using the api-server's external name.

Comment 7 Johnny Liu 2019-11-13 07:46:11 UTC
Verified this bug with 4.3.0-0.nightly-2019-11-13-031554, and PASS.


On bootstrap:
[root@ip-10-0-63-30 ~]# env|grep -i proxy|grep api
NO_PROXY=.cluster.local,.svc,.us-east-2.compute.internal,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.jialiu43-bz.qe.devcluster.openshift.com,etcd-0.jialiu43-bz.qe.devcluster.openshift.com,etcd-1.jialiu43-bz.qe.devcluster.openshift.com,etcd-2.jialiu43-bz.qe.devcluster.openshift.com,localhost,test.no-proxy.com

[root@ip-10-0-63-30 ~]# cat /opt/openshift/auth/kubeconfig
clusters:
- cluster:
    <--snip-->
    server: https://api-int.jialiu43-bz.qe.devcluster.openshift.com:6443
  name: jialiu43-bz

Comment 9 errata-xmlrpc 2020-01-23 11:07:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.