Bug 1594726 - Can't get the master pods during the installation in proxy environment
Summary: Can't get the master pods during the installation in proxy environment
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 3.10.0
Assignee: Vadim Rutkovsky
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-25 09:48 UTC by Gan Huang
Modified: 2018-12-20 21:47 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: node proxy list was not configured during node bootrapping Consequence: node bootstrapping failed in proxy environments Fix: proxy env vars are configured early Result: nodes are being bootstrapped correctly using proxy settings
Clone Of:
Environment:
Last Closed: 2018-12-20 21:42:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Gan Huang 2018-06-25 09:48:57 UTC
Description of problem:
Can't get the master pods during the installation in proxy environment

Version-Release number of the following components:
openshift-ansible-3.10.7-1.git.220.50204c4.el7.noarch.rpm

How reproducible:
always

Steps to Reproduce:
1. Trigger HA installation with haproxy LB set behind proxy


Actual results:
Installation failed at TASK [openshift_control_plane : Wait for all control plane pods to become ready] ***

<--snip-->
FAILED - RETRYING: Wait for all control plane pods to become ready (1 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (1 retries left).
failed: [host-8-249-254.host.centralci.eng.rdu2.redhat.com] (item=controllers) => {"attempts": 60, "changed": false, "failed": true, "item": "controllers", "results": {"cmd": "/usr/bin/oc get pod master-controllers-ghuang-bug-master-etcd-2 -o json -n kube-system", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): pods \"master-controllers-ghuang-bug-master-etcd-2\" not found\n", "stdout": ""}, "state": "list"}
failed: [host-8-240-239.host.centralci.eng.rdu2.redhat.com] (item=controllers) => {"attempts": 60, "changed": false, "failed": true, "item": "controllers", "results": {"cmd": "/usr/bin/oc get pod master-controllers-ghuang-bug-master-etcd-1 -o json -n kube-system", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): pods \"master-controllers-ghuang-bug-master-etcd-1\" not found\n", "stdout": ""}, "state": "list"}
failed: [host-8-250-249.host.centralci.eng.rdu2.redhat.com] (item=controllers) => {"attempts": 60, "changed": false, "failed": true, "item": "controllers", "results": {"cmd": "/usr/bin/oc get pod master-controllers-ghuang-bug-master-etcd-3 -o json -n kube-system", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): pods \"master-controllers-ghuang-bug-master-etcd-3\" not found\n", "stdout": ""}, "state": "list"}

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 11 Vadim Rutkovsky 2018-06-25 16:25:02 UTC
Created https://github.com/openshift/openshift-ansible/pull/8962, I'll reuse the same inventory to verify this fix is sufficient

Comment 12 Vadim Rutkovsky 2018-06-26 08:45:33 UTC
3.10 cherrypick - https://github.com/openshift/openshift-ansible/pull/8979

Comment 13 Vadim Rutkovsky 2018-06-26 15:47:07 UTC
The previous PR was insufficient, https://github.com/openshift/openshift-ansible/pull/8984 for master did the trick on provided inventory

Comment 14 Vadim Rutkovsky 2018-06-26 16:48:45 UTC
The installation however seems to get stuck at approving the nodes later on. All CSRs are in 'Approved,Issued' state, so this might be a misconfiguration

Comment 15 Gan Huang 2018-06-27 13:39:39 UTC
Tested against the latest release-3.10 including the two fixes. No issues found.

Comment 16 Vadim Rutkovsky 2018-06-29 08:35:29 UTC
Fix is available in openshift-ansible-3.10.10-1

Comment 17 Gan Huang 2018-07-02 05:55:12 UTC
Verified in openshift-ansible-3.10.10-1.git.248.0bb6b58.el7.noarch.rpm

[root@qe-ghuang-bug-master-etcd-1 ~]# grep -A 3 "NO_PROXY" /etc/origin/master/master-config.yaml 
        - name: NO_PROXY
          value: .xxxx,.cluster.local,.xxxxx,.svc,10.14.89.4,169.254.169.254,172.16.120.104,172.16.120.17,172.16.120.67,172.31.0.1,qe-ghuang-bug-lb-nfs-1,qe-ghuang-bug-master-etcd-1,qe-ghuang-bug-master-etcd-2,qe-ghuang-bug-master-etcd-3,qe-ghuang-bug-node-1,qe-ghuang-bug-node-2,qe-ghuang-bug-node-registry-router-1


Note You need to log in before you can comment on or make changes to this bug.