Bug 1511870 - Failed to push image in proxy environment
Summary: Failed to push image in proxy environment
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.z
Assignee: Fabian von Feilitzsch
QA Contact: Gan Huang
Johnny Liu
URL:
Whiteboard:
: 1544073 1544682 1575050 (view as bug list)
Depends On: 1527210 1541625
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-10 10:35 UTC by Gan Huang
Modified: 2018-05-08 14:24 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Kubernetes service IP was not added to no_proxy list for the docker-registry Consequence: Internal registry requests would be forced to use the proxy, preventing logins and pushes to the internal registry. Fix: Added the kubernetes service IP to the no_proxy list Result: The internal registry requests are no longer proxied, and logins and pushes to the internal registry succeed as expected.
Clone Of:
Environment:
Last Closed: 2018-05-08 14:24:44 UTC


Attachments (Terms of Use)

Comment 2 Johnny Liu 2017-11-10 11:34:47 UTC
This is blocking testing behind proxy

Comment 3 Gan Huang 2017-11-13 09:25:49 UTC
The registry version seems to be v2.6.2

# oc logs docker-registry-1-fc7f4 |grep distribution_version
time="2017-11-13T08:15:29.183080643Z" level=info msg="start registry" distribution_version="v2.6.2+unknown" kubernetes_version=v1.7.6+a08f5eeb62 openshift_version=v3.7.7

Comment 4 Scott Dodson 2017-11-13 16:25:49 UTC
I think this is a regression and therefore a blocker. I think the fix is to ensure that the kube service ip is added to the no_proxy list. There's another bug on this that's got more information, let me find that.

Comment 5 Scott Dodson 2017-11-13 16:52:07 UTC
I think this is a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1504464 and I think the correct way to fix this is to set KUBERNETES_MASTER='kubernetes.svc.default' on the registry whenever we configure a proxy so that it connects to the api server via dns name rather than ip address.

Since we actually regressed this in 3.6.z this is by definition not a 3.7 blocker. We'll fix this ASAP however, ideally on 3.7 GA day.

Comment 6 Gan Huang 2017-11-14 05:55:16 UTC
(In reply to Scott Dodson from comment #5)
> I think this is a dupe of
> https://bugzilla.redhat.com/show_bug.cgi?id=1504464 and I think the correct
> way to fix this is to set KUBERNETES_MASTER='kubernetes.svc.default' on the

I'm thinking that the correct route should be "kubernetes.default.svc" :)

I tried with setting KUBERNETES_MASTER='kubernetes.default.svc', things still don't work.

After appending `172.30.0.1` to NO_PROXY of docker-registry dc, build succeeded.

# oc env dc/docker-registry NO_PROXY=<--snip-->,172.30.0.1

Comment 10 Johnny Liu 2018-02-07 02:29:26 UTC
I think the fix should be backport to 3.7 branch, and fix this 3.7 bug, this is really a very basic functionality (sti build behind proxy)

Comment 11 Johnny Liu 2018-02-07 02:31:11 UTC
Before backport the PR to 3.7, pls fix Bug 1541625 together, Bug 1541625 is introduced by this PR.

Comment 12 Scott Dodson 2018-02-07 02:49:13 UTC
ACK, lets treat this as the bug to track the backport from master.

Comment 13 Scott Dodson 2018-02-09 15:07:03 UTC
Need to backport these two to release-3.7 for this bug

https://github.com/openshift/openshift-ansible/pull/7055
https://github.com/openshift/openshift-ansible/pull/6215

Comment 14 Scott Dodson 2018-02-12 18:49:24 UTC
*** Bug 1544073 has been marked as a duplicate of this bug. ***

Comment 15 Ben Parees 2018-02-13 14:55:47 UTC
*** Bug 1544682 has been marked as a duplicate of this bug. ***

Comment 16 Fabian von Feilitzsch 2018-02-13 21:22:24 UTC
backports: https://github.com/openshift/openshift-ansible/pull/7137

Comment 17 Scott Dodson 2018-02-22 21:51:25 UTC
In openshift-ansible-3.7.32-1

Comment 18 Johnny Liu 2018-02-23 06:02:30 UTC
Verified this bug with openshift-ansible-3.9.0-0.48.0.git.0.2fb33db.el7.noarch, and PASS.

# oc env dc docker-registry --list |grep -i proxy
NO_PROXY=.centralci.eng.rdu2.redhat.com,.cluster.local,.svc,169.254.169.254,172.16.120.106,172.16.120.64,172.31.0.1
HTTP_PROXY=http://file.rdu.redhat.com:3128
HTTPS_PROXY=http://file.rdu.redhat.com:3128

kubernetes svc IP (172.31.0.1) is added into NO_PROXY list, sti build succeed.

# oc get po -n install-test
NAME                             READY     STATUS      RESTARTS   AGE
mongodb-1-4w6ln                  1/1       Running     0          2h
nodejs-mongodb-example-1-build   0/1       Completed   0          2h
nodejs-mongodb-example-1-r5g6c   1/1       Running     0          2h

Comment 19 Scott Dodson 2018-05-08 14:24:44 UTC
Fixed in 	openshift-ansible-3.7.42-1 and later

Comment 20 Scott Dodson 2018-05-08 14:24:57 UTC
*** Bug 1575050 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.