Bug 1459102 - Standalone registry ignores proxy settings when pulling through
Standalone registry ignores proxy settings when pulling through
Status: NEW
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
3.5.0
Unspecified Unspecified
medium Severity medium
: ---
: 3.6.z
Assigned To: Kenny Woodson
Johnny Liu
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-06 05:52 EDT by Johan Swensson
Modified: 2017-07-17 14:31 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
registry log (53.94 KB, text/plain)
2017-06-06 05:52 EDT, Johan Swensson
no flags Details

  None (edit)
Description Johan Swensson 2017-06-06 05:52:15 EDT
Created attachment 1285321 [details]
registry log

Description of problem:
When using pull through via a standalone registry proxy settings are ignores so the pull fails.

Version-Release number of selected component (if applicable):
3.5.5.15

How reproducible:
Every time

Steps to Reproduce:
1. Deploy a standalone registry with proxy configured, for example

openshift_http_proxy=http://proxy.lab.example.com:3128
openshift_https_proxy=https://proxy.lab.example.com:3128

openshift_generate_no_proxy_hosts=True


2. Make sure firewall blocks any outgoing requests from the registry, as everything should be going through the proxy.

3. Configure image streams as desired

3. Try to pull an image through the standalone registry
docker pull docker-registry-default.registry.lab.example.com/openshift3/ose-pod:v3.5

Actual results:
Pull fails.

10.128.0.1 - - [06/Jun/2017:09:36:12 +0000] "GET /healthz HTTP/2.0" 200 0 "" "Go-http-client/2.0"
time="2017-06-06T09:36:13.105475796Z" level=error msg="error getting remote repository for image \"registry.access.redhat.com/openshift3/ose-pod:latest\": Get https://registry.access.redhat.com/v2/: dial tcp 209.132.182.63:443: getsockopt: connection refused" go.version=go1.7.4 http.request.host=docker-registry-default.registry.lab.example.com http.request.id=14a1d47c-b05e-4d39-b432-41b12051e3fc http.request.method=GET http.request.remoteaddr="10.128.0.1:50484" http.request.uri="/v2/openshift3/ose-pod/manifests/v3.5" http.request.useragent="docker/1.12.6 go/go1.7.4 kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=f64477be-3e69-442c-9313-e50eac3e684d openshift.auth.user=anonymous openshift.logger=registry vars.name="openshift3/ose-pod" vars.reference=v3.5 
time="2017-06-06T09:36:13.105738307Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="Get https://registry.access.redhat.com/v2/: dial tcp 209.132.182.63:443: getsockopt: connection refused" err.message="manifest unknown" go.version=go1.7.4 http.request.host=docker-registry-default.registry.lab.example.com http.request.id=14a1d47c-b05e-4d39-b432-41b12051e3fc http.request.method=GET http.request.remoteaddr="10.128.0.1:50484" http.request.uri="/v2/openshift3/ose-pod/manifests/v3.5" http.request.useragent="docker/1.12.6 go/go1.7.4 kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=10.730361201s http.response.status=404 http.response.written=278 instance.id=f64477be-3e69-442c-9313-e50eac3e684d openshift.auth.user=anonymous openshift.logger=registry vars.name="openshift3/ose-pod" vars.reference=v3.5 


Expected results:
The pull should succeed.

Additional info:
This was working with a 3.4 standalone registry.
Comment 1 Oleg Bulatov 2017-06-15 06:12:45 EDT
Does the http_proxy environment variable exist in the dockerregistry process?
```
$ sudo cat /proc/$(pidof dockerregistry)/environ | xargs -0 -I {} echo {} | grep -i proxy
http_proxy=127.0.0.1:8080
```
Comment 2 Johan Swensson 2017-06-15 06:33:47 EDT
It does not exist in the pod registry.

master-config.yaml:
(this is from a different environment so some addresses are different from before):

admissionConfig:
  pluginConfig:
    BuildDefaults:
      configuration:
        apiVersion: v1
        env:
        - name: HTTP_PROXY
          value: http://192.168.122.62:3128
        - name: HTTPS_PROXY
          value: https://192.168.122.62:3128
        - name: NO_PROXY
          value: .cluster.local,registry.example.com
        - name: http_proxy
          value: http://192.168.122.62:3128
        - name: https_proxy
          value: https://192.168.122.62:3128
        - name: no_proxy
          value: .cluster.local,registry.example.com
        gitHTTPProxy: http://192.168.122.62:3128
        gitHTTPSProxy: https://192.168.122.62:3128
        gitNoProxy: .cluster.local,registry.example.com
        kind: BuildDefaultsConfig
        resources:
          limits: {}
          requests: {}
Comment 3 Johan Swensson 2017-06-15 06:35:14 EDT
And from /etc/sysconfig/docker:

HTTP_PROXY='http://192.168.122.62:3128'
HTTPS_PROXY='https://192.168.122.62:3128'
NO_PROXY='.cluster.local,registry.example.com'
Comment 4 Oleg Bulatov 2017-06-15 09:57:17 EDT
/etc/sysconfig/docker and BuildDefaults are not related to the registry.

dockerregistry has to be in the registry pod, try to inspect it using this command:

oc -n default rsh dc/docker-registry cat /proc/1/environ | xargs -n 1 -0
Comment 5 Johan Swensson 2017-06-15 10:06:38 EDT
That was what I did and it's not set.
If it's required to manually add it to the deploymentconfig even tho it's configured in the installer maybe this is a doc bug?
Comment 6 Oleg Bulatov 2017-06-15 10:45:33 EDT
# These options configure HTTP_PROXY, HTTPS_PROXY, and NOPROXY environment
# variables for docker and master services.
#openshift_http_proxy=http://USER:PASSWORD@IPADDR:PORT
#openshift_https_proxy=https://USER:PASSWORD@IPADDR:PORT
#openshift_no_proxy='.hosts.example.com,some-host.com'

The registry is neither Docker nor a master service. And as I can see, the registry's environment variables are not tunable from the inventory file. So I guess you should manually add it to the deploymentconfig.
Comment 8 Scott Dodson 2017-06-27 15:43:54 EDT
If openshift.common.https_proxy or openshift.common.http_proxy != '' then we should set HTTP_PROXY, HTTPS_PROXY, and NO_PROXY to openshift.common.http_proxy openshift.common.https_proxy and openshift.common.no_proxy respectively.

Note You need to log in before you can comment on or make changes to this bug.