Bug 1333595 - Docker registry sync does not use HTTP proxy configuration [NEEDINFO]
Summary: Docker registry sync does not use HTTP proxy configuration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Container Management - Runtime
Version: 6.2.0
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: Unspecified
Assignee: Sebastian Gräßl
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-05 23:06 UTC by Geoff Gatward
Modified: 2021-03-11 14:33 UTC (History)
48 users (show)

Fixed In Version: tfm-rubygem-runcible-1.12.0.4-1,foreman-1.15.6.43-1,tfm-rubygem-runcible-1.12.0.7-1,foreman-1.15.6.46-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1572304 (view as bug list)
Environment:
Last Closed: 2018-08-22 20:07:08 UTC
Target Upstream Version:
rbeyel: needinfo? (sgraessl)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 20201 0 Normal Closed Global HTTP proxy for outgoing requests 2021-02-03 03:01:14 UTC
Github theforeman foreman pull 4640 0 'None' closed Fixes #20201 - Global HTTP proxy for outgoing requests 2021-02-03 03:01:14 UTC
Red Hat Knowledge Base (Solution) 2906801 0 None None None 2017-02-03 20:29:21 UTC

Description Geoff Gatward 2016-05-05 23:06:13 UTC
Description of problem:
Adding a new Docker registry on a Satellite inside a network that requires the use of a HTTP proxy to access the internet results in 

Unable to log in to this Docker Registry - No route to host - connect(2) for "registry.access.redhat.com" port 443

Again, tcpdump is showing direct connect attempts to the upstream IP instead of using the proxy configured during installation.


Possibly related to BZ 1332395 for same issue with OSTree replication


Version-Release number of selected component (if applicable):
6.2.0 Beta 1 (RHEL7)



Steps to Reproduce:
1. Configure Satellite to use http proxy, ensure that direct internet connection is not possible.
2. Create new Docker registry either via the WebUI or with 
hammer docker registry create --name "Red Hat Openshift3" --url https://registry.access.redhat.com/openshift3


Actual results:
Could not create the registry:
  Unable to log in to this Docker Registry - No route to host - connect(2) for "registry.access.redhat.com" port 443



Expected results:
Connection to registry succeeds via web proxy

Comment 1 Geoff Gatward 2016-05-05 23:26:01 UTC
I found /etc/pulp/server/plugins.conf.d/docker_importer.json  which DOES seem to contain the correct proxy information (identical to the yum_importer.json which does work...   however docker is definitely not using these proxy settings.

root@sat62[/etc/pulp/server/plugins.conf.d] # cat docker_importer.json
{
        "proxy_host": "http://proxy.example.org",

        "proxy_port": 8080,


    "proxy_username": null,


    "proxy_password": null

}

Comment 4 Michael Hrivnak 2016-06-06 15:30:17 UTC
It works for me. I think there was an outage recently with registry.access.redhat.com, because I also had noticed the "no route to host" error. It seems to be working now.

I did this just now with pulp 2.8:

pulp-admin docker repo create --repo-id=rhel --feed=https://registry.access.redhat.com --upstream-name=rhel7 --enable-v2=false --enable-v1=true
pulp-admin docker repo sync run --repo-id=rhel


That worked both with a proxy specified in the json config, and without.

Comment 7 Michael Hrivnak 2016-06-07 16:10:44 UTC
Using iftop, I verified that it definitely used the proxy and did not communicate directly with registry.access.redhat.com.

Perhaps there is a file permission issue preventing pulp from reading your customer's docker_importer.json config file?

Comment 8 Mike McCune 2016-06-16 23:18:08 UTC
I verified that Pulp, in Satellite 6.2 Beta 2 is using the proxy for docker syncs.

1) configured satellite to use a squid proxy I have in my lab:

 satellite-installer --katello-proxy-password redhat --katello-proxy-port 8888 --katello-proxy-url http://proxy.example.com --katello-proxy-username admin 

this properly configures all the importer configs in 

/etc/pulp/server/plugins.conf.d

docker_importer.json  iso_importer.json  puppet_importer.json  yum_importer.json


# cat docker_importer.json
{
        "proxy_host": "http://proxy.example.com",
    
        "proxy_port": 8888,
    
        "proxy_username": "admin",
    
        "proxy_password": "redhat"
    
}

you should never have to hand configure these files.

2) added an iptables rule to restrict access to only my http proxy host

3) Synced a Docker repo from docker hub: registry-1.docker.io

4) Verified access and logging in squid on the proxy itself:


1466118758.647     88 10.8.0.147 TCP_MISS/200 3363 CONNECT registry-1.docker.io:443 admin DIRECT/54.175.37.127 -
1466118758.838     94 10.8.0.147 TCP_MISS/200 3736 CONNECT registry-1.docker.io:443 admin DIRECT/54.175.37.127 -

Geoff, can you verify you configured the proxy via the installation routine?

Also, are you sure the proxy is allowing access to registry.access.redhat.com ?

I'm going to move this off of 6.2 blocking status as we can't reproduce it internally and proxy access appears to be working fine.

Comment 10 Andrew Schofield 2016-07-07 21:45:59 UTC
We are 6.2 Beta 2 on RHEL7 and are also seeing this:

# cat /etc/pulp/server/plugins.conf.d/docker_importer.json
{
        "proxy_host": <proxy server>,
        "proxy_port": 8080,
    "proxy_username": null,
    "proxy_password": null
}


# ls -lZ /etc/pulp/server/plugins.conf.d/docker_importer.json
-rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 /etc/pulp/server/plugins.conf.d/docker_importer.json

From production.log:

2016-07-07 17:11:29 [app] [I] Started POST "/registries" for 169.37.195.11 at 2016-07-07 17:11:29 -0400
2016-07-07 17:11:29 [app] [I] Processing by RegistriesController#create as HTML
2016-07-07 17:11:29 [app] [I]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"RydJrtLo9i1hFcDtJI2SxbcHT5kkHG2w77SjyC8z7wY=", "docker_registry"=>{"name"=>"registry.access.redhat.com", "url"=>"http://registry.access.redhat.com/", "description"=>"RedHat Docker registry", "username"=>"", "password"=>"[FILTERED]", "location_ids"=>[""], "organization_ids"=>["", "3"]}, "commit"=>"Submit"}
2016-07-07 17:11:30 [app] [I] Failed to save: Unable to log in to this Docker Registry - getaddrinfo: Name or service not known

Our installer was run as:

satellite-installer --scenario satellite --certs-server-cert <crt file> --certs-server-cert-req <csr file> --certs-server-key <key file> --certs-server-ca-cert <ca file> --katello-proxy-port 8080 --katello-proxy-url <proxy server> --certs-update-all

Comment 11 Karl Abbott 2016-07-18 17:51:27 UTC
Mike,

Any thoughts on what Andrew mentioned in comment 10?

Cheers,
Karl

Comment 12 Mike McCune 2016-09-02 17:14:23 UTC
Karl,

All our testing syncs content through a proxy internally so this should be working. Going to get more details from one of the testers who worked on this in 6.2 and get his input.

Roman,

Any comments on testing we did with 6.2 around syncing Docker content through a proxy? I'd expect the above to work.

Comment 13 Roman Plevka 2016-09-05 18:34:20 UTC
(In reply to Mike McCune from comment #12)
> Karl,
> 
> All our testing syncs content through a proxy internally so this should be
> working. Going to get more details from one of the testers who worked on
> this in 6.2 and get his input.
> 
> Roman,
> 
> Any comments on testing we did with 6.2 around syncing Docker content
> through a proxy? I'd expect the above to work.

Aren't we confusing 2 different things here?
There's foreman-docker plugin and then there's katello docker repository support. The latter uses pulp and the config mentioned in Comment #1

I doublechecked and docker registry-related requests don't go through the proxy,
however adding a docker repository with URL pointing to external registry does (on sync).

Comment 14 ruben 2016-11-30 13:53:42 UTC
Having this exact same issue on a new Satellite 6.2 (final) installation behind a proxy:

/root# rpm -qa | grep satellite
tfm-rubygem-foreman_theme_satellite-0.1.33-1.el7sat.noarch
satellite-cli-6.2.4-1.0.el7sat.noarch
satellite-6.2.4-1.0.el7sat.noarch
satellite-installer-6.2.0.12-1.el7sat.noarch


Exact error is:

Unable to save
Unable to log in to this Docker Registry - getaddrinfo: Name or service not known




When we add the hostname (registry.access.redhat.com) to the /etc/hosts file with the correct IP, error changes into:

Unable to save
Unable to log in to this Docker Registry - Connection timed out - connect(2) for "registry.access.redhat.com" port 443



This used to work fine on Satellite 6.1. Note that in earlier versions we did not have to specify the protocol (https) in the URL either, but now we do; I assume there were code-changes on this level for Satellite 6.2


tcpdump output at time of save without hosts entry:


/root# tcpdump -ni any port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
14:50:17.141487 IP 172.21.145.40.36555 > 172.22.4.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:17.141490 IP 172.21.145.40.36555 > 172.22.4.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:17.141493 IP 172.21.145.40.36555 > 172.22.4.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:17.141493 IP 172.21.145.40.36555 > 172.22.4.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:18.142548 IP 172.21.145.40.45622 > 172.22.2.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:18.142550 IP 172.21.145.40.45622 > 172.22.2.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:18.142553 IP 172.21.145.40.45622 > 172.22.2.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:18.142553 IP 172.21.145.40.45622 > 172.22.2.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:19.143569 IP 172.21.145.40.36555 > 172.22.4.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:19.143571 IP 172.21.145.40.36555 > 172.22.4.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:19.143574 IP 172.21.145.40.36555 > 172.22.4.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:19.143574 IP 172.21.145.40.36555 > 172.22.4.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:20.144376 IP 172.21.145.40.45622 > 172.22.2.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:20.144378 IP 172.21.145.40.45622 > 172.22.2.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:20.144380 IP 172.21.145.40.45622 > 172.22.2.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:20.144380 IP 172.21.145.40.45622 > 172.22.2.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)


Note that 172.21.145.40 is the Satellite server, 172.22.4.11 / 172.22.2.11 are internal DNS servers configured in /etc/resolv.conf. Expected behaviour is that this request goes through the HTTP proxy, rather than a direct connection to the host.


tcpdump output at time of save with hosts entry:


/root# tcpdump -ni any host 209.132.182.63
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
14:52:43.615013 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98678963 ecr 0,nop,wscale 7], length 0
14:52:43.615016 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98678963 ecr 0,nop,wscale 7], length 0
14:52:44.615366 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98679964 ecr 0,nop,wscale 7], length 0
14:52:44.615368 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98679964 ecr 0,nop,wscale 7], length 0
14:52:46.619367 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98681968 ecr 0,nop,wscale 7], length 0
14:52:46.619369 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98681968 ecr 0,nop,wscale 7], length 0

Again we see direct outgoing connections to the IP, rather than going through the configured proxy.

Comment 16 Michael Hrivnak 2016-12-12 21:16:51 UTC
I'm changing the component to Container Management since there does not appear to be a Pulp bug.

Comment 19 Tom McKay 2017-01-24 15:06:18 UTC
Created redmine issue http://projects.theforeman.org/issues/18217 from this bug

Comment 20 pm-sat@redhat.com 2017-01-24 17:09:09 UTC
Upstream bug assigned to paji@redhat.com

Comment 31 Matt Millard 2017-05-15 18:46:40 UTC
Tried applying the hotfix for 1333595 with no luck. Appears to be written against 6.3 instead of 6.2 as it is going after foreman_docker 3.0.0 and not foreman_docker 2.0.1.11 that is shown on my 6.2 Satellite.

[root@example satellite-hotfix-20170505-c09955e]# cd /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_docker-3.0.0
-bash: cd: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_docker-3.0.0: No such file or directory
[root@ul2647 satellite-hotfix-20170505-c09955e]# locate docker_registry.rb
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_docker-2.0.1.11/app/models/docker_registry.rb
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman_docker-0.0.6/lib/hammer_cli_foreman_docker/docker_registry.rb
[root@ul2647 satellite-hotfix-20170505-c09955e]# ./satellite-hotfix -n 1333595
Applying 1333595: Use proxy setting for registries

Applying 1333595: file 1333595-use-proxy-setting-for-registry-requests.patch
patch -fup1 --verbose -d  -r /root/satellite-hotfix-20170505-c09955e/backups/20170515-1840-337-1333595/1333595.rej --dry-run < /root/satellite-hotfix-20170505-c09955e/patches/1333595-use-proxy-setting-for-registry-requests.patch
patch: **** Can't change to directory -r : No such file or directory

ERROR APPLYING HOTFIX! Already applied? Try dry revert (-nr) to confirm.

Comment 41 Danial khatib 2017-08-04 09:36:39 UTC
Any progress on this issue ?

Comment 43 Sebastian Gräßl 2017-08-14 11:20:50 UTC
Currently there are two PRs in upstream in progress. One will allow to set a proxy for all outgoing HTTP requests[1], the second will allow to specify a HTTP proxy for a compute resources.

Once they are done and merged the configuration can happen in the UI and not require the hotfix or changes in foreman-docker.

[1] https://github.com/theforeman/foreman/pull/4640
[2] https://github.com/theforeman/foreman/pull/4616

Comment 47 Sebastian Gräßl 2017-10-12 09:53:42 UTC
There is now a feature in upstream[1] core to set a HTTP proxy for outgoing http and https requests. Documentation for this is also about to be merged into upstream docs[2].

[1] https://github.com/theforeman/foreman/pull/4640
[2] https://github.com/theforeman/theforeman.org/pull/948

Comment 49 Sean Mullen 2017-11-20 18:30:12 UTC
Regarding comment 31 by Matt Millard:
I'm having the same issue when trying to do a dry run on this hotfix.  The problem is not that ti's written against Satellite 6.3.

The "satellite-hotfix" command included in this hotfix is broken.  It generates the command line "patch -fup1 --verbose -d  -r ...".  In the patch command, -d = "Directory" and expects an argument.  Because nothing is being provided as an argument for -d, the patch command interprets the next argument (-r) to be the value associated with -d and as a result, patch attempts to cd into "-r".

My Results:
./satellite-hotfix -n 1333595
Applying 1333595: Use proxy setting for registries

Applying 1333595: file 1333595-use-proxy-setting-for-registry-requests.patch
patch -fup1 --verbose -d  -r /root/satellite-hotfix-20170505-c09955e/backups/20171120-1819-128-1333595/1333595.rej --dry-run < /root/satellite-hotfix-20170505-c09955e/patches/1333595-use-proxy-setting-for-registry-requests.patch
patch: **** Can't change to directory -r : No such file or directory

ERROR APPLYING HOTFIX! Already applied? Try dry revert (-nr) to confirm.

Comment 50 Sean Mullen 2017-11-20 18:44:01 UTC
Digging a bit more, there are more issues.  The exact error Matt mentions and that I see is a result of the bad command.  Looking at satellite-hotfix, the dir setting for this patch is /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_docker-3.0.0 which does not exist on my server.

This was Matt's point I believe ... I'd misread Matt's comment, he was referring to foreman-docker 3.0, not Satellite 6.3.

He's correct there, this was written for foreman-docker 3.0.0, not foreman_docker-2.0.1.11 which is installed on our systems.

Comment 51 Sean Mullen 2017-11-20 19:11:16 UTC
I manually added the changes to the applicable files and created /etc/foreman/settings.d/foreman-docker.yaml and things still did not work (note, I had to create the folder settings.d so I'm thinking that may have been added to 3.0.0 and ignored in 2.0.1.11).

I then put the proxy settings into /etc/foreman/settings.yaml and restarted HTTPD.  Once this was done, I was able to save the Red Hat Docker registry and the proxy was used as expected.

Comment 61 Patrick Creech 2018-05-22 15:24:38 UTC
This failed to cherrypick downstream, Sebastian can you take a look?  We are trying to get this in for 6.3.2

Comment 63 Patrick Creech 2018-05-24 18:59:58 UTC
I was able to pull someone in to help me get this picked

Comment 65 Lukas Pramuk 2018-05-31 10:22:28 UTC
FailedQA.

@satellite-6.3.1-3.el7sat.noarch (6.3.2 Snap1)
foreman-1.15.6.43-1.el7sat.noarch
tfm-rubygem-runcible-1.12.0.4-3.el7sat.noarch

In the middle of functionality testing I found there are bad strings for the new setting:


Nightly: 

HTTP(S) proxy		Sets a proxy for all outgoing HTTP connections.
HTTP(S) proxy except hosts 	Set hostnames to which requests are not to be proxied

vs. 

Satellite6.3.2:

HTTP(S) Capsule 	Sets a Capsule for all outgoing HTTP connections.
HTTP(S) Capsule except hosts 	Set hostnames to which requests are not to be proxied

>>> in downstream http proxy should still read the same "http proxy" instead of wrong "http capsule"

Comment 66 Lukas Pramuk 2018-05-31 13:26:52 UTC
OK. 

The bad strings is the different component therefore I filed BZ 1584706 against Branding and switching this BZ back to ONQA.

Comment 67 Lukas Pramuk 2018-06-01 09:11:54 UTC
VERIFIED.

@satellite-6.3.1-3.el7sat.noarch (6.3.2 Snap1)
foreman-1.15.6.43-1.el7sat.noarch
tfm-rubygem-runcible-1.12.0.4-3.el7sat.noarch

by the following reproducer/testing:

1) Set setting "HTTP(S) proxy" to authed proxy
 http://admin@redhat@proxy.example.com:3128

2) Create docker registry and see traffic hitting proxy

3) Set setting "HTTP(S) proxy" to unauthed proxy
 http://proxy.example.com:3401

4) Create docker registry and see traffic hitting unauthed proxy

5) Set setting "HTTP(S) proxy" to non-existing proxy
 http://nonononoproxy.example.com:3401

6) Try to create docker registry and see it failed
(x) Unable to save
Unable to log in to this Docker Registry - Proxied request failed with: getaddrinfo: Name or service not known (SocketError)

7) Check all other stuff is also using proxy (tries to use nonexisting proxy) since this is general http-proxy RFE (ie. not only registries)

*/compute_resources/1-http docker/edit
Proxied request failed with: getaddrinfo: Name or service not known (SocketError)

*/redhat_access/insights
Oops, we're sorry but something went wrong Failed to open TCP connection ...

>>> http-proxy is used for all requests

But even for candlepin/katello !!! (ultimate breakage) and even if you set valid proxy since request for candlepin (8443/tcp) cannot be proxied and are refused. Any katello page is throwing "403 Forbidden"

Unless you specify Satellite FQDN in "HTTP(S) proxy except hosts" which is really really tricky and wouldn't be obvoius to many CUs.

Filing new BZ 1585069 with this issue (as discussed with Dev and his Manager)

Comment 74 Lukas Pramuk 2018-08-06 14:19:04 UTC
RE-VERIFIED.

@satellite-6.3.3-1.el7sat.noarch (6.3.3 Snap1)
foreman-1.15.6.48-1.el7sat.noarch
tfm-rubygem-runcible-1.12.0.7-1.el7sat.noarch

by the following reproducer/testing:

1) Set setting "HTTP(S) proxy" to authed proxy
 http://admin@redhat@proxy.example.com:3128

2) Create docker registry and see traffic hitting proxy

3) Set setting "HTTP(S) proxy" to unauthed proxy
 http://proxy.example.com:3401

4) Create docker registry and see traffic hitting unauthed proxy

5) Set setting "HTTP(S) proxy" to non-existing proxy
 http://nonononoproxy.example.com:3401

6) Try to create docker registry and see it failed
(x) Unable to save
Unable to log in to this Docker Registry - Proxied request failed with: getaddrinfo: Name or service not known (SocketError)

7) Check all other stuff is also using proxy (tries to use nonexisting proxy) since this is general http-proxy RFE (ie. not only registries)

*/compute_resources/1-http docker/edit
Proxied request failed with: getaddrinfo: Name or service not known (SocketError)

*/redhat_access/insights
Oops, we're sorry but something went wrong Failed to open TCP connection ...

>>> http-proxy is used for all requests

Comment 77 errata-xmlrpc 2018-08-22 20:07:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2550


Note You need to log in before you can comment on or make changes to this bug.