Bug 1333595

I found /etc/pulp/server/plugins.conf.d/docker_importer.json  which DOES seem to contain the correct proxy information (identical to the yum_importer.json which does work...   however docker is definitely not using these proxy settings.

root@sat62[/etc/pulp/server/plugins.conf.d] # cat docker_importer.json
{
        "proxy_host": "http://proxy.example.org",

        "proxy_port": 8080,


    "proxy_username": null,


    "proxy_password": null

}

It works for me. I think there was an outage recently with registry.access.redhat.com, because I also had noticed the "no route to host" error. It seems to be working now.

I did this just now with pulp 2.8:

pulp-admin docker repo create --repo-id=rhel --feed=https://registry.access.redhat.com --upstream-name=rhel7 --enable-v2=false --enable-v1=true
pulp-admin docker repo sync run --repo-id=rhel


That worked both with a proxy specified in the json config, and without.

Using iftop, I verified that it definitely used the proxy and did not communicate directly with registry.access.redhat.com.

Perhaps there is a file permission issue preventing pulp from reading your customer's docker_importer.json config file?

I verified that Pulp, in Satellite 6.2 Beta 2 is using the proxy for docker syncs.

1) configured satellite to use a squid proxy I have in my lab:

 satellite-installer --katello-proxy-password redhat --katello-proxy-port 8888 --katello-proxy-url http://proxy.example.com --katello-proxy-username admin 

this properly configures all the importer configs in 

/etc/pulp/server/plugins.conf.d

docker_importer.json  iso_importer.json  puppet_importer.json  yum_importer.json


# cat docker_importer.json
{
        "proxy_host": "http://proxy.example.com",
    
        "proxy_port": 8888,
    
        "proxy_username": "admin",
    
        "proxy_password": "redhat"
    
}

you should never have to hand configure these files.

2) added an iptables rule to restrict access to only my http proxy host

3) Synced a Docker repo from docker hub: registry-1.docker.io

4) Verified access and logging in squid on the proxy itself:


1466118758.647     88 10.8.0.147 TCP_MISS/200 3363 CONNECT registry-1.docker.io:443 admin DIRECT/54.175.37.127 -
1466118758.838     94 10.8.0.147 TCP_MISS/200 3736 CONNECT registry-1.docker.io:443 admin DIRECT/54.175.37.127 -

Geoff, can you verify you configured the proxy via the installation routine?

Also, are you sure the proxy is allowing access to registry.access.redhat.com ?

I'm going to move this off of 6.2 blocking status as we can't reproduce it internally and proxy access appears to be working fine.

We are 6.2 Beta 2 on RHEL7 and are also seeing this:

# cat /etc/pulp/server/plugins.conf.d/docker_importer.json
{
        "proxy_host": <proxy server>,
        "proxy_port": 8080,
    "proxy_username": null,
    "proxy_password": null
}


# ls -lZ /etc/pulp/server/plugins.conf.d/docker_importer.json
-rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 /etc/pulp/server/plugins.conf.d/docker_importer.json

From production.log:

2016-07-07 17:11:29 [app] [I] Started POST "/registries" for 169.37.195.11 at 2016-07-07 17:11:29 -0400
2016-07-07 17:11:29 [app] [I] Processing by RegistriesController#create as HTML
2016-07-07 17:11:29 [app] [I]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"RydJrtLo9i1hFcDtJI2SxbcHT5kkHG2w77SjyC8z7wY=", "docker_registry"=>{"name"=>"registry.access.redhat.com", "url"=>"http://registry.access.redhat.com/", "description"=>"RedHat Docker registry", "username"=>"", "password"=>"[FILTERED]", "location_ids"=>[""], "organization_ids"=>["", "3"]}, "commit"=>"Submit"}
2016-07-07 17:11:30 [app] [I] Failed to save: Unable to log in to this Docker Registry - getaddrinfo: Name or service not known

Our installer was run as:

satellite-installer --scenario satellite --certs-server-cert <crt file> --certs-server-cert-req <csr file> --certs-server-key <key file> --certs-server-ca-cert <ca file> --katello-proxy-port 8080 --katello-proxy-url <proxy server> --certs-update-all

Mike,

Any thoughts on what Andrew mentioned in comment 10?

Cheers,
Karl

Karl,

All our testing syncs content through a proxy internally so this should be working. Going to get more details from one of the testers who worked on this in 6.2 and get his input.

Roman,

Any comments on testing we did with 6.2 around syncing Docker content through a proxy? I'd expect the above to work.

(In reply to Mike McCune from comment #12)
> Karl,
> 
> All our testing syncs content through a proxy internally so this should be
> working. Going to get more details from one of the testers who worked on
> this in 6.2 and get his input.
> 
> Roman,
> 
> Any comments on testing we did with 6.2 around syncing Docker content
> through a proxy? I'd expect the above to work.

Aren't we confusing 2 different things here?
There's foreman-docker plugin and then there's katello docker repository support. The latter uses pulp and the config mentioned in Comment #1

I doublechecked and docker registry-related requests don't go through the proxy,
however adding a docker repository with URL pointing to external registry does (on sync).

Having this exact same issue on a new Satellite 6.2 (final) installation behind a proxy:

/root# rpm -qa | grep satellite
tfm-rubygem-foreman_theme_satellite-0.1.33-1.el7sat.noarch
satellite-cli-6.2.4-1.0.el7sat.noarch
satellite-6.2.4-1.0.el7sat.noarch
satellite-installer-6.2.0.12-1.el7sat.noarch


Exact error is:

Unable to save
Unable to log in to this Docker Registry - getaddrinfo: Name or service not known




When we add the hostname (registry.access.redhat.com) to the /etc/hosts file with the correct IP, error changes into:

Unable to save
Unable to log in to this Docker Registry - Connection timed out - connect(2) for "registry.access.redhat.com" port 443



This used to work fine on Satellite 6.1. Note that in earlier versions we did not have to specify the protocol (https) in the URL either, but now we do; I assume there were code-changes on this level for Satellite 6.2


tcpdump output at time of save without hosts entry:


/root# tcpdump -ni any port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
14:50:17.141487 IP 172.21.145.40.36555 > 172.22.4.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:17.141490 IP 172.21.145.40.36555 > 172.22.4.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:17.141493 IP 172.21.145.40.36555 > 172.22.4.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:17.141493 IP 172.21.145.40.36555 > 172.22.4.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:18.142548 IP 172.21.145.40.45622 > 172.22.2.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:18.142550 IP 172.21.145.40.45622 > 172.22.2.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:18.142553 IP 172.21.145.40.45622 > 172.22.2.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:18.142553 IP 172.21.145.40.45622 > 172.22.2.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:19.143569 IP 172.21.145.40.36555 > 172.22.4.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:19.143571 IP 172.21.145.40.36555 > 172.22.4.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:19.143574 IP 172.21.145.40.36555 > 172.22.4.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:19.143574 IP 172.21.145.40.36555 > 172.22.4.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:20.144376 IP 172.21.145.40.45622 > 172.22.2.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:20.144378 IP 172.21.145.40.45622 > 172.22.2.11.domain: 4624+ A? registry.access.redhat.com. (44)
14:50:20.144380 IP 172.21.145.40.45622 > 172.22.2.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)
14:50:20.144380 IP 172.21.145.40.45622 > 172.22.2.11.domain: 53804+ AAAA? registry.access.redhat.com. (44)


Note that 172.21.145.40 is the Satellite server, 172.22.4.11 / 172.22.2.11 are internal DNS servers configured in /etc/resolv.conf. Expected behaviour is that this request goes through the HTTP proxy, rather than a direct connection to the host.


tcpdump output at time of save with hosts entry:


/root# tcpdump -ni any host 209.132.182.63
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
14:52:43.615013 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98678963 ecr 0,nop,wscale 7], length 0
14:52:43.615016 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98678963 ecr 0,nop,wscale 7], length 0
14:52:44.615366 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98679964 ecr 0,nop,wscale 7], length 0
14:52:44.615368 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98679964 ecr 0,nop,wscale 7], length 0
14:52:46.619367 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98681968 ecr 0,nop,wscale 7], length 0
14:52:46.619369 IP 172.21.145.40.56472 > 209.132.182.63.https: Flags [S], seq 3613553403, win 29200, options [mss 1460,sackOK,TS val 98681968 ecr 0,nop,wscale 7], length 0

Again we see direct outgoing connections to the IP, rather than going through the configured proxy.

I'm changing the component to Container Management since there does not appear to be a Pulp bug.

Created redmine issue http://projects.theforeman.org/issues/18217 from this bug

Upstream bug assigned to paji

Tried applying the hotfix for 1333595 with no luck. Appears to be written against 6.3 instead of 6.2 as it is going after foreman_docker 3.0.0 and not foreman_docker 2.0.1.11 that is shown on my 6.2 Satellite.

[root@example satellite-hotfix-20170505-c09955e]# cd /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_docker-3.0.0
-bash: cd: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_docker-3.0.0: No such file or directory
[root@ul2647 satellite-hotfix-20170505-c09955e]# locate docker_registry.rb
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_docker-2.0.1.11/app/models/docker_registry.rb
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman_docker-0.0.6/lib/hammer_cli_foreman_docker/docker_registry.rb
[root@ul2647 satellite-hotfix-20170505-c09955e]# ./satellite-hotfix -n 1333595
Applying 1333595: Use proxy setting for registries

Applying 1333595: file 1333595-use-proxy-setting-for-registry-requests.patch
patch -fup1 --verbose -d  -r /root/satellite-hotfix-20170505-c09955e/backups/20170515-1840-337-1333595/1333595.rej --dry-run < /root/satellite-hotfix-20170505-c09955e/patches/1333595-use-proxy-setting-for-registry-requests.patch
patch: **** Can't change to directory -r : No such file or directory

ERROR APPLYING HOTFIX! Already applied? Try dry revert (-nr) to confirm.

Any progress on this issue ?

Currently there are two PRs in upstream in progress. One will allow to set a proxy for all outgoing HTTP requests[1], the second will allow to specify a HTTP proxy for a compute resources.

Once they are done and merged the configuration can happen in the UI and not require the hotfix or changes in foreman-docker.

[1] https://github.com/theforeman/foreman/pull/4640
[2] https://github.com/theforeman/foreman/pull/4616

There is now a feature in upstream[1] core to set a HTTP proxy for outgoing http and https requests. Documentation for this is also about to be merged into upstream docs[2].

[1] https://github.com/theforeman/foreman/pull/4640
[2] https://github.com/theforeman/theforeman.org/pull/948

Regarding comment 31 by Matt Millard:
I'm having the same issue when trying to do a dry run on this hotfix.  The problem is not that ti's written against Satellite 6.3.

The "satellite-hotfix" command included in this hotfix is broken.  It generates the command line "patch -fup1 --verbose -d  -r ...".  In the patch command, -d = "Directory" and expects an argument.  Because nothing is being provided as an argument for -d, the patch command interprets the next argument (-r) to be the value associated with -d and as a result, patch attempts to cd into "-r".

My Results:
./satellite-hotfix -n 1333595
Applying 1333595: Use proxy setting for registries

Applying 1333595: file 1333595-use-proxy-setting-for-registry-requests.patch
patch -fup1 --verbose -d  -r /root/satellite-hotfix-20170505-c09955e/backups/20171120-1819-128-1333595/1333595.rej --dry-run < /root/satellite-hotfix-20170505-c09955e/patches/1333595-use-proxy-setting-for-registry-requests.patch
patch: **** Can't change to directory -r : No such file or directory

ERROR APPLYING HOTFIX! Already applied? Try dry revert (-nr) to confirm.

Digging a bit more, there are more issues.  The exact error Matt mentions and that I see is a result of the bad command.  Looking at satellite-hotfix, the dir setting for this patch is /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_docker-3.0.0 which does not exist on my server.

This was Matt's point I believe ... I'd misread Matt's comment, he was referring to foreman-docker 3.0, not Satellite 6.3.

He's correct there, this was written for foreman-docker 3.0.0, not foreman_docker-2.0.1.11 which is installed on our systems.

I manually added the changes to the applicable files and created /etc/foreman/settings.d/foreman-docker.yaml and things still did not work (note, I had to create the folder settings.d so I'm thinking that may have been added to 3.0.0 and ignored in 2.0.1.11).

I then put the proxy settings into /etc/foreman/settings.yaml and restarted HTTPD.  Once this was done, I was able to save the Red Hat Docker registry and the proxy was used as expected.

This failed to cherrypick downstream, Sebastian can you take a look?  We are trying to get this in for 6.3.2

I was able to pull someone in to help me get this picked

FailedQA.

@satellite-6.3.1-3.el7sat.noarch (6.3.2 Snap1)
foreman-1.15.6.43-1.el7sat.noarch
tfm-rubygem-runcible-1.12.0.4-3.el7sat.noarch

In the middle of functionality testing I found there are bad strings for the new setting:


Nightly: 

HTTP(S) proxy		Sets a proxy for all outgoing HTTP connections.
HTTP(S) proxy except hosts 	Set hostnames to which requests are not to be proxied

vs. 

Satellite6.3.2:

HTTP(S) Capsule 	Sets a Capsule for all outgoing HTTP connections.
HTTP(S) Capsule except hosts 	Set hostnames to which requests are not to be proxied

>>> in downstream http proxy should still read the same "http proxy" instead of wrong "http capsule"

OK. 

The bad strings is the different component therefore I filed BZ 1584706 against Branding and switching this BZ back to ONQA.

VERIFIED.

@satellite-6.3.1-3.el7sat.noarch (6.3.2 Snap1)
foreman-1.15.6.43-1.el7sat.noarch
tfm-rubygem-runcible-1.12.0.4-3.el7sat.noarch

by the following reproducer/testing:

1) Set setting "HTTP(S) proxy" to authed proxy
 http://admin@redhat@proxy.example.com:3128

2) Create docker registry and see traffic hitting proxy

3) Set setting "HTTP(S) proxy" to unauthed proxy
 http://proxy.example.com:3401

4) Create docker registry and see traffic hitting unauthed proxy

5) Set setting "HTTP(S) proxy" to non-existing proxy
 http://nonononoproxy.example.com:3401

6) Try to create docker registry and see it failed
(x) Unable to save
Unable to log in to this Docker Registry - Proxied request failed with: getaddrinfo: Name or service not known (SocketError)

7) Check all other stuff is also using proxy (tries to use nonexisting proxy) since this is general http-proxy RFE (ie. not only registries)

*/compute_resources/1-http docker/edit
Proxied request failed with: getaddrinfo: Name or service not known (SocketError)

*/redhat_access/insights
Oops, we're sorry but something went wrong Failed to open TCP connection ...

>>> http-proxy is used for all requests

But even for candlepin/katello !!! (ultimate breakage) and even if you set valid proxy since request for candlepin (8443/tcp) cannot be proxied and are refused. Any katello page is throwing "403 Forbidden"

Unless you specify Satellite FQDN in "HTTP(S) proxy except hosts" which is really really tricky and wouldn't be obvoius to many CUs.

Filing new BZ 1585069 with this issue (as discussed with Dev and his Manager)

RE-VERIFIED.

@satellite-6.3.3-1.el7sat.noarch (6.3.3 Snap1)
foreman-1.15.6.48-1.el7sat.noarch
tfm-rubygem-runcible-1.12.0.7-1.el7sat.noarch

by the following reproducer/testing:

1) Set setting "HTTP(S) proxy" to authed proxy
 http://admin@redhat@proxy.example.com:3128

2) Create docker registry and see traffic hitting proxy

3) Set setting "HTTP(S) proxy" to unauthed proxy
 http://proxy.example.com:3401

4) Create docker registry and see traffic hitting unauthed proxy

5) Set setting "HTTP(S) proxy" to non-existing proxy
 http://nonononoproxy.example.com:3401

6) Try to create docker registry and see it failed
(x) Unable to save
Unable to log in to this Docker Registry - Proxied request failed with: getaddrinfo: Name or service not known (SocketError)

7) Check all other stuff is also using proxy (tries to use nonexisting proxy) since this is general http-proxy RFE (ie. not only registries)

*/compute_resources/1-http docker/edit
Proxied request failed with: getaddrinfo: Name or service not known (SocketError)

*/redhat_access/insights
Oops, we're sorry but something went wrong Failed to open TCP connection ...

>>> http-proxy is used for all requests

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2550

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days