Description of problem: Syncing authenticated repos fail with RPM1004: Error retrieving metadata: Not found this is now happening on both authenticated and unauth http proxy According to squid logs it looks like the credentials (for the repo) are not even being sent to it. After disabling the proxy, sync works just fine. This is a regression since this used to work in previous snaps Version-Release number of selected component (if applicable): 6.2.0 snap 8.2 How reproducible: always Steps to Reproduce: 1. configure satellite to use http proxy 2. configure a custom http authenticated repo 3. sync it Actual results: RPM1004: Error retrieving metadata: Not found Expected results: Sync works Additional info: attaching pulp logs, production log and squid access logs
Created attachment 1148199 [details] squid access log and conf
Created attachment 1148200 [details] pulp syslog messages
Created attachment 1148201 [details] foreman/production.log
Roman, can we get access to the environment where this is occuring? Is this for custom, RH or any type of content? Thanks!
Michael, Any thoughts on this? Looking pulp related to me.
It may be this fix from upstream: https://github.com/pulp/nectar/commit/1cd4eedb82fda41935e2596bd99e2839df0e8a0d
I'm not sure it is because roman says its a problem even without an authenticated proxy 'this is now happening on both authenticated and unauth http proxy'
Created attachment 1161066 [details] pulp syslog This is the part of the /var/log/messages where the failure occurs. It contains the traceback. Looks like the credentials are not being passed to the server in authorization header.
Upstream nightly fails on an authenticated proxy, but works properly on an unauthenticated proxy.
I applied that nectar diff [1] to a Satellite6 installation (SNAP GA 13.1) and it still fails on authenticated proxies. [1] https://github.com/pulp/nectar/commit/1cd4eedb82fda41935e2596bd99e2839df0e8a0d
I can reproduce the failed sync on a basic auth repo even without Satellite6 being behind a proxy, which makes me think this issue doesn't have to do with Satellite6 being behind a proxy, but rather then communication between Katello and Pulp when Satellite6 is creating a repository with basic auth.
Welp, I believe I misspoke in comment 14. Satellite6 is able to sync a basicauth repo without a proxy successfully.
Could this be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1316229 ? What version of python-nectar is installed when you see this issue?
(In reply to Michael Hrivnak from comment #16) > Could this be a duplicate of > https://bugzilla.redhat.com/show_bug.cgi?id=1316229 ? > > What version of python-nectar is installed when you see this issue? Tha BZ has been created by me as well but only applied to Authed proxy (non-auth proxy setup worked back then) - That's why I opened a new one.
(In reply to Michael Hrivnak from comment #16) > Could this be a duplicate of > https://bugzilla.redhat.com/show_bug.cgi?id=1316229 ? > > What version of python-nectar is installed when you see this issue? python-nectar-1.5.1-3.el7sat.noarch
The following works for me on pulp 2.8.4 beta 2: I put the same exact config for the authenticated proxy into /etc/pulp/server/plugins.conf.d/yum_importer.json I restarted pulp workers so they would pick up the new config. pulp-admin rpm repo create --repo-id=rplevka --feed=https://rplevka.fedorapeople.org/fakerepo01/ --basicauth-user=admin --basicauth-pass=changeme pulp-admin rpm repo sync run --repo-id=rplevka I verified that it was indeed using the proxy, and I saw the sync succeed. I think the difference is that katello is putting the repo credentials in the URL instead of using the settings. I tried that, and it failed as described in this bug. I'll investigate.
My testing shows that the only case that fails is when all of these are true: - the proxy requires auth - the repo requires auth - repo auth credentials are specified in the URL All three of those must be true in order to see the sync fail with the "not found" error.
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
(partly) VERIFIED [root@bkr-hv02-guest27 ~]# hammer -u admin -p changeme repository create --name foo --product foo --organization-id 1 --content-type yum --url "https://admin:changeme@rplevka.fedorapeople.org/fakerepo01/" Repository created [root@bkr-hv02-guest27 ~]# [root@bkr-hv02-guest27 ~]# hammer repository list --organization-id 1 [Foreman] Password for admin: ---|------|---------|--------------|------------------------------------------------------------ ID | NAME | PRODUCT | CONTENT TYPE | URL ---|------|---------|--------------|------------------------------------------------------------ 1 | foo | foo | yum | https://admin:changeme@rplevka.fedorapeople.org/fakerepo01/ ---|------|---------|--------------|------------------------------------------------------------ [root@bkr-hv02-guest27 ~]# [root@bkr-hv02-guest27 ~]# hammer -u admin -p changeme repository synchronize --id 1 [....................................................................................] [100%] tthere is a problem though with credentials using unsafe chars that need to be urlencoded. e.g. https://%40dmin:changeme@rplevka.fedorapeople.org/fakerepo01/ - this used to work before introducing this bug. Am i supposed to open a new BZ or can we consider this as a regression?
> Am i supposed to open a new BZ or can we consider this as a regression? I'd consider that a new bug.
Sorry but further checks revealed a regression: the urlencoded special chars (especially @) no longer work even when not using a http proxy. This might cause regressions for customers who have been using auth repos with special characters in the credentials (which is quite common). # satellite-installer --katello-proxy-url "" Installing Done [100%] [.....................................] Success! * Satellite is running at https://***.com * To install additional capsule on separate machine continue by running: capsule-certs-generate --capsule-fqdn "$CAPSULE" --certs-tar "~/$CAPSULE-certs.tar" The full log is at /var/log/foreman-installer/satellite.log # hammer -u admin -p changeme repository info --id 2 | grep URL URL: https://%40dmin:changeme@rplevka.fedorapeople.org/fakerepo01/ # hammer repository synchronize --id 2 [Foreman] Password for admin: [.................................................................................................................................................................] [100%] No new packages. Error: RPM1004: Error retrieving metadata: Unauthorized
Roman, Logs and/or stack traces would greatly help me in assisting you with this new possibly related issue.
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
VERIFIED on sat6.2.7-2 by automation (https://github.com/SatelliteQE/robottelo/blob/master/tests/foreman/cli/test_repository.py#L198) $ pytest -k test_positive_synchronize_auth_puppet_repo test_repository.py === test session starts === platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.32, pluggy-0.4.0 rootdir: /home/rplevka/work/rplevka/robottelo, inifile: plugins: xdist-1.14 collected 65 items test_repository.py . === 64 tests deselected === === 1 passed, 64 deselected in 232.05 seconds ===
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0197