Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1639676 - Unable to persistently set redirect_host for lazy sync to empty value
Summary: Unable to persistently set redirect_host for lazy sync to empty value
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.3.4
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: 6.5.0
Assignee: Evgeni Golov
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-10-16 11:21 UTC by Pavel Moravec
Modified: 2022-03-13 15:47 UTC (History)
10 users (show)

Fixed In Version: katello-installer-base-3.9.0-0, pulp-2.18.0-0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-14 12:38:12 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
script to create a test repository for this BZ (580 bytes, application/x-shellscript)
2019-02-20 11:05 UTC, Evgeni Golov 		no flags Details
test output from Satellite 6.3 (so before this fix) (3.15 KB, text/plain)
2019-02-20 11:06 UTC, Evgeni Golov 		no flags Details
test output from Satellite 6.5 (with this fix) (3.77 KB, text/plain)
2019-02-20 11:06 UTC, Evgeni Golov 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 25266 0 Normal Closed installer should not set pulp's redirect_host to $::fqdn 2021-02-01 17:12:45 UTC
Pulp Redmine 4092 0 Normal CLOSED - CURRENTRELEASE redirect_host defaults to socket.getfqdn() instead of None 2018-12-04 22:30:41 UTC
Red Hat Knowledge Base (Solution) 3655581 0 None None None 2018-10-16 15:26:14 UTC
Red Hat Product Errata RHSA-2019:1222 0 None None None 2019-05-14 12:38:19 UTC

Description Pavel Moravec 2018-10-16 11:21:08 UTC 
Description of problem:
Assume a use case where multi-homed Satellite or Capsule needs to be accessed from clients via _different_ alternative hostnames, like caps-altname1 and caps-altname2 and neither is primary FQDN of the Capsule.

Then there is no way to permanently configure lazy sync.

(I expect the "multi-home clients" use case is just one of few similar ones here, like HA capsules + multihomed Caps is another use case)

The cause is, httpd responds with redirect URL (to contact squid) with hostname specified in /etc/pulp/server.conf, [lazy] section, redirect_host property. This property is basically required to have multiple values, or from the code, it must be empty (such that hostname from the request is used directly).


BUT redirect_host has hardcoded default value:

/usr/lib/python2.7/site-packages/pulp/server/config.py :

    'lazy': {
        'redirect_host': socket.getfqdn(),

So the workaround is to overwrite it to empty value in /etc/pulp/server.conf :

[lazy]
redirect_host:


BUT that will be overwritten by satellite-installer. Making the change permanent via 

pulp::lazy_redirect_host: ''

in /etc/foreman-installer/custom-hiera.yaml is _not_ a solution, since it purges away the "redirect_host: " line completely, hence pulp fallbacks to default primary hostname again.

So I dont see a way how to make the config change permanent - either:
- pulp default value for redirect_host must be empty
- or lazy_redirect_host with empty value in custom hiera must set empty redirect_host in /etc/pulp/server.conf


Version-Release number of selected component (if applicable):
Sat 6.3.4


How reproducible:
100%


Steps to Reproduce:
1. Try to configure Capsule to be accessed from 2 clients, each accessing the Capsule via different alternate hostname each.
2. run satellite-installer (to verify the config is persistent)
3. Try to fetch a package from lazy sync repo


Actual results:
3. fails for either or both clients


Expected results:
3. works for both clients concurrently


Additional info:
kudos for egolotov and ewound for some background info here
Comment 1 Pavel Moravec 2018-10-16 12:49:18 UTC 
ewoud++ for the solution:

line

pulp::lazy_redirect_host: ' '

in /etc/foreman-installer/custom-hiera.yaml is the trick to have in /etc/pulp/server.conf :

redirect_host: <empty-spaces-here>

that sets the required setting.
Comment 2 Pavel Moravec 2018-10-16 15:26:14 UTC 
(In reply to Pavel Moravec from comment #1)
> ewoud++ for the solution:
> 
> line
> 
> pulp::lazy_redirect_host: ' '
> 
> in /etc/foreman-installer/custom-hiera.yaml is the trick to have in
> /etc/pulp/server.conf :
> 
> redirect_host: <empty-spaces-here>
> 
> that sets the required setting.

So it is just configuration issue and not a bug. Described in KCS 3655581.
Comment 3 Satellite Program 2018-10-22 10:08:19 UTC 
Upstream bug assigned to egolov
Comment 4 Satellite Program 2018-10-22 10:08:21 UTC 
Upstream bug assigned to egolov
Comment 5 pulp-infra@redhat.com 2018-10-22 12:01:49 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 6 pulp-infra@redhat.com 2018-10-22 12:01:52 UTC 
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 7 pulp-infra@redhat.com 2018-10-23 15:31:37 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 9 Satellite Program 2018-10-31 14:08:28 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25266 has been resolved.
Comment 10 pulp-infra@redhat.com 2018-11-20 19:31:51 UTC 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 12 pulp-infra@redhat.com 2018-12-04 22:30:42 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 13 jcallaha 2019-01-17 22:07:14 UTC 
Failed QA in Satellite 6.5.0 Snap 11

Modified the config, performed an upgrade, then checked the config again. The value was stripped.

-bash-4.2# grep redirect_host /etc/pulp/server.conf 
# redirect_host:
redirect_host: my.test.host
-bash-4.2# 
-bash-4.2# satellite-installer --upgrade
Resetting puppet server version param...
Upgrading, to monitor the progress on all related services, please do:
  foreman-tail | tee upgrade-$(date +%Y-%m-%d-%H%M).log
Upgrade Step: stop_services...
Redirecting to 'foreman-maintain service'
Running Stop Services
================================================================================
...
Upgrade Step 7/7: katello:upgrades:3.11:update_puppet_repos. foreman-rake upgrade:run finished successfully!
Upgrade completed!
-bash-4.2# grep redirect_host /etc/pulp/server.conf 
# redirect_host:
Comment 14 pulp-infra@redhat.com 2019-01-17 22:31:23 UTC 
Requesting needsinfo from upstream developer dkliban, ttereshc, daviddavis because the 'FailedQA' flag is set.
Comment 15 David Davis 2019-01-21 14:03:02 UTC 
I think this is a problem in the installer. Setting NEEDINFO to egolov.
Comment 16 pulp-infra@redhat.com 2019-01-21 14:33:06 UTC 
Requesting needsinfo from upstream developer dkliban, ttereshc, daviddavis because the 'FailedQA' flag is set.
Comment 17 Evgeni Golov 2019-01-24 07:55:24 UTC 
Jake,

I think the initial BZ description lead you to a wrong verification path ;)

In Pulp < 2.18, we had to explicitly set "redirect_host" to an empty string to make it redirect using the HTTP Host header value (instead of the value of "redirect_host").
With Pulp 2.18, the redirect_host setting behaves "correctly" and does not need to be set to an empty string to trigger the desired behavior.

See Pulp changes
 https://pulp.plan.io/issues/4092
and
 https://pulp.plan.io/issues/4120

After these were in, we updated the installer in https://projects.theforeman.org/issues/25266.

So the correct verification steps would be:
1/ setup a on_demand repo
2/ try to access a (not yet downloaded) file in that repo with a hostname that points at the satellite, but is not the satellites FQDN (using /etc/hosts or something, using HTTP instead of HTTPS would make things easier, as you don't have to care for proper certs in that case
3/ see pulp redirect to the streamer at http://the-name-you-used.example.com/pulp/streamer/… and not http://satellite.example.com/pulp/streamer/…
Comment 18 pulp-infra@redhat.com 2019-01-24 08:01:45 UTC 
Requesting needsinfo from upstream developer dkliban, ttereshc, daviddavis because the 'FailedQA' flag is set.
Comment 19 Evgeni Golov 2019-02-20 11:05:39 UTC 
Created attachment 1536645 [details]
script to create a test repository for this BZ
Comment 20 Evgeni Golov 2019-02-20 11:06:15 UTC 
Created attachment 1536646 [details]
test output from Satellite 6.3 (so before this fix)
Comment 21 Evgeni Golov 2019-02-20 11:06:41 UTC 
Created attachment 1536647 [details]
test output from Satellite 6.5 (with this fix)
Comment 22 Evgeni Golov 2019-02-20 11:15:08 UTC 
To test this, you need a recent Katello and Pulp, e.g. as in the Satellite 6.5 snaps we ship.

This BZ has a script attached, that will create and sync a test-repository with on_demand policy:

# bash create-test-repo.sh 
Organization created.
Product created.
Repository created.
New packages: 32 (76.7 KB).

After the repository is synced, you can try to access its content using curl:
curl -I -X GET --location http://<SOME_NAME>/pulp/repos/Test_Organization/Library/custom/Test_Product/Zoo/Packages/b/bear-4.1-1.noarch.rpm
(we only want to see headers, so -I -X GET. we need curl to follow redirects, so --location)

An example output for Satellite 6.3 (broken) and Satellite 6.5 (working) is attached.

On Satellite 6.5 you can substitute anything for <SOME_NAME> that will reach your apache. I've used localhost and $HOSTNAME. In both cases you see a "Location: http://<SOME_NAME>:80/streamer/…" header, followed by a successful RPM download.
On Satellite 6.3 (and 6.4, but I don't have one handy), you'd see that the Location header *always* includes the $HOSTNAME of the machine, not what you've supplied in the original request.

Note: if you want to test this with protected (Red Hat) repositories via HTTPS, you'll have to tell curl to use the entitlement certificates of your machine in the request, as otherwise Pulp will not allow you to access the repositories.

Note2: Pulp will move the streamed RPMs to their proper location at some point, so you might not see the redirect again if you retry the same RPM at a later point. Just pick another one in that case (a listing can be found at https://jlsherrill.fedorapeople.org/fake-repos/needed-errata/)
Comment 23 jcallaha 2019-02-20 15:38:17 UTC 
Verified in Satellite 6.5.0 Snap 15

Followed the verification steps outlined in #17 and #22

First, setup the fake redirect host (note that you only want to have the domain name

-bash-4.2# grep "my.sat.host" /etc/pulp/server.conf
redirect_host: my.sat.host

Second, I added an entry to my client's hosts file for the fake domain, linking back to my satellite's IP

After that, I downloaded a package. 

[root@testhost2 ~]# curl -I -X GET --location http://<my actual satellite>/pulp/repos/Default_Organization/Library/custom/custom/fake/Packages/a/Antelope-10.6.9-1.elfake.noarch.rpm
HTTP/1.1 302 Found
Date: Wed, 20 Feb 2019 15:34:13 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux)
Content-Length: 0
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Location: http://my.sat.host:80/streamer/var/lib/pulp/content/units/rpm/60/bcb129a6ca5274adfd35721c1116069594579c45e86616f4c58ed1ad01fcd1/Antelope-10.6.9-1.elfake.noarch.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTAuMTMuMTI5LjEyNyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS82MC9iY2IxMjlhNmNhNTI3NGFkZmQzNTcyMWMxMTE2MDY5NTk0NTc5YzQ1ZTg2NjE2ZjRjNThlZDFhZDAxZmNkMS9BbnRlbG9wZS0xMC42LjktMS5lbGZha2Uubm9hcmNoLnJwbSIsICJleHBpcmF0aW9uIjogMTU1MDY3Njk0M30%3D;signature=ehgBDmxhTPzCcRGhmykK149x3Xq0R8EUAa9R-8gbGXe34JCVYjIWewYwfN4FgN6iEc40P9KtoQZVrLwUs872gS9ZzyU_1TIYXjYNO1B2OUEZGeZT59CPjResY8vxrbTeFm_-reXjGA3_LaP0unhA14XyczmAOxbzkbfPPb1aJXvoE7sK7dWuv2I-6T7Y1_9Sw9EZ1qCnPNqYzK1ehSA5ggWyN9_7LouH1RwDRpPXd5mjW1h6LO-DJW4dpmM-ng7yl2mocVj85gF1BdXCX3zEe5-48lhInlUL16Ckn9CtOlZ8kccEyPAfq77JRMfwDsX4v3WKYayLGo4We0S7RbFohQ%3D%3D
Content-Type: text/html; charset=utf-8

HTTP/1.1 200 OK
Date: Wed, 20 Feb 2019 15:34:14 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips
Apptime: D=5309
Content-Length: 13032
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Gitproject: (null)
Last-Modified: Sat, 08 Sep 2012 20:54:47 GMT
Accept-Ranges: bytes
Expires: Wed, 20 Feb 2019 16:04:14 GMT
Appserver: people02.fedoraproject.org
ETag: "32e8-4c936ef4c13c0"
Cache-Control: public, s-maxage=86400, max-age=86400
Content-Type: application/x-rpm
X-Cache: MISS from <my actual satellite>
X-Cache-Lookup: MISS from <my actual satellite>:3128
Via: 1.1 <my actual satellite> (squid/3.5.20)
Via: 1.1 <my actual satellite>


The redirect completed successfully
Comment 26 errata-xmlrpc 2019-05-14 12:38:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222
Note You need to log in before you can comment on or make changes to this bug.