Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1893007

Summary: After Satellite 6.8 upgrade, systems assigned to custom content view are getting 404 errors while downloading repodata. baseurl contain " %2F " in it
Product: Red Hat Satellite Reporter: Ashish Humbe <ahumbe>
Component: CandlepinAssignee: Barnaby Court <bcourt>
Status: CLOSED ERRATA QA Contact: Lai <ltran>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.8.0CC: amarirom, hakon.gislason, jbhatia, jjansky, jsherril, ktordeur, ldelouw, mcasabur, mlinden, mmccune, naijain, ofalk, patalber, pcfe, pcreech, pdudley, pdwyer, pgiustin, rankumar, rsebasti, saydas, sraut, swachira, yferszt
Target Milestone: 6.8.1Keywords: PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: candlepin-3.1.22-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1893836 1894035 (view as bug list) Environment:
Last Closed: 2020-11-11 14:19:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1893836    
Bug Blocks: 1894035    

Description Ashish Humbe 2020-10-29 20:56:10 UTC
Description of problem:
After Satellite 6.8 upgrade, systems assigned to custom content view are getting 404 errors while downloading repodata, because the repo URL generated by candlepin contain "%2F" in it.

yum fails with errors like: 

# yum update
Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager
https://satellite.example.com/pulp/repos/Org1/Prod%2FRHEL7-Capsule/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.


cat /var/log/rhsm/rhsm.log
    2020-10-29 10:53:24,073 [ERROR] subscription-manager:15121:MainThread @cache.py:181 - Error updating system data on the server
2020-10-29 10:53:24,073 [ERROR] subscription-manager:15121:MainThread @cache.py:182 - '%' must be followed by '%' or '(', found: '%2FRHEL7-2020-10/content/dist/rhel/server/7/7Server/$basearch/sat-tools/6.7/os'
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cache.py", line 173, in update_check
    self._sync_with_server(uep, consumer_uuid)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cache.py", line 493, in _sync_with_server
    combined_profile = self.current_profile
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cache.py", line 445, in current_profile
    enabled_repos = get_profile('enabled_repos').collect()
  File "/usr/lib64/python2.7/site-packages/rhsm/profile.py", line 357, in get_profile
    profile = PROFILE_MAP[profile_type]()
  File "/usr/lib64/python2.7/site-packages/rhsm/profile.py", line 195, in __init__
    self._enabled_repos = EnabledRepos(repo_file)
  File "/usr/lib64/python2.7/site-packages/rhsm/profile.py", line 153, in __init__
    self.content = self.__generate()
  File "/usr/lib64/python2.7/site-packages/rhsm/profile.py", line 135, in __generate
    "baseurl": [self._replace_vars(config.get(section, "baseurl"))]
  File "/usr/lib/python2.7/site-packages/iniparse/compat.py", line 226, in get
    return self._interpolate(section, option, value, d)
  File "/usr/lib/python2.7/site-packages/iniparse/compat.py", line 302, in _interpolate
    self._interpolate_some(option, L, rawval, section, vars, 1)
  File "/usr/lib/python2.7/site-packages/iniparse/compat.py", line 343, in _interpolate_some
    "'%' must be followed by '%' or '(', found: " + repr(rest))
InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%2FRHEL7-2020-10/content/dist/rhel/server/7/7Server/$basearch/sat-tools/6.7/os'


Note: this issue is not seen on systems using "Default Organization View". 

Version-Release number of selected component (if applicable):
Satellite 6.8

How reproducible:
Always

Steps to Reproduce:
1. Upgrade to Satellite 6.8
2. Run below commands on client system with custom CV assigned to it.

# subscription-manager refresh

# yum repolist 

# grep baseurl /etc/yum.repos.d/redhat.repo | sort



Actual results:

/etc/yum.repos.d/redhat.com file contain baseurl with %2F in it 

baseurl = https://satellite.example.com/pulp/repos/Org1/Prod%2FRHEL7-Capsule/content/dist/rhel/server/7/$releasever/$basearch/optional/os
baseurl = https://satellite.example.com/pulp/repos/Org1/Prod%2FRHEL7-Capsule/content/dist/rhel/server/7/$releasever/$basearch/os
baseurl = https://satellite.example.com/pulp/repos/Org1/Prod%2FRHEL7-Capsule/content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/os
baseurl = https://satellite.example.com/pulp/repos/Org1/Prod%2FRHEL7-Capsule/content/dist/rhel/server/7/7Server/$basearch/ansible/2.8/os
baseurl = https://satellite.example.com/pulp/repos/Org1/Prod%2FRHEL7-Capsule/content/dist/rhel/server/7/7Server/$basearch/extras/os


Content of the entitlement certificate looks like: 

Content:
        Type: yum
        Name: Red Hat Enterprise Linux 7 Server (RPMs)
        Label: rhel-7-server-rpms
        Vendor: Red Hat
        URL: /Org1/Prod%2FRHEL7-Capsule/content/dist/rhel/server/7/$releasever/$basearch/os
        GPG: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
        Enabled: True
        Expires: 1
        Required Tags: rhel-7-server
        Arches: x86_64

Comment 1 Paul Dudley 2020-10-29 21:04:39 UTC
*** Bug 1893004 has been marked as a duplicate of this bug. ***

Comment 9 Justin Sherrill 2020-10-30 14:29:58 UTC
We worked with a customer and have added an additional step to the workaround (step 1 in this case):


1. Edit /etc/pulp/repo_auth.conf  change:

enabled: true

to

enabled: false


2.  Create a file  /etc/httpd/conf.d/05-foreman-ssl.d/rewrite.conf
with contents:

AllowEncodedSlashes NoDecode
RewriteEngine On
RewriteRule    "^/pulp/repos/(.*)%2F(.*)"  "/pulp/repos/$1/$2" [R]


3.  restart httpd:  systemctl restart httpd


Please keep in mind that anytime you re-run the installer you will need to reapply these changes.  This is a temporary workaround until we have a more permanent fix.

Comment 10 Justin Sherrill 2020-10-30 15:36:41 UTC
Note: we are working on capsule workaround instructions now.

Comment 11 Justin Sherrill 2020-10-30 16:30:26 UTC
On capsules, perform these changes to workaround the problem: 


1.  Edit /etc/pulp/repo_auth.conf change:

enabled: true

to

enabled: false

2.   Edit the file /etc/httpd/conf.d/05-pulp-https.conf and find these two lines:

alias /pub /var/www/html/pub

<Location /pub>

Between these two lines, insert the following:

####### BZ 1893007 workaround
AllowEncodedSlashes NoDecode
RewriteEngine On
RewriteRule "^/pulp/repos/(.*)%2F(.*)" "/pulp/repos/$1/$2" [R]
#######

So the final result will look like:

alias /pub /var/www/html/pub

####### BZ 1893007 workaround
AllowEncodedSlashes NoDecode
RewriteEngine On
RewriteRule "^/pulp/repos/(.*)%2F(.*)" "/pulp/repos/$1/$2" [R]
#######

<Location /pub>

3.    restart httpd: systemctl restart httpd

Comment 13 Paul Dudley 2020-11-02 16:02:46 UTC
Just a note that the workaround provided succeeds for content but does not succeeds for other things like the package-upload plugin or katello-package-upload, expect these to fail due to the URL problem.

Comment 15 Mike McCune 2020-11-06 15:56:16 UTC
*** Satellite 6.8.0 Hotfix Available ***

1) Download http://people.redhat.com/~mmccune/hotfix/hotfix-1893561.tar to your Satellite

2) verify sha256:

$ sha256sum hotfix-1893561.tar 
b9ea3bd6c2c9c09e60a7ac7238d9771a611e17a876112dd764a94abc21f45437  hotfix-1893561.tar

3) stop services:

satellite-maintain service stop

4) Install:

tar xvf hotfix-1893561.tar
rpm -Uvh candlepin*.rpm

5) restart:

satellite-maintain service start

6) resume operations

Comment 18 Paul Dudley 2020-11-09 16:05:10 UTC
After installing the hotfix I'm still not able to correct the url. Tried:
- installed rpms
- restarted Satellite
- refreshed manifest
- refreshed content host certs
- unregistered/re-registered content host

Satellite:
~~~
[root@satellite ~]# rpm -qa candlepin*
candlepin-selinux-3.1.22-1.el7sat.noarch
candlepin-3.1.22-1.el7sat.noarch
~~~

Content host:
~~~
[root@ch ~]# subscription-manager refresh ; yum repolist
1 local certificate has been deleted.
All local data refreshed
Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

https://satellite.example.com/pulp/repos/ghost/Library%2Frhel_7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
...
repo id                                                                                 repo name                                                                                       status
rhel-7-server-rpms/7Server/x86_64                                                       Red Hat Enterprise Linux 7 Server (RPMs)                                                        0
repolist: 0
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager
~~~

Perhaps missing something, not sure what.

Thanks!
Paul

Comment 19 Justin Sherrill 2020-11-09 16:23:55 UTC
Paul,  You may need an installer run   'satellite-installer --upgrade' (although i think its just satellite-installer now in 6.8) in order to migrate the candlepin db to drop existing certs.  Then upon checkin systems should pull down their new certs.  Note that this doesn't fully explain the unregistering/reregistering case, but definitely try an installer run after upgrading the rpms.

Comment 20 Paul Dudley 2020-11-09 16:48:16 UTC
Justin,

Thanks! Running the satellite-installer with no options was just fine. After that a refresh on the client was the only thing needed for the certs to right themselves.

Comment 32 errata-xmlrpc 2020-11-11 14:19:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.8.0.1 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5089