RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1945670 - osbuild-composer fails when multiple custom repos are defined on the Satellite server
Summary: osbuild-composer fails when multiple custom repos are defined on the Satellit...
Keywords:
Status: CLOSED DUPLICATE of bug 1896185
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: osbuild-composer
Version: 8.3
Hardware: All
OS: Linux
medium
medium
Target Milestone: beta
: ---
Assignee: Image Builder team
QA Contact: Release Test Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-01 15:15 UTC by Christophe Besson
Modified: 2021-04-07 12:57 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-04-07 12:57:23 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)

Description Christophe Besson 2021-04-01 15:15:12 UTC 
Description of problem:
When multiple repos are defined, there are multiple key/cert pairs in /etc/pki/entitlement. It seems osbuild does not pick up the good cert for the good repo and then stops with a 403 error. Stracing the behaviour shows only 1 cert is read by osbuild-composer, and replacing that key/cert pair by the debug cert of the Sat server fixes the issue.

Version-Release number of selected component (if applicable):
python3-osbuild-18-3.el8.noarch
osbuild-selinux-18-3.el8.noarch
osbuild-18-3.el8.noarch
osbuild-ostree-18-3.el8.noarch
osbuild-composer-worker-20.1-1.el8.x86_64
osbuild-composer-20.1-1.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Setup several custom repos from a Satellite server like this:
# dnf repolist
Updating Subscription Management repositories.
repo id                                                                                   repo name
DefaultOrganization_CustomProduct2_CustomRepo2                                            CustomRepo2
DefaultOrganization_CustomProduct3_CustomRepo3                                            CustomRepo3
DefaultOrganization_CustomProduct_CustomRepo1                                             CustomRepo1
rhel-8-for-x86_64-appstream-rpms                                                          Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-baseos-rpms                                                             Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)

2. Create a minimal blueprint
3. Run a depsolve
# composer-cli blueprints depsolve test

Actual results:
2021-04-01 15:11:28,851: test: DNF error occured: RepoError: Error occurred when setting up repo: Failed to download metadata for repo '0': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

From the service status:
Apr 01 15:00:03 XXX osbuild-composer[11890]: Errors during downloading metadata for repository '0':
Apr 01 15:00:03 XXX osbuild-composer[11890]:   - Status code: 403 for https://XXXXXX/pulp/repos/DefaultOrganization/Library/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml (IP: 10.8.29.148)


Additional info:
* on our testing system, here is the association between certs and repos:
# grep -e name -e clientcert /etc/yum.repos.d/redhat.repo | sed 'N;s/\n/ -> /'
name = CustomRepo1 -> sslclientcert = /etc/pki/entitlement/7933703651961095112.pem
name = Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) -> sslclientcert = /etc/pki/entitlement/8048553451084698847.pem
name = CustomRepo3 -> sslclientcert = /etc/pki/entitlement/3122869533309421052.pem
name = Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) -> sslclientcert = /etc/pki/entitlement/8048553451084698847.pem
name = CustomRepo2 -> sslclientcert = /etc/pki/entitlement/460051576411210318.pem

* the strace shows only /etc/pki/entitlement/3122869533309421052.pem is read.
Comment 1 Martin Sehnoutka 2021-04-06 08:25:12 UTC 
Hello,

as opposed to lorax-composer, osbuild-composer does not pick up system repositories. See this part of the documentation for the way to define custom repositories:
https://www.osbuild.org/guides/user-guide/managing-repositories.html#custom-3rd-party-repositories

The fact that osbuild-composer reads /etc/pki/entitlement/3122869533309421052.pem is an unfortunate implementation detail (this file is encountered first while reading alphabetically sorted files in the directory).

I'm not aware of any integration between osbuild-composer and Satellite, so the repositories need to be defined manually using "composer-cli sources".
Comment 2 Christophe Besson 2021-04-07 08:19:39 UTC 
Just saw the following, this one could be marked as a duplicate:
https://bugzilla.redhat.com/show_bug.cgi?id=1896185

Unfortunately, it does not work, making osbuild-composer not usable behind a Satellite server.
I'm not sure we can consider this as an unfortunate implementation detail, it always picks up the first cert (the wrong one) and then fails. Replacing that cert/key by the debug one from the Sat server works, so that would mean there is no way to associate the good key pair with the repos.
Comment 3 Martin Sehnoutka 2021-04-07 12:57:23 UTC 
You are right, it is not just unfortunate implementation detail. So far we only have the workaround from the BZ you linked. I'm closing this as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1896185 and we will continue the discussion there.

*** This bug has been marked as a duplicate of bug 1896185 ***
Note You need to log in before you can comment on or make changes to this bug.