Bug 1945670

Summary:	osbuild-composer fails when multiple custom repos are defined on the Satellite server
Product:	Red Hat Enterprise Linux 8	Reporter:	Christophe Besson <cbesson>
Component:	osbuild-composer	Assignee:	Image Builder team <osbuilders>
Status:	CLOSED DUPLICATE	QA Contact:	Release Test Team <release-test-team-automation>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	8.3	CC:	jpasqual, msehnout
Target Milestone:	beta	Keywords:	Reproducer
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-04-07 12:57:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Christophe Besson 2021-04-01 15:15:12 UTC

Description of problem:
When multiple repos are defined, there are multiple key/cert pairs in /etc/pki/entitlement. It seems osbuild does not pick up the good cert for the good repo and then stops with a 403 error. Stracing the behaviour shows only 1 cert is read by osbuild-composer, and replacing that key/cert pair by the debug cert of the Sat server fixes the issue.

Version-Release number of selected component (if applicable):
python3-osbuild-18-3.el8.noarch
osbuild-selinux-18-3.el8.noarch
osbuild-18-3.el8.noarch
osbuild-ostree-18-3.el8.noarch
osbuild-composer-worker-20.1-1.el8.x86_64
osbuild-composer-20.1-1.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Setup several custom repos from a Satellite server like this:
# dnf repolist
Updating Subscription Management repositories.
repo id                                                                                   repo name
DefaultOrganization_CustomProduct2_CustomRepo2                                            CustomRepo2
DefaultOrganization_CustomProduct3_CustomRepo3                                            CustomRepo3
DefaultOrganization_CustomProduct_CustomRepo1                                             CustomRepo1
rhel-8-for-x86_64-appstream-rpms                                                          Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-baseos-rpms                                                             Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)

2. Create a minimal blueprint
3. Run a depsolve
# composer-cli blueprints depsolve test

Actual results:
2021-04-01 15:11:28,851: test: DNF error occured: RepoError: Error occurred when setting up repo: Failed to download metadata for repo '0': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

From the service status:
Apr 01 15:00:03 XXX osbuild-composer[11890]: Errors during downloading metadata for repository '0':
Apr 01 15:00:03 XXX osbuild-composer[11890]:   - Status code: 403 for https://XXXXXX/pulp/repos/DefaultOrganization/Library/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml (IP: 10.8.29.148)


Additional info:
* on our testing system, here is the association between certs and repos:
# grep -e name -e clientcert /etc/yum.repos.d/redhat.repo | sed 'N;s/\n/ -> /'
name = CustomRepo1 -> sslclientcert = /etc/pki/entitlement/7933703651961095112.pem
name = Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) -> sslclientcert = /etc/pki/entitlement/8048553451084698847.pem
name = CustomRepo3 -> sslclientcert = /etc/pki/entitlement/3122869533309421052.pem
name = Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) -> sslclientcert = /etc/pki/entitlement/8048553451084698847.pem
name = CustomRepo2 -> sslclientcert = /etc/pki/entitlement/460051576411210318.pem

* the strace shows only /etc/pki/entitlement/3122869533309421052.pem is read.

Comment 1 Martin Sehnoutka 2021-04-06 08:25:12 UTC

Hello,

as opposed to lorax-composer, osbuild-composer does not pick up system repositories. See this part of the documentation for the way to define custom repositories:
https://www.osbuild.org/guides/user-guide/managing-repositories.html#custom-3rd-party-repositories

The fact that osbuild-composer reads /etc/pki/entitlement/3122869533309421052.pem is an unfortunate implementation detail (this file is encountered first while reading alphabetically sorted files in the directory).

I'm not aware of any integration between osbuild-composer and Satellite, so the repositories need to be defined manually using "composer-cli sources".

Comment 2 Christophe Besson 2021-04-07 08:19:39 UTC

Just saw the following, this one could be marked as a duplicate:
https://bugzilla.redhat.com/show_bug.cgi?id=1896185

Unfortunately, it does not work, making osbuild-composer not usable behind a Satellite server.
I'm not sure we can consider this as an unfortunate implementation detail, it always picks up the first cert (the wrong one) and then fails. Replacing that cert/key by the debug one from the Sat server works, so that would mean there is no way to associate the good key pair with the repos.

Comment 3 Martin Sehnoutka 2021-04-07 12:57:23 UTC

You are right, it is not just unfortunate implementation detail. So far we only have the workaround from the BZ you linked. I'm closing this as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1896185 and we will continue the discussion there.

*** This bug has been marked as a duplicate of bug 1896185 ***