Regarding the steps:
Please do not symlink redhat-uep.pem any more. osbuild-composer is looking for it because there is an edge case for subscriptions with simple content access. This is used, for example, when building cross-distro images but in the usual use case the certificate is loaded but not used.
(Ref. upstream: https://github.com/osbuild/osbuild-composer/pull/1405/commits/0e613daa157f241d7a02e88deecb32366f4a5c0d)

Now I can see two different issues in your bug report. Please correct me if I got this wrong:

1. osbuild-composer loads /etc/pki/entitlement/3665926663619286915.pem but it should load /etc/pki/entitlement/6874917509862528732.pem (and corresponding key of course).
2. the repository override in /etc/osbuild-composer/repositories/rhel-85.json specifies rhsm: "true", but composer-cli reports rhsm = false

I believe the first issue is the pressing one. It would be great if you could share the repo override and redhat.repo files with me. If not, is there an entry in redhat.repo file that actually uses the certificate /etc/pki/entitlement/3665926663619286915.pem? I would like to see how it compares to the URL https://<CUSTOM_SAT_URL>/dist/rhel8/8/x86_64/baseos/os. It is possible that the algorithm is somehow confusing these two URLs. If that is the case, we need to update the implementation.

Thanks.

I suggested this because it helped going a little bit further.

In my reproducer I had the following error when I don't make the symlink:

Nov 17 15:06:16 <HOSTNAME> osbuild-composer[16493]:   - Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://<INTERNAL_SAT_HOSTNAME>/pulp/repos/JJAN_ORG/Library/RHEL8/content/dist/rhel8/8.5/x86_64/baseos/os/repodata/repomd.xml [SSL certificate problem: self signed certificate in certificate chain]

I thought this and the fact there is an issue with the key/pair association were both related to 'rhsm = false' seen by the composer.

I answer yes to your both questions.

Attaching a tarball as private with all you need.

(In reply to Christophe Besson from comment #2)
> I suggested this because it helped going a little bit further.
> 
> In my reproducer I had the following error when I don't make the symlink:
> 
> Nov 17 15:06:16 <HOSTNAME> osbuild-composer[16493]:   - Curl error (60):
> Peer certificate cannot be authenticated with given CA certificates for
> https://<INTERNAL_SAT_HOSTNAME>/pulp/repos/JJAN_ORG/Library/RHEL8/content/
> dist/rhel8/8.5/x86_64/baseos/os/repodata/repomd.xml [SSL certificate
> problem: self signed certificate in certificate chain]

I'm suspicious about this URL:
https://<INTERNAL_SAT_HOSTNAME>/pulp/repos/JJAN_ORG/Library/RHEL8/content/dist/rhel8/8.5/x86_64/baseos/os/repodata/repomd.xml
is it possible that the URL is written like this in redhat.repo file?
https://<INTERNAL_SAT_HOSTNAME>/pulp/repos/JJAN_ORG/Library/RHEL8/content/dist/rhel8/$releasever/x86_64/baseos/os/repodata/repomd.xml
if yes, it would make osbuild-composer assume the URL looks like this
https://<INTERNAL_SAT_HOSTNAME>/pulp/repos/JJAN_ORG/Library/RHEL8/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml
so it wouldn't be able to match the keys.

Can you have a look please?

Also if it was the case, where does the $releasever value come from? Is it defined in /etc/dnf/vars or does it come from redhat-release package (rpm -q --provides redhat-release)?

> 
> I thought this and the fact there is an issue with the key/pair association
> were both related to 'rhsm = false' seen by the composer.
> 
> I answer yes to your both questions.
> 
> Attaching a tarball as private with all you need.

I can't access anymore to this machine which has been shared by a colleague from the Satellite team, but I can access another one with a similar configuration.

I confirm $releasever is present very frequently in redhat.repo, and this variable can come from at least 2 places.

1) /etc/dnf/vars/releasever but this is manual and not frequent
2) /var/lib/rhsm/cache/releasever.json, by default it is "null", which means the release is not locked to a specific version (i.e. 8.5), in this case the baseurl contains $releasever, which is expanded at runtime to rhel8/8 (the latest). If the release is locked with `subscription-manager release --set=8.5`, the $releasever appears to be directly expanded / hardcoded into the redhat.repo file.

From my understanding, the simple way to check the "expansion" of $releasever is to run a `dnf repolist -v`.
Please note this is only an observation.

It would greatly help to have an option telling to use the host repositories without entering into such details, as it is error-prone, and it will cause issues after a RHEL minor version upgrade (for the baseurls, but also for the repo overrides filename). If this is not difficult to implement, maybe an helper tool to create this configuration.

I forgot to tell your suspicion is correct, the issue is reproducible on this new machine when redhat.repo contains the variable (when the release is locked), but not when the release is locked (with redhat.repo hardcoded with the version). It seems it picks up the right key pair, whereas there are other custom repos (with the key pair of baseos/appstream not coming first, alphanumerically speaking).
And it chooses the right cacert (katello-server).

That means we have a workaround, locking the release to "8" (latest) or "8.5", depending on the repos configured on the Satellite server (sometimes it does not have both).

Thanks!

Correcting a typo to avoid confusion.

I forgot to tell your suspicion is correct, the issue is reproducible on this new machine when redhat.repo contains the variable (when the release is _NOT_ locked), but not when the release is locked (with redhat.repo hardcoded with the version). It seems it picks up the right key pair, whereas there are other custom repos (with the key pair of baseos/appstream not coming first, alphanumerically speaking).
And it chooses the right cacert (katello-server).

That means we have a workaround, locking the release to "8" (latest) or "8.5", depending on the repos configured on the Satellite server (sometimes it does not have both).

Thanks!

I'm glad there is a fairly simple solution. I'll arrange a knowledge-base article about this.

Thank you! :)

Regarding the usage of system repos and/or a helper tool: We want to improve the repository management. It is complicated by the fact that composer can build different distributions than the host system and also it can have multiple worker process distributed across multiple hosts all connecting to a single composer instance. But thanks for your feedback, it is good to know that there is an interest in using system repositories directly.

I planned to do the KCS after the confirmation of the customer that it fixes the issue for them.
I'll link it to the BZ afterwards, feel free to review/modify it once I shared it.

Ok, perfect, thanks!

Sorry to come with bad news, $releasever has been hardcoded but the issue still occurs for the customer.
However, it is a true issue since it is there by default on Satellite clients, and I can easily reproduce.

In the fresh attachment above, they initially got a "SSL certificate problem: self signed certificate in certificate chain" error, and retried the cacert symlink to go further. They still get a 403 error after that, indicating the wrong key pair is used (the fallback code).

That is not a good news. I'm trying to reproduce the issue, but so far it seems to pick the right key for me.

The fallback code is not going to work in environment where multiple keys are present in /etc/pki/entitlement. It pick the first one sorted alphabetically so unless the right key is the first one, it will fail.

Thanks for the new attachment and I'll keep you updated.

We are also hitting this issue.

We have a primary composite view that includes RHEL repos, Zabbix Repos, Treasure Data agent repos and EPEL repos. This gives us a total of 4 subscriptions related to our base content view, and as such, 4 sets of keys in /etc/pki/entitlement.

403 errors occur as long as all keys are present. If I remove the subscriptions for everything but RHEL it works as there is only key present, but I loose access to repos containing packages I need.

*** Bug 2043717 has been marked as a duplicate of this bug. ***

msehnout is correct in comment #1 about there being two issues.
I've verified with a CSS customer that the below two solutions resolves the SSL issues:
1) Placing the .pem file(s) in the /usr/share/pki/ca-trust-source/anchors/ directory fixes the SSL issue, specifically the `Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://<CUSTOM_SAT_URL>/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml [SSL certificate problem: self signed certificate in certificate chain]` error for CSS customer.
Got this from KCS article: https://access.redhat.com/solutions/5773421, which looks like it will also fix the 403 error noted in the bug description.

2) Locking the release to a specific minor version resolved another SSL error, `osbuild-worker[30358]: osbuild.host.RemoteError: RuntimeError: curl: error downloading {'url': 'https://<CUSTOM_SAT_URL>/content/dist/rhel8/$releasever/x86_64/baseos/os/Packages/<path-to-rpm>', 'secrets': {'ssl_ca_cert': '/etc/rhsm/ca/katello-server-ca.pem', 'ssl_client_key': '/etc/pki/entitlement/<num>-key.pem ', 'ssl_client_cert': '/etc/pki/entitlement/<num>.pem'}}: error code 22`

Adding again the last case (03421267).
It wouldn't have fail if the customer didn't have multiple keys (for multiple repos).

Symptom:
A CA cert error or an HTTP 403 error while trying to depsolve.
After checking the configuration, the JSON repo overrides are properly defined.

The issue can be seen in the osbuild-worker@1 log:
Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://XXXXXXXXX/RHEL8/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml [SSL certificate problem: self signed certificate in certificate chain]

Usually the support suggests to symlink the RH cacert to point to the Sat/Katello one to workaround the issue.
But that will work only if there is only one key pair in /etc/pki/entitlement.

The strace confirms the worker didn't pick up the right key pair (it seems it has chosen the first alphanumerically and it was unlucky...), explaining the issue, that becomes:
Status code: 403 for https://XXXXXXXXXXXXXXXX/Dev/RHEL8/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml 

The strace also shows /etc/yum.repos.d/redhat.repo permissions were too restrictive:
openat(AT_FDCWD</usr/libexec/osbuild-composer>, "/etc/yum.repos.d/redhat.repo", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) <0.000064>

# namei -l /etc/yum.repos.d/redhat.repo
f: /etc/yum.repos.d/redhat.repo
dr-xr-xr-x root root /
drwxr-xr-x root root etc
drwxr-xr-x root root yum.repos.d
-rw-r----- root root redhat.repo            <<<<

Fixing the perms from 0640 to 0644 resolved the issue for this customer.
osbuild-composer should report that EACCES and not ignore it.

Adding another use case for this problem.

The customer has configured a custom repo on its Satellite server, hence they have 2 key pairs in /etc/pki/entitlement.
Unfortunately, the custom repo comes with the first key pair, alphanumerically speaking.

They use RHEL 8.7 with the latest osbuild* versions:
osbuild-65-1.el8
osbuild-composer-62-3.el8_7

They correctly configured repo overrides, and that works for RHEL 8.7 but when they choose another distro from the blueprint ("rhel-86"), that does not work:

# composer-cli blueprints depsolve RHEL8.6
ERROR: BlueprintsError: RHEL8.6: DNF error occurred: RepoError: There was a problem reading a repository: Failed to download metadata for repo '59df49401dd00ec6d5141c3077df6abe32c4eef3752a0c100b4d105a06c67e08' [baseos: https://XXX/pulp/content/XXX/Library/content/dist/rhel8/8.6/x86_64/baseos/os/]: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
blueprint: RHEL8.6 v0.0.1

This is because the baseurl from /etc/yum.repos.d/redhat.repo (with $releasever hardcoded or not) does not match, and in this case osbuild-composer is not able to choose which key pair to choose, and always choose the first one in this case, leading to a CAcert error (using redhat cacert instead of the Satellite one) or a 403 error if the limited "workaround" always given by the support is done (a symlink from /etc/rhsm/ca/redhat-uep.pem to /etc/rhsm/ca/katello-server-ca.pem as explained in https://access.redhat.com/solutions/5773421).

Steps to reproduce easily without a Satellite server:

1. Install a fresh RHEL 8.7
2. Simulate another repo by just touching a new key pair that will come at first, alphanumerically

    # touch /etc/pki/entitlement/{000,000-key}.pem

3. Create a blueprint with distro = "rhel-86" and try to depsolve it.


The number of cases attached to this BZ does not reflect the number of customers impacted by this issue, as it can take several forms, and the symlink workaround hides the issue as long as there is only one key pair on the system.

Suggested workaround that worked for the custy.

Generate a debug certificate from your Satellite server.
The idea is to follow the 3.3 and 3.4 prerequisites sections from:

https://access.redhat.com/documentation/en-us/red_hat_satellite/6.11/html/managing_content/managing_organizations_content-management#Creating_an_Organization_Debug_Certificate_content-management

Quoting the doc:

In the Satellite web UI, navigate to Administer > Organizations.
Select an organization that you want to generate a debug certificate for.
Click Generate and Download.
Save the certificate file in a secure location. 

Open the X.509 certificate, for example, for the default organization:
$ vi 'Default Organization-key-cert.pem'

Copy the contents of the file from -----BEGIN RSA PRIVATE KEY----- to -----END RSA PRIVATE KEY-----, into a key.pem file.
Copy the contents of the file from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE-----, into a cert.pem file.

Ensure at first it works for you with curl, as explained in §3.4 - "for curl users":

# curl -O -v --cert cert.pem --key key.pem --cacert /etc/rhsm/ca/katello-server-ca.pem https://XXX/pulp/content/XXX/content/dist/rhel8/8.6/x86_64/baseos/os/repodata/repomd.xml

If you get an HTTP 200 return code, that means the metadata repomd.xml is properly downloaded.
If it works, you can then overwrite all the keys from /etc/pki/entitlement with key.pem and all the certs (the pem files without "-key") with cert.pem.
Finally, make the pem files immutable to be sure they're not overwritten by rhsmcertd:

# chattr +i /etc/pki/entitlement/*

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.