Bug 2221864

According to the description and discussion it looks like that the issue is not in DNF but the change that triggers the problem is in subscription manager. DNF itself is unable to generate repositories or repo files. Changing the component.

Hi,

this is Pino Toscano from the subscription-manager team.

Can you please describe how the entitlement secrets (SCA entitlement certificates, most likely) are available in the pods? From the above instructions, I do not see how they are attached in the pod, and in which locations.

According to my knowledge, the instructions to follow are:
https://docs.openshift.com/container-platform/4.12/cicd/builds/running-entitled-builds.html
The current behaviour in 9.2 was changed because of the way entitled builds are done in OpenShift, as bug 2108549; this means that the entitlement secrets must be put in non-container locations, because the expectations that we got from the OpenShift team is that RHEL pods running in OCP are to not be considered as containers, to ensure subscription-manager is usable inside them.

This means that:

(In reply to Noam Angel from comment #0)
> [root@cluster-entitled-build-ubi9 /]# ls -l /etc/pki/entitlement-host/
> total 176
> -rw-r--r--. 1 root root 87758 Jul 11 06:24 entitlement-key.pem
> -rw-r--r--. 1 root root 87758 Jul 11 06:24 entitlement.pem
> [root@cluster-entitled-build-ubi9 /]# ls -l /etc/pki/entitlement
> total 0

... this does not seem correct. Can you please try to follow the above instructions?

we use cluster wide entitlement from OCP4.8 same procedure worked. only for ocp4.13 its not working.
we dont build CI/CD containers and cannot mount secret on the container. we checked mounting the secret and ubi9 was able to get packages but it wont work for "Network Operator" 

MOFED container needs prerequisite packages to compile with current kernel version on cluster hence why we need the cluster wide entitlement to work.


we expect similar behaver and procedure to work
Such breaking changes was not announced

https://examples.openshift.pub/cluster-configuration/full-cluster-entitlement/

Hello,

(In reply to Noam Angel from comment #15)
> https://examples.openshift.pub/cluster-configuration/full-cluster-entitlement/

This creates entitlement certificates in /etc/pki/entitlement/, which is the correct location; OTOH, the description of this bug shows the certificates in /etc/pki/entitlement-host/, which is not correct, and it does not match the linked template;
- can you please check what's in the generated container?
- can you please check whether you are using some local customized version of that template, in which the location of entitlement certificates was changed to /etc/pki/entitlement-host/ (maybe to workaround some older issue/situation)?

Thanks!

you can take any openshift cluster of from 4.8 and this is where the key will be mounted on, dnf use to recognize the key in  "/etc/pki/entitlement-host/" and worked
issue is the RHEL9.x and UBI9 does not work that why and dont care the key in "/etc/pki/entitlement-host/". if i where to take UBI8 on OCP4.13 OpenShit cluster the month is the same /etc/pki/entitlement-host/
and dnf works

(In reply to Noam Angel from comment #17)
> you can take any openshift cluster of from 4.8 and this is where the key
> will be mounted on, dnf use to recognize the key in 
> "/etc/pki/entitlement-host/" and worked
> issue is the RHEL9.x and UBI9 does not work that why and dont care the key
> in "/etc/pki/entitlement-host/". if i where to take UBI8 on OCP4.13 OpenShit
> cluster the month is the same /etc/pki/entitlement-host/
> and dnf works

dnf has nothing to do with /etc/pki/entitlement-host/ or /etc/pki/entitlement/: those locations are set by the dnf plugin of subscription-manager in the subscription repositories for dnf.

Also, from what I know, the instructions and templates for older versions of OCP referenced only /etc/pki/entitlement/: this is why I'm asking questions about /etc/pki/entitlement-host/ in your case. And yes, it does matter.

i dont mount this file to /etc/pki/entitlement-host/, this is how OpenShift behave


same OpenShift cluster with cluster wide entitlement:
  * UBI8 - downloads packages works, key at /etc/pki/entitlement-host/
  * UBI9 - downloads packages dont work, key at /etc/pki/entitlement-host/

I believe the customer is trying to do what is listed in https://cloud.redhat.com/blog/how-to-use-entitled-image-builds-to-build-drivercontainers-with-ubi-on-openshift or https://access.redhat.com/solutions/5820151 (but this is dated; for an older unsupported version of OCP). 

As such, I have been running some test in the CEE Resource Hub 4.13 cluster (https://resourcehub.redhat.com/resources/quickcluster/clusters/1958), to confirm a few facts. Most of my testing has been based off using SCA (Simple Content Access) entitlement content from within OpenShift. To understand how this works you have to read https://docs.openshift.com/container-platform/4.13/support/remote_health_monitoring/insights-operator-simple-access.html or the following: 

> Insights Operator periodically imports your simple content access entitlements from OpenShift Cluster Manager Hybrid Cloud Console and stores them in the etc-pki-entitlement secret in the openshift-config-managed namespace.

- Note: https://cloud.redhat.com/blog/how-to-build-images-with-rhel-subscriptions-on-openshift gives an over view of this. 

Because of this I have a secret or entitlement on the OpenShift cluster (however its not necessarily shared or accessible to builds or pods; and the docs for OCP on this don't make this clear; xref
https://docs.openshift.com/container-platform/4.13/cicd/builds/running-entitled-builds.html#builds-source-secrets-entitlements_running-entitled-builds). Fundamentally the docs 'suggest' that you should be access this secret (in a shared manner) however I don't believe this to be a correct assumption. 

- Note: this secret is titled/named 'etc-pki-entitlement' and lives in the 'openshift-config-managed' namespace on the cluster. 

The reason I say this is because of the following docs: 

https://docs.openshift.com/container-platform/4.13/cicd/builds/running-entitled-builds.html#builds-running-entitled-builds-with-sharedsecret-objects_running-entitled-builds

These docs and and https://stackoverflow.com/questions/46297949/sharing-secret-across-namespaces go into describing that scerets can't be shared across namespace boundaries; so to make the secrete accessible or usable by the pods I am creating once either has to enable a Technology Preview CSI driver; or run a command like the following: 

# oc get -n openshift-config-managed secrets etc-pki-entitlement -o yaml | sed "s/namespace: .*/namespace: $(oc project -q)/" | oc create -f -

- Note: this is in alignment with solutions or other media/materials that have you get entitlement information from the customer portal and save it into the cluster (what is odd about instructions like this (https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/cluster-entitlement.html) is that they you save this as a machine config and suggest in the materials that this things in the container to be updated for subscription-management uses).

- Followup: @dornelas I many need you to help me track down; how that works between rhcos and the runtime. 

With all of what I explained above, this leads me to believe that @ptoscano is correct in pointing out: 

> dnf has nothing to do with /etc/pki/entitlement-host/

In-fact this directory is just a symlink to /run/secrets/etc-pki-entitlement

> sh-4.4# ls -la /etc/pki/entitlement-host 
> lrwxrwxrwx. 1 root root 32 Jun 23 02:05 /etc/pki/entitlement-host -> /run/secrets/etc-pki-entitlement 

In short this directory is just looking for a pod/container secret named etc-pki-entitlement. 
xref: https://www.redhat.com/sysadmin/new-podman-secrets-command kube or oci containers all work the same way here. 

Because https://raw.githubusercontent.com/openshift-psap/blog-artifacts/master/how-to-use-entitled-builds-with-ubi/0004-cluster-wide-entitled-pod.yaml doesn't have a secret; this directory is empty in all of my tests (or doesn't exsist), using both ubi8 and ubi9. 

This results in the following happening: 

> sh-4.4# dnf search kernel-devel --showduplicates
> Updating Subscription Management repositories.
> Unable to read consumer identity
> Subscription Manager is operating in container mode.

> This system is not registered with an entitlement server. You can use subscription-manager to register.
> ... 

- Note: `This system is not registered with an entitlement server. You can use subscription-manager to register.`

At the end of the day this looks like the secret mapping to the /etc/pki/entitlement-host location is just incorrect and should be directed at /etc/pki/entitlement (but this doesn't explain why the samples that were originally posted with ubi8 work; they didn't for me). 

Other good reads: 
https://cloud.redhat.com/blog/the-path-to-improving-the-experience-with-rhel-entitlements-on-openshift
https://cloud.redhat.com/blog/addressing-the-experience-with-rhel-entitlements-with-openshift

These are related bugs to what I saw in my testing: 

Bug 1970560 - yum commands fail during entitled builds using ubi8 image
Bug 2101292 - "yum update" command fail in UBI 8.0 - UBI 8.5 container when subscription-manager / RHSM credentials injected

Bug 2216079 - Curl error (77): Problem with the SSL CA cert when using UBI9 image on OpenShift Container Platform 4

After talking to dornelas further about this; and testing this a bit more, there is indeed a bug in the behavior of `in_container` (https://github.com/candlepin/subscription-manager/blob/subscription-manager-1.29.33-1/src/rhsm/config.py#L153-L156) because when in OCP this returns False!

In short; this is confirmed by doing the following: 

1. Follow https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/cluster-entitlement.html
   - Note: This method isn't what OCP recommends for providing entitlements to pods/containers.
           However this is made possible by a long standing container / host data passing, see [0] and [1]

2. modify https://github.com/candlepin/subscription-manager/blob/subscription-manager-1.29.33-1/src/rhsm/config.py#L153-L156 (by editing the /usr/lib64/python3.9/site-packages/rhsm/config.py in the container so this returns True; instead of False).

When this returns True (your in a container); for both UBI 8 and 9 DNF (the search) behaves correctly (and searches the rhel repositories). When this returns False (for UBI 9) as described in the bugs description; dnf fails to operate correctly.

- Note: /etc/pki/entitlement/ is empty; and /etc/pki/entitlement-host/ points to /run/secrets/etc-pki-entitlement; which because of the machine config settings; has mounted values from the host, and https://github.com/candlepin/subscription-manager/blob/subscription-manager-1.29.33-1/src/rhsm/config.py#L396-L397 defines where to get the entitlements from by overwriting the entitlementcertdir to be /etc/pki/entitlement-host/ and NOT /etc/pki/entitlement/.

- Note: because (in this testing) the host; entitlement passing is in place my testing doesn't trust the use of (Simple Content Access) entitlement content from within OpenShift using https://docs.openshift.com/container-platform/4.13/support/remote_health_monitoring/insights-operator-simple-access.html

TEST POD: Run with `oc debug -f pod[8|9].yaml`                     ## Align version to text in file.
>> apiVersion: v1
>> kind: Pod
>> metadata:
>>   name: cluster-entitled-build-pod-[8|9]                        ## Pick a version
>> spec:
>>   containers:
>>     - name: cluster-entitled-build
>>       image: registry.access.redhat.com/ubi[8|9]:latest         ## Pick a version
>>       command: [ "/bin/sh", "-c", "dnf search kernel-devel --showduplicates" ]
>>       securityContext:
>>            runAsUser: 0
>>   #     volumeMounts:
>>   #       - name: etc-pki-entitlement
>>   #         readOnly: true
>>   #         mountPath: /etc/pki/entitlement
>>   # volumes:
>>   #   - name: etc-pki-entitlement
>>   #     secret:
>>   #       secretName: etc-pki-entitlement
>>   restartPolicy: Never

[0] https://github.com/containers/common/blob/main/docs/containers-mounts.conf.5.md
[1] https://github.com/containers/common/blob/v0.55.2/pkg/subscriptions/subscriptions.go#L220-L306

Hello Team,

Do we have an update further on the issue ?

Regards

Sachin Sharma