Repo sync still fails. Any news when the problem will be fixed?

Sayan. Make sure that the entitlements that go into the manifest are refreshed after the code update. Most show the proper sat-tools urls, but a couple show the pre code-fix issues.

Sayan, can you confirm with the other commenters here the steps they took to remedy the problem? Perhaps there is another step in addition to the ones you took.

Sayan, according to the staus on the system you supplied, your Satellite is running 4.1.14 Candlepin
{"mode":"NORMAL","modeReason":null,"modeChangeTime":null,"result":true,"version":"4.1.14","release":"${release}","standalone":true,"timeUTC":"2022-10-24T23:39:31+0530","rulesSource":"database","rulesVersion":"5.43","managerCapabilities":["instance_multiplier","derived_product","vcpu","cert_v3","hypervisors_heartbeat","remove_by_pool_id","syspurpose","storage_band","cores","hypervisors_async","org_level_content_access","guest_limit","ram","batch_bind","combined_reporting"],"keycloakRealm":null,"keycloakAuthUrl":null,"keycloakResource":null}

The fix is in 4.1.17 as noted above.

Does the entitlement get generated in the portal for the Satellite system, or does the entitlement get generated by the Satellite system itself?

The entitlement get generated in the portal for the Satellite system, which just imports and uses it to fetch content.

Then that entitlement should have access based on the update to the portal.

It is possible that the entitlements on the Satellite system will need to be refreshed to pick up the change.

Satellite manifests are created on the portal side of the house. Satellite CP version isn't relevant for this issue.

With a bit more time since the fix and now - we're seeing a number of customers who reflect both my and Sayan's experience. Sometimes a simple refresh is solving the problem. Sometimes it isn't. 

For example, a manifest with a tag of "created":"2022-10-24T17:33:48.933+00:00" uploaded to a Satellite for a recent case and still the Satellite retrieves a 403 on a sync.

Looking at the Authorized Content URLs for certificates there I'm unable to find '/content/dist/layered/rhel8/x86_64/sat-tools/6.10/os' for example. Uploading the manifest to the BZ though in case it helps.

As far as I know, the entitlement that gives the Satellite system access is not in the Manifest. They are from the portal directly. Have you tried doing a refresh on the certs on the Satellite system?

We need someone in Satellite for this. I fixed a bug in Candlepin that was dropping certain urls on entitlement creation. I am not entirely sure how all that gets propagated to the systems downstream.

The exact entitlement in the manifest is not used by the clients. It is regenerated by candlepin in satellite for each consumer. The actual content that is imported into Satellite is from the large block of JSON in the --ENTITLEMENT DATA-- section of the certificate.

So if the clients are not getting good urls in the authorized section, it is likely because the Satellite is not generating good certs for the clients. That would require an update to the Candlepin instance in Satellite.

I don't know why the mainfests coming from the portal would be wrong as they updated the system to include the bug fix as far as I know.

All entitlements for manifest consumers have been marked as needing a refresh. When a manifest refresh is called, the new manifest will contain new entitlements that will have the proper list of authorized URLs.

This takes the place of the fix in Comment 47.

Hello William,

I was on leave for the last few days, but I see Joniel, Paul, and Hao have already worked on the BZ and shared the necessary information. 

If I understand correctly, We ( RedHat ) have applied all necessary fixes at our end and affected users are only required to refresh their manifest in satellite and that is good enough to have the reported behavior fixed at satellite level?



-- Sayan

(In reply to Sayan Das from comment #49)
> Hello William,
> 
> I was on leave for the last few days, but I see Joniel, Paul, and Hao have
> already worked on the BZ and shared the necessary information. 
> 
> If I understand correctly, We ( RedHat ) have applied all necessary fixes at
> our end and affected users are only required to refresh their manifest in
> satellite and that is good enough to have the reported behavior fixed at
> satellite level?
> 
> 
> 
> -- Sayan


Never mind, I had tested on another Satellite and the simple manifest refresh did the trick this time. 

One of the external Satellite users has also confirmed that a simple manifest refresh fixed the issue. So I am good here ..

I was directed by Red Hat Confirmed Stateside Support to update/upgrade to a Red Hat Satellite version that includes candlepin-4.2.10-1 or candlepin-4.1.17-1. 
This turned out to be RH Satellite v6.11.4 for our server. 
When you upgrade to v6.11.x:
     Red Hat Satellite Tools 6.8 for RHEL 8 x86_64 RPMs
     Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 RPMs
     Red Hat Satellite Tools 6.9 for RHEL 8 x86_64 RPMs
     Red Hat Satellite Tools 6.10 for RHEL 8 x86_64 RPMs

Red Hat Satellite Tools for THEL 8 has been replaced by Red Hat Satellite Client 6 for RHEL 8.

So I no longer have the issue, but I am not sure if Candlepin update would resolve.

Verified.

Version Tested: Satellite 6.13 Snap 13.0 

Steps to Verify:
1) Import a manifest and enable Satellite Tools repo for RHEL8.
2) Sync the repos.

Result:
Satellite Tools repos can be enabled and synced successfully.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.13 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2097