This is perhaps related to bz 2047675 - except this BZ seems to catch on the CP side of things and 2047675 catches on the Pulp side of things.

I believe receiving this error is specific to 6.9 > 6.10 upgrade rather than 6.10 fresh installs. If a repository exists but doesn't have a product 'owner' in cp any  this error can come up during several tasks after an upgrade. 

For example, if on 6.9 a repository is orphaned for any reason (losing an ELS sub perhaps) the repo and it's contents will be migrated successfully, but with many actions we'll receive this error against the orphaned repository.

An example error when running a CV publish:
~~~
Input:
---
content_view_id: 4
environment_id: 1
cp_environment_id: 26b94739ca64bdef3279c257871edcde
new_content_id: 
session_id: 6dc01561-0a9a-48d1-a0c8-6f0c5bb0e002
remote_user: admin
remote_cp_user: admin
current_request_id: 6dc01561-0a9a-48d1-a0c8-6f0c5bb0e002
current_timezone: TZ/TZ
current_organization_id: 1
current_location_id: 
current_user_id: 5

Output:
---
add_ids:
- '7408'
- '7396'
delete_ids: []
Error:

Katello::Errors::CandlepinError

Unable to find content with the ID "7408".

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/propagate_candlepin_errors.rb:21:in
  `rescue in propagate_candlepin_errors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/propagate_candlepin_errors.rb:18:in
  `propagate_candlepin_errors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/propagate_candlepin_errors.rb:13:in
  `finalize'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/remote_action.rb:20:in
  `block in finalize'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
~~~

For this above case, the content IDs are not in the CV trying to be published. That ID is only mentioned as part of this error and nowhere else:
~~~
 △ rg 7408 tmp/task-export20220204-24361-huksr7/a003f11e-92d7-4bd2-b342-3760dbf22b94.html 
93732:- '7408'
93747:      Unable to find content with the ID "7408".
~~~

Uploading this example task to help with more info.

For this rake it may make sense to create a new rake task for the issue since correct_repositories has/should only affect katello and Pulp. Perhaps run the new rake first and then correct_repos after.

Similar to how katello:reimport can get stuck on missing puppet env repos and the katello:correct_puppet_environments is needed to correct those before a reimport can succeed.

Do we have any idea if these content sets that are missing are Red hat repositories or custom?  

Or why exactly they are going missing.  I see paul, you mention that they are orphaned, but from what?  I new manifest being imported (and a subscription no longer existing, or a content no longer existing in the manifest)?

I wasn't under the impression that candlepin ever removed 'content' from a product once it was added.  Its also unclear if this has anything to do with 6.10, or just happened to be noticed in 6.10 due to the migration. 

If its red hat content, i'm not even sure what we katello could do, as we don't normally add/remove RH content from products, and i'm not sure if we have access to do that.  Any additional information would be very helpful.

Adding needinfo based upon comment 7

In the couple of cases I looked closely at, it was orphaned content by a subscription falling off - ELS or EUS content that used to be provided on the manifest but no longer is. They still kept the actual content, and it was still available on the Satellite, only subscription access had fallen off. I couldn't reproduce this problem in a 6.10 that hadn't been migrated/updated. 

In another case, orphaned content (ELS for this case) had a sync plan created for them during previous versions. Before the 6.10 migration this sync plan ran as expected; it would run, the syncs would fail with a warning due to no longer having certs to access the repos. In the state caused by this issue, the cause that pointed us here was the customer trying to disable a repository and hitting the error noted here, 'Katello::Errors::CandlepinError: Unable to find content with the ID "5874"'.

In a Satellite with this issue we believe it should be possible to reproduce the problem with most any disable repo task or content view publish or promote task. @phess has a reproducer for this and some helpful bits to add also. 

Thanks!

Connecting redmine issue https://projects.theforeman.org/issues/35572 from this bug

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35572 has been resolved.

Note: This bug is introduced by https://bugzilla.redhat.com/show_bug.cgi?id=2136851

Steps for QA to reproduce/test:

1. Create a Product, name it 'files' for example
2. Create a new repo of type 'yum' in it and provide a URL.
3. Save the settings
4. Remove the URL in the repo
5. Save

7. Attempt to delete the repository it will fail with:
 bad argument (expected URI object or URI string)

8. Try to add a dummy URL it will fail:
An error occurred saving the Repository: undefined method `label' for nil:NilClass

9. Try to refresh the manifest files, you will get an error because of missing content.

Katello::Errors::CandlepinError: Unable to find content with the ID "1666022673761".

[root@xx~]# su - postgres -c "psql candlepin -c \"select uuid, content_id, contentUrl, label, type, vendor from cp2_content where content_id = '1666022673761'\""
               uuid               |  content_id   |          contenturl          |            label            | type | vendor 
----------------------------------+---------------+------------------------------+-----------------------------+------+--------
 8a95ca7d83e3aea00183e6b08d62012f | 1666022673761 | /custom/files/yara_for_RHEL8 | MINFIN_files_yara_for_RHEL8 | yum  | Custom

Verified on 6.13 - Snap 11

Steps to Verify:
1. Create a Product, name it 'files' for example
2. Create a new repo of type 'yum' in it and provide a URL.
3. Save the settings
4. Remove the URL in the repo
5. Save
6.Attempt to delete the repository, change the URL in the repository and refresh the manifest files.

Expected Results:
All operations should succeed

Actual Results:
All operations succeed.

Additional Test:
1) Add the above repo to a content view and attempt to publish it.

Expected Results:
Satellite will throw an error before attempting the publish,'Product: 'testo', Repository: 'testerton' ' in this content view does not exist in the backend system [ Candlepin ]. Remove the invalid repository before publishing again.

Actual Results:
Satellite will throw an error before attempting the publish,'Product: 'testo', Repository: 'testerton' ' in this content view does not exist in the backend system [ Candlepin ]. Remove the invalid repository before publishing again.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.13 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2097