This issue can happen when one or more custom products have been deleted from the Katello but not yet deleted from Candlepin. When user performs an import/refresh manifest, Katello tries to import the deleted custom products back to Katello. The bug is, Katello thought that all the importing products are Red Hat products so it assigns "Red Hat" providers to all of them.

# app/models/katello/glue/provider.rb
"engineering_product_id?" method will match all CP product id with digits only(regex \d+) which includes custom products because they are 11 digit numbers, such as '856945816336'
-----------------------------------
      def import_products_from_cp
        cp_products = ::Katello::Resources::Candlepin::Product.all(organization.label, [:id, :name, :multiplier, :productContent])
        cp_products = cp_products.select { |prod| Glue::Candlepin::Product.engineering_product_id?(prod['id']) }  <======================

        prod_content_importer = Katello::ProductContentImporter.new

        Katello::Logging.time("Imported #{cp_products.size} products") do
          cp_products.each do |product_json|
            product = import_product(product_json)
            prod_content_importer.add_product_content(product, product_json['productContent']) if product.redhat?
          end
        end
-----------------------------------

# app/models/katello/glue/candlepin/product.rb
----------------------------------
    def self.import_from_cp(attrs, organization)  <==== call by import_product()
      import_logger = attrs[:import_logger]

      product_attrs = {'name' => attrs['name'],
                       'cp_id' => attrs['id'],
                       'label' => Util::Model.labelize(attrs['name']),
                       'multiplier' => attrs['multiplier'],
                       'organization_id' => organization.id,
                       'provider_id' => organization.redhat_provider.id}  <================== assigns "Red Hat" provider to the importing product 

      Product.create!(product_attrs)
----------------------------------



Steps to reproduce:
1) Create a custom product on Satellite.

2) Run the following command to delete the custom product from Katello database only
-----------------------------------
foreman-rake console
product = Katello::Product.find_by_name("<PRODUCT NAME>")
product.provider    <====================== This should be an "Anonymous" provider
product.destroy
exit
-----------------------------------

3) Trigger manifest refresh on Satellite and wait for it to finish successfully

4) Now we will see that the deleted product got imported from Candlepin with wrong provider.
For example:
-----------------------
irb(main):021:0> product = Katello::Product.find_by_name("test_custom_product")
=> #<Katello::Product id: 314, name: "test_custom_product", description: nil, cp_id: "450776116221", multiplier: nil, provider_id: 2, created_at: "2022-12-15 02:46:41", updated_at: "2022-12-15 02:46:41", gp...
irb(main):022:0> product.provider
=> #<Katello::Provider id: 2, name: "Red Hat", description: nil, provider_type: "Red Hat", organization_id: 1, created_at: "2022-02-21 09:41:28", updated_at: "2022-02-21 09:41:28", task_status_id: nil>   <============ Provider becomes "Red Hat"!!!
------------------------

5) Run reimport script.

# foreman-rake katello:reimport
------------------------
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Importing Katello::Subscription
Importing Katello::Pool
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Subscription can't be blank, A Pool and its Subscription cannot belong to different organizations
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/validations.rb:80:in `raise_validation_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/validations.rb:53:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:318:in `block in save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:318:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/suppressor.rb:48:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/lib/katello/lazy_accessor.rb:67:in `save!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/persistence.rb:55:in `create!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:115:in `block in create!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:407:in `block in scoping'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:787:in `_scoping'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:407:in `scoping'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:115:in `create!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:124:in `first_or_create!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/models/katello/glue/candlepin/candlepin_object.rb:31:in `block in import_candlepin_record'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/lib/katello/util/support.rb:97:in `active_record_retry'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/models/katello/glue/candlepin/candlepin_object.rb:30:in `import_candlepin_record'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/models/katello/glue/candlepin/pool.rb:66:in `import_candlepin_record'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/models/katello/glue/candlepin/candlepin_object.rb:17:in `block in import_candlepin_records'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/models/katello/glue/candlepin/candlepin_object.rb:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/models/katello/glue/candlepin/candlepin_object.rb:16:in `import_candlepin_records'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/models/katello/glue/candlepin/candlepin_object.rb:53:in `block in import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation/delegation.rb:88:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation/delegation.rb:88:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/app/models/katello/glue/candlepin/candlepin_object.rb:51:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/lib/katello/tasks/reimport.rake:27:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/lib/katello/tasks/reimport.rake:25:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.50/lib/katello/tasks/reimport.rake:25:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
------------------------

Created redmine issue https://projects.theforeman.org/issues/36025 from this bug

Upstream bug assigned to jlenz

Upstream bug assigned to jlenz

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.13 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2097