Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1880947

Summary: Satellite fails with "HTTP error (500 - Internal Server Error): PG::UniqueViolation: ERROR: duplicate key value violates unique constraint" while running concurrent registrations
Product: Red Hat Satellite Reporter: Imaan <ikaur>
Component: Hosts - ContentAssignee: Pavel Moravec <pmoravec>
Status: CLOSED ERRATA QA Contact: Imaan <ikaur>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.8.0CC: jturel, parmstro, pmendezh, pmoravec, pveiga
Target Milestone: 6.13.0Keywords: Performance, Reopened, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-03 13:20:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Imaan 2020-09-21 08:24:54 UTC 
Description of problem:

Satellite fails with "HTTP error (500 - Internal Server Error): PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint" while ruining concurrent registrations

Version-Release number of selected component (if applicable):

katello_version = katello-3.16.0-1.el7sat.noarch
satellite_version = satellite-6.8.0-1.el7sat.noarch

Steps to Reproduce:

- Try to run concurrent registration on Satellite 6.8

Actual results: 

~~~


HTTP error (500 - Internal Server Error): PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_operatingsystems_on_title

~~~

Expected results: It should register at least 300 hosts concurrently (300 number in comparison to Satellite 6.7) 

Additional info: 

when pool size is set : 30 (suggested on BZ - https://projects.theforeman.org/issues/29370) : concurrent registration failed with specified error. 

Similar or related to this bug 

https://bugzilla.redhat.com/show_bug.cgi?id=1878067
Comment 1 Imaan 2022-01-31 07:54:22 UTC 
This issue has been resolved in sat 6.9 & 6.10. 

So, closing this bugzilla. Thankyou.
Comment 2 Pavel Moravec 2022-09-02 20:44:15 UTC 
I can reproduce on Sat6.10 and I bet on 6.11 it would be the same.

Key point is to:
- concurrently register systems with the same OS title (like "RedHat 8.6")
- while that OS was *not* listed among OSes in the Satellite

Particular reproducer:

1) Have 5 RHEL8.6 systems named "my-rhel86-[1-5]", with katello-ca-latest installed / prepared to run sub-man register
2) Have some AK (optionally)
3) Have no RHEL8.6 system on that Satellite
3) Run below script:

#!/bin/bash

SYSTEMS=5

while true; do
        echo "$(date): registering all $SYSTEMS systems"
        for i in $(seq 1 $SYSTEMS); do
                ssh my-rhel86-${i} "subscription-manager register --activationkey=ak_library --org=RedHat" &
        done
        wait
        sleep 5
        echo "$(date): registered all systems:"
        hammer host list --search os_minor=6
        echo "$(date): deleting all $SYSTEMS systems"
        for i in $(seq 1 $SYSTEMS); do
                ssh my-rhel86-${i} "subscription-manager unregister" &
        done
        wait
        echo "$(date): unregstered, deleting hosts.."
        for i in $(seq 1 $SYSTEMS); do
                echo "host delete --name my-rhel86-${i}"
        done | hammer shell
        hammer host list --search os_minor=6
        hammer os delete --title "RedHat 8.6"
done


During 15 minutes / 30 loops executions, I got 6 errors:
2022-09-02T22:35:23 [I|app|fa74ed72] Started POST "/rhsm/consumers?owner=RedHat&activation_keys=ak_library" for 10.44.130.50 at 2022-09-02 22:35:23 +0200
2022-09-02T22:35:23 [I|app|fa74ed72] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
2022-09-02T22:35:23 [I|app|fa74ed72]   Parameters: {"type"=>"system", "name"=>"pmoravec-rhel86-4.satotest.redhat.com", "facts"=>"[FILTERED]", "installedProducts"=>[{"productId"=>"479", "productName"=>"Red Hat Enterprise Linux for x86_64", "version"=>"8.6", "arch"=>"x86_64"}], "contentTags"=>["rhel-8-x86_64", "rhel-8"], "role"=>"", "addOns"=>[], "usage"=>"", "serviceLevel"=>"", "owner"=>"RedHat", "activation_keys"=>"ak_library"}
2022-09-02T22:35:24 [I|aud|fa74ed72] Nic::Managed (179) create event on mac 
..
2022-09-02T22:35:26 [I|app|fa74ed72] ForemanWebhooks::EventSubscriber: host_destroyed.event.foreman event received
2022-09-02T22:35:26 [E|kat|fa74ed72] ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_operatingsystems_on_title"
 fa74ed72 | DETAIL:  Key (title)=(RedHat 8.6) already exists.
 fa74ed72 | 
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `async_exec_params'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:656:in `execute_and_clear'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:135:in `exec_insert'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:132:in `exec_insert'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:166:in `insert'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `insert'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:375:in `_insert_record'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:932:in `_create_record'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/counter_cache.rb:166:in `_create_record'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/locking/optimistic.rb:70:in `_create_record'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/attribute_methods/dirty.rb:211:in `_create_record'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/callbacks.rb:331:in `block in _create_record'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:135:in `run_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:825:in `_run_create_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/callbacks.rb:331:in `_create_record'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/timestamp.rb:110:in `_create_record'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:905:in `create_or_update'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/callbacks.rb:327:in `block in create_or_update'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:135:in `run_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/callbacks.rb:327:in `create_or_update'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/timestamp.rb:128:in `create_or_update'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:503:in `save!'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:53:in `save!'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `block in save!'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `save!'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:48:in `save!'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:55:in `create!'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/models/katello/rhsm_fact_parser.rb:68:in `operatingsystem'
 fa74ed72 | /usr/share/foreman/app/models/host/base.rb:166:in `block in set_non_empty_values'
 fa74ed72 | /usr/share/foreman/app/models/host/base.rb:165:in `each'
 fa74ed72 | /usr/share/foreman/app/models/host/base.rb:165:in `set_non_empty_values'
 fa74ed72 | /usr/share/foreman/app/models/host/base.rb:159:in `populate_fields_from_facts'
 fa74ed72 | /usr/share/foreman/app/models/host/managed.rb:505:in `populate_fields_from_facts'
 fa74ed72 | /usr/share/foreman/app/models/concerns/puppet_host_extensions.rb:3:in `populate_fields_from_facts'
 fa74ed72 | /usr/share/foreman/app/services/host_fact_importer.rb:50:in `block (2 levels) in parse_facts'
 fa74ed72 | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram'
 fa74ed72 | /usr/share/foreman/app/services/host_fact_importer.rb:49:in `block in parse_facts'
 fa74ed72 | /usr/share/foreman/app/services/host_fact_importer.rb:90:in `block in skipping_orchestration'
 fa74ed72 | /usr/share/foreman/app/models/concerns/orchestration.rb:124:in `without_orchestration'
 fa74ed72 | /usr/share/foreman/app/services/host_fact_importer.rb:89:in `skipping_orchestration'
 fa74ed72 | /usr/share/foreman/app/services/host_fact_importer.rb:45:in `parse_facts'
 fa74ed72 | /usr/share/foreman/app/services/host_fact_importer.rb:34:in `import_facts'
 fa74ed72 | /usr/share/foreman/app/models/host/base.rb:143:in `import_facts'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/models/katello/host/subscription_facet.rb:220:in `update_facts'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/services/katello/registration_manager.rb:204:in `create_in_candlepin'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/services/katello/registration_manager.rb:158:in `block in register_host'
 fa74ed72 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
 fa74ed72 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/services/katello/registration_manager.rb:156:in `register_host'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/services/katello/registration_manager.rb:40:in `process_registration'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:241:in `consumer_activate'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:195:in `process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:37:in `repackage_message'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 fa74ed72 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 fa74ed72 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 fa74ed72 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/controllers/katello/concerns/api/api_controller.rb:50:in `process_action'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:136:in `process'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:39:in `process'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:190:in `dispatch'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:254:in `dispatch'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `each'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `serve'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in `block in serve'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `each'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in `serve'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/lib/katello/middleware/event_daemon.rb:10:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.3.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 fa74ed72 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 fa74ed72 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/cookies.rb:648:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:101:in `run_callbacks'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in `call_app'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:28:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 fa74ed72 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/lib/katello/prevent_json_parsing.rb:12:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/configuration.rb:249:in `call'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:77:in `block in handle_request'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:338:in `with_force_shutdown'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:76:in `handle_request'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/server.rb:438:in `process_client'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:145:in `block in spawn_thread'
 fa74ed72 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-09-02T22:35:26 [I|app|fa74ed72] Completed 500 Internal Server Error in 2917ms (Views: 0.4ms | ActiveRecord: 452.3ms | Allocations: 299745)
Comment 3 Pavel Moravec 2022-09-02 20:52:53 UTC 
I think that:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.60/app/models/katello/rhsm_fact_parser.rb:68:in `operatingsystem'  :

       ::Operatingsystem.find_by(os_attributes) || ::Operatingsystem.create!(os_attributes)

should be changed to:

        ::Operatingsystem.find_or_create_by(os_attributes)
Comment 4 Bryan Kearney 2022-09-03 20:02:06 UTC 
Upstream bug assigned to pmoravec
Comment 5 Bryan Kearney 2022-09-03 20:02:08 UTC 
Upstream bug assigned to pmoravec
Comment 6 Bryan Kearney 2022-09-19 12:02:17 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35485 has been resolved.
Comment 13 errata-xmlrpc 2023-05-03 13:20:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.13 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2097