Description of problem: While testing concurrent registration of systems in SatPerf, we've had found a lot of failures like the following: - Content_host: - Registration log: ~~~ # # Running registration # This system is currently not registered. All local data removed Updating Subscription Management repositories. Unable to read consumer identity This system is not registered with an entitlement server. You can use subscription-manager to register. No match for argument: katello-ca-consumer* No packages marked for removal. Dependencies resolved. Nothing to do. Complete! Updating Subscription Management repositories. Unable to read consumer identity This system is not registered with an entitlement server. You can use subscription-manager to register. Error: There are no enabled repositories in "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d". Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information. ~~~ - /var/log/rhsm/rhsm.log ~~~ [root@containerhost-6-container56 ~]# cat /var/log/rhsm/rhsm.log . . . 2023-04-20 10:43:26,458 [ERROR] subscription-manager:402:MainThread @managercli.py:229 - Error during registration: The read operation timed out 2023-04-20 10:43:26,459 [ERROR] subscription-manager:402:MainThread @managercli.py:230 - The read operation timed out Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 1992, in _do_command service_level=self.options.service_level, File "/usr/lib64/python3.6/site-packages/rhsmlib/services/register.py", line 111, in register jwt_token=jwt_token File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 1154, in registerConsumer return self.conn.request_post(url, params, headers=headers) File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 937, in request_post return self._request("POST", method, params, headers=headers) File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 967, in _request info=info, headers=headers, cert_key_pairs=cert_key_pairs) File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 749, in _request response = conn.getresponse() File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse response.begin() File "/usr/lib64/python3.6/http/client.py", line 320, in begin version, status, reason = self._read_status() File "/usr/lib64/python3.6/http/client.py", line 281, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib64/python3.6/socket.py", line 586, in readinto return self._sock.recv_into(b) File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into return self.read(nbytes, buffer) File "/usr/lib64/python3.6/ssl.py", line 833, in read return self._sslobj.read(len, buffer) File "/usr/lib64/python3.6/ssl.py", line 590, in read v = self._sslobj.read(len, buffer) socket.timeout: The read operation timed out . . . ~~~ - Satellite server: ~~~ [root@satellite ~]# grep -r -e 2023-04-20T10:4 -e '20/Apr/2023:10:4' /var/log/ | grep -e 172.21.49.253 -e containerhost-6-container56 -e 7ee9b255 /var/log/messages:Apr 20 10:42:53 satellite pulpcore-api[129890]: pulp [7ee9b255-b414-424e-9658-52f47d5cfc3e]: - - [20/Apr/2023:10:42:53 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.6p219" /var/log/messages:Apr 20 10:42:53 satellite pulpcore-api[129890]: pulp [7ee9b255-b414-424e-9658-52f47d5cfc3e]: - - [20/Apr/2023:10:42:53 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4206 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.6p219" /var/log/messages:Apr 20 10:42:53 satellite pulpcore-api[129890]: pulp [7ee9b255-b414-424e-9658-52f47d5cfc3e]: - - [20/Apr/2023:10:42:53 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.6p219" /var/log/messages:Apr 20 10:42:53 satellite pulpcore-api[129890]: pulp [7ee9b255-b414-424e-9658-52f47d5cfc3e]: - - [20/Apr/2023:10:42:53 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 4206 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.6p219" /var/log/foreman/production.log:2023-04-20T10:42:53 [I|app|7ee9b255] Started POST "/rhsm/consumers?owner=Default_Organization&activation_keys=AK_RHEL8" for 172.21.49.127 at 2023-04-20 10:42:53 +0000 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|app|7ee9b255] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON /var/log/foreman/production.log:2023-04-20T10:42:53 [I|app|7ee9b255] Parameters: {"type"=>"system", "name"=>"containerhost-6-container56.red.ddns.perf.redhat.com", "facts"=>"[FILTERED]", "installedProducts"=>[{"productId"=>"479", "productName"=>"Red Hat Enterprise Linux for x86_64", "version"=>"8.7", "arch"=>"x86_64"}], "contentTags"=>["rhel-8-x86_64", "rhel-8"], "role"=>"", "addOns"=>[], "usage"=>"", "serviceLevel"=>"", "owner"=>"Default_Organization", "activation_keys"=>"AK_RHEL8"} /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on mac /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on ip /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on type Nic::Managed /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on name containerhost-6-container56.red.ddns.perf.redhat.com /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on host_id 20238 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on subnet_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on domain_id 1 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on attrs {} /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on provider /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on username /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on password [redacted] /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on virtual false /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on link true /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on identifier /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on tag /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on attached_to /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on managed true /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on mode balance-rr /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on attached_devices /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on bond_options /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on primary true /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on provision true /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on compute_attributes {} /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on execution true /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on ip6 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Nic::Managed (20250) create event on subnet6_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on name containerhost-6-container56.red.ddns.perf.redhat.com /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on root_pass /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on architecture_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on operatingsystem_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on ptable_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on medium_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on build false /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on comment /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on disk /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on installed_at /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on model_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on hostgroup_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on owner_id 1 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on owner_type User /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on enabled true /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on puppet_ca_proxy_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on managed false /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on use_image /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on image_file /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on uuid /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on compute_resource_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on puppet_proxy_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on certname /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on image_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on organization_id 1 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on location_id 2 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on otp /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on realm_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on compute_profile_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on provision_method /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on grub_pass /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on discovery_rule_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on openscap_proxy_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on pxe_loader /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on initiated_at /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on build_errors /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Host::Base (20238) create event on ansible_role_ids [] /var/log/foreman/production.log:2023-04-20T10:42:53 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: host_created.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on host_id 20238 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on uuid 3ce52096-9256-4956-84ca-0cb884433867 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on content_view_id 2 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on lifecycle_environment_id 2 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on kickstart_repository_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on content_source_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on installable_security_errata_count 0 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on installable_enhancement_errata_count 0 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on installable_bugfix_errata_count 0 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on applicable_rpm_count 0 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on upgradable_rpm_count 0 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on applicable_module_stream_count 0 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on upgradable_module_stream_count 0 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on applicable_deb_count 0 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::ContentFacet (20234) create event on upgradable_deb_count 0 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on host_id 20238 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on uuid 3ce52096-9256-4956-84ca-0cb884433867 /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on service_level /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on release_version /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on autoheal false /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on registered_at 2023-04-18 20:15:16 UTC /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on registered_through /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on user_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on hypervisor false /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on hypervisor_host_id /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on purpose_usage /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on purpose_role /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on dmi_uuid /var/log/foreman/production.log:2023-04-20T10:42:53 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) create event on pool_ids [] /var/log/foreman/production.log:2023-04-20T10:42:53 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: host_updated.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:42:53 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: host_updated.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:43:22 [I|app|7ee9b255] Import facts for 'containerhost-6-container56.red.ddns.perf.redhat.com' completed. Added: 99, Updated: 0, Deleted 0 facts /var/log/foreman/production.log:2023-04-20T10:43:22 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: host_updated.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:43:23 [I|aud|7ee9b255] Nic::Managed (20250) update event on mac , e6:1b:32:15:54:3e /var/log/foreman/production.log:2023-04-20T10:43:23 [I|aud|7ee9b255] Nic::Managed (20250) update event on identifier , eth0 /var/log/foreman/production.log:2023-04-20T10:43:25 [I|aud|7ee9b255] Host::Base (20238) update event on architecture_id , 1 /var/log/foreman/production.log:2023-04-20T10:43:25 [I|aud|7ee9b255] Host::Base (20238) update event on operatingsystem_id , 1 /var/log/foreman/production.log:2023-04-20T10:43:25 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: host_updated.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:43:27 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: host_updated.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:43:30 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) update event on autoheal false, true /var/log/foreman/production.log:2023-04-20T10:43:30 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) update event on registered_at 2023-04-18 20:15:16 UTC, 2023-04-20 10:43:10 UTC /var/log/foreman/production.log:2023-04-20T10:43:30 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: host_updated.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:43:31 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: status_changed.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:43:32 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: host_updated.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:43:32 [I|app|7ee9b255] ForemanWebhooks::EventSubscriber: host_updated.event.foreman event received /var/log/foreman/production.log:2023-04-20T10:43:32 [I|aud|7ee9b255] Katello::Host::SubscriptionFacet (20237) update event on registered_through , satellite.red.ddns.perf.redhat.com /var/log/foreman/production.log:2023-04-20T10:43:33 [I|app|7ee9b255] Completed 200 OK in 39872ms (Views: 2.2ms | ActiveRecord: 11734.1ms | Allocations: 3772374) /var/log/httpd/foreman_access.log:172.21.49.253 - - [20/Apr/2023:10:40:05 +0000] "GET /pub/host-registration.sh HTTP/1.1" 304 - "-" "ansible-httpget" /var/log/httpd/foreman_access.log:172.21.49.253 - - [20/Apr/2023:10:49:07 +0000] "GET /pub/host-registration.sh HTTP/1.1" 304 - "-" "ansible-httpget" ~~~ Version-Release number of selected component (if applicable): Satelite 6.13 Snap 18.0 How reproducible: Try to register as much systems as possible concurrently Steps to Reproduce: 1. Run registration script generated on the Satellite server: curl -sS --insecure 'https://$SAT_FQDN/register?activation_keys=$AK&force=true&organization_id=1' -H 'Authorization: Bearer $BEARER' | bash 2. 3. Actual results: Registration fails Expected results: Registration succeeds Additional info: I can share sosreport, production.log and foreman-ssl* logs privately.
Hi, do you have the satellite/capsule log with the actual error? The attached Satellite log doesn't have any error so it's hard to say what's the root cause.
Hi Leos, I'll close this bug because I've been unable to reproduce it again.