Red Hat Bugzilla – Bug 1630548
Available repositories from repository-set are incomplete or missing
Last modified: 2018-11-01 12:03:03 EDT
Description of problem: It seems a change in Satellite 6.3.3 introduced a strange issue with regards to incomplete/missing repositories from a repository-set. This does not appear to be an issue with the manifest itself. As an example, in the WebUI this would translate to [Content]->[Red Hat Repositories]->[RPMs]->[Red Hat Enterprise Linux 6 Server (RPMs)] Once expanded, there are repositories which are missing. To better illustrate this, hammer output shows something like the following: $ hammer repository-set available-repositories --product-id 148 --id 168 -----------------------------------------|--------|---------|---------------|-------- NAME | ARCH | RELEASE | REGISTRY NAME | ENABLED -----------------------------------------|--------|---------|---------------|-------- Red Hat Enterprise Linux 6 Server (RPMs) | i386 | 6.1 | | no Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.1 | | no Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.10 | | no Red Hat Enterprise Linux 6 Server (RPMs) | i386 | 6.10 | | no Red Hat Enterprise Linux 6 Server (RPMs) | i386 | 6.4 | | no Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.4 | | no Red Hat Enterprise Linux 6 Server (RPMs) | i386 | 6.5 | | no Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.5 | | no Red Hat Enterprise Linux 6 Server (RPMs) | i386 | 6.6 | | no Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.6 | | no Red Hat Enterprise Linux 6 Server (RPMs) | i386 | 6.7 | | no Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.7 | | no Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6Server | | no Red Hat Enterprise Linux 6 Server (RPMs) | i386 | 6Server | | no -----------------------------------------|--------|---------|---------------|-------- Notice the missing releases 6.2, 6.3, 6.8, 6.9... Subsequent requests at refreshing the available repositories may show that releases that were missing appear, while those that previously appeared disappear and vice versa. In any case, one can imagine that this is a huge inconvenience to say the least. Version-Release number of selected component (if applicable): Satellite 6.3.3 How reproducible: Always. Steps to Reproduce: 1. Import the attached manifest (or any manifest really) 2. List available repositories to be enabled via WebUI or hammer: $ hammer repository-set available-repositories --product-id 148 --id 168 3. Observe incomplete/missing repositories Actual results: Incomplete/missing repositories. Expected results: No incomplete/missing repositories. Additional info: Tentative workaround available in the comments.
Here's the exception that is getting thrown on the GET requests to the CDN: ActiveRecord::ConnectionTimeoutError could not obtain a database connection within 5.000 seconds (waited 5.000 seconds) /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:189:in `block in wait_poll' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:180:in `loop' /opt/rh/rh-ror42/root/usr/s hare/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:180:in `wait_poll' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/li b/active_record/connection_adapters/abstract/connection_pool.rb:135:in `block in poll' /opt/rh/rh-ruby23/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize' /opt/rh/rh-ror42/r oot/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:145:in `synchronize' /opt/rh/rh-ror42/root/usr/share/gems/gems/activereco rd-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:133:in `poll' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_ada pters/abstract/connection_pool.rb:425:in `acquire_connection' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_p ool.rb:349:in `block in checkout' /opt/rh/rh-ruby23/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active _record/connection_adapters/abstract/connection_pool.rb:348:in `checkout' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection' /opt/rh/rh-ruby23/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_handling.rb:113:in `retrieve_connection' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_handling.rb:87:in `connection' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:48:in `connection' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:980:in `create_binds' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:954:in `build_where' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:584:in `where!' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:574:in `where' /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/querying.rb:10:in `where' /usr/share/foreman/app/models/setting.rb:87:in `[]' /usr/share/foreman/lib/foreman/http_proxy.rb:4:in `http_proxy' /usr/share/foreman/lib/foreman/http_proxy.rb:13:in `proxy_http_request?' /usr/share/foreman/lib/foreman/http_proxy/net_http_extension.rb:7:in `request' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/resources/cdn.rb:96:in `block in get' /opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:853:in `start' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/resources/cdn.rb:95:in `get' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/resources/cdn.rb:126:in `fetch_substitutions' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/util/path_with_substitutions.rb:31:in `resolve_substitutions' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/util/cdn_var_substitutor.rb:62:in SNIP So this was caused by https://bugzilla.redhat.com/show_bug.cgi?id=1333595 What is happening is that katello is spinning up a thread to download each listing file and they are running all at once. Upon each request the foreman code checks the http_proxy setting to see if it should apply an http proxy to the GET request. It loads the setting from the database. As many as ~10-15 requests could happen at once. With a timeout of 5 seconds I would not expect this to cause a problem, as this setting should easily be able to be loaded 15 times within 5 seconds. Its possible this is worsened becuase all this is running inside dynflow, but even then I wouldn't expect this to occur... yet it is. This is why increasing the pool size helps, the queries for this setting are not waiting on pool slots to open up in order to load the setting. Possible solutions to the problem: 1) spawn fewer threads at at time (leading to slower responses for this data) 2) include some way to ignore this http proxy request for certain requests, since we already have a way to configure this proxy 3) Figure out why this is timing out? When i looked at the postgresql slow query logging, nothing showed up as being slower than 500ms.
*** Bug 1642205 has been marked as a duplicate of this bug. ***
*** Bug 1642925 has been marked as a duplicate of this bug. ***
Created redmine issue https://projects.theforeman.org/issues/25366 from this bug
Upstream bug assigned to stbenjam@redhat.com