Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1630548 - Available repositories from repository-set are incomplete or missing
Summary: Available repositories from repository-set are incomplete or missing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.3.3
Hardware: x86_64
OS: Linux
high
high
Target Milestone: 6.6.0
Assignee: Stephen Benjamin
QA Contact: Lai
URL:
Whiteboard:
: 1642205 1642925 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-19 00:31 UTC by Neal Kim
Modified: 2022-03-13 15:35 UTC (History)
8 users (show)

Fixed In Version: foreman-1.22.0-0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-22 12:46:44 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 25366 0 High Closed http_proxy code does not return db connection to pool quickly enough 2020-12-04 14:51:42 UTC
Red Hat Knowledge Base (Article) 3615591 0 None None None 2018-09-24 20:48:04 UTC
Red Hat Product Errata RHSA-2019:3172 0 None None None 2019-10-22 12:47:07 UTC

Description Neal Kim 2018-09-19 00:31:07 UTC 
Description of problem:

It seems a change in Satellite 6.3.3 introduced a strange issue with regards to incomplete/missing repositories from a repository-set. This does not appear to be an issue with the manifest itself.

As an example, in the WebUI this would translate to [Content]->[Red Hat Repositories]->[RPMs]->[Red Hat Enterprise Linux 6 Server (RPMs)]

Once expanded, there are repositories which are missing. To better illustrate this, hammer output shows something like the following:

$ hammer repository-set available-repositories --product-id 148 --id 168

-----------------------------------------|--------|---------|---------------|--------
NAME                                     | ARCH   | RELEASE | REGISTRY NAME | ENABLED
-----------------------------------------|--------|---------|---------------|--------
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.1     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.1     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.10    |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.10    |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.4     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.4     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.5     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.5     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.6     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.6     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.7     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.7     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6Server |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6Server |               | no     
-----------------------------------------|--------|---------|---------------|--------

Notice the missing releases 6.2, 6.3, 6.8, 6.9...

Subsequent requests at refreshing the available repositories may show that releases that were missing appear, while those that previously appeared disappear and vice versa. In any case, one can imagine that this is a huge inconvenience to say the least. 


Version-Release number of selected component (if applicable):

Satellite 6.3.3


How reproducible:

Always.


Steps to Reproduce:

1. Import the attached manifest (or any manifest really)
2. List available repositories to be enabled via WebUI or hammer:

$ hammer repository-set available-repositories --product-id 148 --id 168

3. Observe incomplete/missing repositories


Actual results:

Incomplete/missing repositories.


Expected results:

No incomplete/missing repositories.


Additional info:

Tentative workaround available in the comments.
Comment 3 Justin Sherrill 2018-09-19 01:03:10 UTC 
Here's the exception that is getting thrown on the GET requests to the CDN:

ActiveRecord::ConnectionTimeoutError could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)  


/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:189:in `block in 
wait_poll'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:180:in `loop'
/opt/rh/rh-ror42/root/usr/s
hare/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:180:in `wait_poll'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/li
b/active_record/connection_adapters/abstract/connection_pool.rb:135:in `block in poll'
/opt/rh/rh-ruby23/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ror42/r
oot/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:145:in `synchronize'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activereco
rd-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:133:in `poll'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_ada
pters/abstract/connection_pool.rb:425:in `acquire_connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_p
ool.rb:349:in `block in checkout'
/opt/rh/rh-ruby23/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active
_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/opt/rh/rh-ruby23/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_handling.rb:87:in `connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:48:in `connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:980:in `create_binds'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:954:in `build_where'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:584:in `where!'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:574:in `where'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/querying.rb:10:in `where'
/usr/share/foreman/app/models/setting.rb:87:in `[]'
/usr/share/foreman/lib/foreman/http_proxy.rb:4:in `http_proxy'
/usr/share/foreman/lib/foreman/http_proxy.rb:13:in `proxy_http_request?'
/usr/share/foreman/lib/foreman/http_proxy/net_http_extension.rb:7:in `request'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/resources/cdn.rb:96:in `block in get'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:853:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/resources/cdn.rb:95:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/resources/cdn.rb:126:in `fetch_substitutions'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/util/path_with_substitutions.rb:31:in `resolve_substitutions'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/util/cdn_var_substitutor.rb:62:in
SNIP



So this was caused by https://bugzilla.redhat.com/show_bug.cgi?id=1333595

What is happening is that katello is spinning up a thread to download each listing file and they are running all at once.  Upon each request the foreman code checks the http_proxy setting to see if it should apply an http proxy to the GET request.  It loads the setting from the database.  As many as ~10-15 requests could happen at once.  With a timeout of 5 seconds I would not expect this to cause a problem, as this setting should easily be able to be loaded 15 times within 5 seconds.  Its possible this is worsened becuase all this is running inside dynflow, but even then I wouldn't expect this to occur... yet it is.  

This is why increasing the pool size helps, the queries for this setting are not waiting on pool slots to open up in order to load the setting.  

Possible solutions to the problem:  

1) spawn fewer threads at at time (leading to slower responses for this data)
2) include some way to ignore this http proxy request for certain requests, since we already have a way to configure this proxy
3) Figure out why this is timing out?  When i looked at the postgresql slow query logging, nothing showed up as being slower than 500ms.
Comment 6 Brad Buckingham 2018-10-26 14:36:00 UTC 
*** Bug 1642205 has been marked as a duplicate of this bug. ***
Comment 7 Brad Buckingham 2018-10-26 21:03:25 UTC 
*** Bug 1642925 has been marked as a duplicate of this bug. ***
Comment 9 Stephen Benjamin 2018-11-01 15:07:11 UTC 
Created redmine issue https://projects.theforeman.org/issues/25366 from this bug
Comment 10 Satellite Program 2018-11-01 16:02:58 UTC 
Upstream bug assigned to stbenjam
Comment 11 Satellite Program 2018-11-01 16:03:03 UTC 
Upstream bug assigned to stbenjam
Comment 12 Satellite Program 2018-11-04 17:02:48 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25366 has been resolved.
Comment 13 vkaushik 2019-05-21 15:02:09 UTC 
Verified

@Satellite 6.6.0 snap-3
@Foreman 1.22.0-0.5


Verification Steps:
1. Import a manifest with some valid subscriptions in it (non-empty).
2. Check the available repositories in Redhat repositories from UI. Search for Red Hat Enterprise Linux 6 Server (RPMs)
2. Also check through hammer cli:
# hammer repository-set available-repositories --product-id 132 --organization-id 2 --id 168
3. Observe the available repositories.


Observation:
1. All available repositories are seen on UI.
2. Hammer cli output also shows all available repositories for the particular repository set.

# hammer repository-set available-repositories --product-id 132 --organization-id 2 --id 168
-----------------------------------------|--------|---------|--------
NAME                                     | ARCH   | RELEASE | ENABLED
-----------------------------------------|--------|---------|--------
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6Server | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.10    | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.9     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.8     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.7     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.6     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.5     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.4     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.3     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.2     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.1     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6Server | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.10    | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.9     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.8     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.7     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.6     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.5     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.4     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.3     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.2     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.1     | no     
-----------------------------------------|--------|---------|--------
Comment 17 errata-xmlrpc 2019-10-22 12:46:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3172
Note You need to log in before you can comment on or make changes to this bug.