Bug 1630548

Summary:	Available repositories from repository-set are incomplete or missing
Product:	Red Hat Satellite	Reporter:	Neal Kim <nkim>
Component:	Content Management	Assignee:	Stephen Benjamin <stbenjam>
Status:	CLOSED ERRATA	QA Contact:	Lai <ltran>
Severity:	high	Docs Contact:
Priority:	high
Version:	6.3.3	CC:	ajoseph, cmarinea, dsynk, egolov, gerben, gpulido, jsherril, rplevka
Target Milestone:	6.6.0	Keywords:	PrioBumpGSS, Regression, Triaged
Target Release:	Unused
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	foreman-1.22.0-0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-10-22 12:46:44 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Neal Kim 2018-09-19 00:31:07 UTC

Description of problem:

It seems a change in Satellite 6.3.3 introduced a strange issue with regards to incomplete/missing repositories from a repository-set. This does not appear to be an issue with the manifest itself.

As an example, in the WebUI this would translate to [Content]->[Red Hat Repositories]->[RPMs]->[Red Hat Enterprise Linux 6 Server (RPMs)]

Once expanded, there are repositories which are missing. To better illustrate this, hammer output shows something like the following:

$ hammer repository-set available-repositories --product-id 148 --id 168

-----------------------------------------|--------|---------|---------------|--------
NAME                                     | ARCH   | RELEASE | REGISTRY NAME | ENABLED
-----------------------------------------|--------|---------|---------------|--------
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.1     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.1     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.10    |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.10    |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.4     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.4     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.5     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.5     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.6     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.6     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.7     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.7     |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6Server |               | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6Server |               | no     
-----------------------------------------|--------|---------|---------------|--------

Notice the missing releases 6.2, 6.3, 6.8, 6.9...

Subsequent requests at refreshing the available repositories may show that releases that were missing appear, while those that previously appeared disappear and vice versa. In any case, one can imagine that this is a huge inconvenience to say the least. 


Version-Release number of selected component (if applicable):

Satellite 6.3.3


How reproducible:

Always.


Steps to Reproduce:

1. Import the attached manifest (or any manifest really)
2. List available repositories to be enabled via WebUI or hammer:

$ hammer repository-set available-repositories --product-id 148 --id 168

3. Observe incomplete/missing repositories


Actual results:

Incomplete/missing repositories.


Expected results:

No incomplete/missing repositories.


Additional info:

Tentative workaround available in the comments.

Comment 3 Justin Sherrill 2018-09-19 01:03:10 UTC

Here's the exception that is getting thrown on the GET requests to the CDN:

ActiveRecord::ConnectionTimeoutError could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)  


/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:189:in `block in 
wait_poll'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:180:in `loop'
/opt/rh/rh-ror42/root/usr/s
hare/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:180:in `wait_poll'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/li
b/active_record/connection_adapters/abstract/connection_pool.rb:135:in `block in poll'
/opt/rh/rh-ruby23/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ror42/r
oot/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:145:in `synchronize'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activereco
rd-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:133:in `poll'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_ada
pters/abstract/connection_pool.rb:425:in `acquire_connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_p
ool.rb:349:in `block in checkout'
/opt/rh/rh-ruby23/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active
_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/opt/rh/rh-ruby23/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_handling.rb:87:in `connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:48:in `connection'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:980:in `create_binds'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:954:in `build_where'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:584:in `where!'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/relation/query_methods.rb:574:in `where'
/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/querying.rb:10:in `where'
/usr/share/foreman/app/models/setting.rb:87:in `[]'
/usr/share/foreman/lib/foreman/http_proxy.rb:4:in `http_proxy'
/usr/share/foreman/lib/foreman/http_proxy.rb:13:in `proxy_http_request?'
/usr/share/foreman/lib/foreman/http_proxy/net_http_extension.rb:7:in `request'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/resources/cdn.rb:96:in `block in get'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:853:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/resources/cdn.rb:95:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/resources/cdn.rb:126:in `fetch_substitutions'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/util/path_with_substitutions.rb:31:in `resolve_substitutions'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.78/app/lib/katello/util/cdn_var_substitutor.rb:62:in
SNIP



So this was caused by https://bugzilla.redhat.com/show_bug.cgi?id=1333595

What is happening is that katello is spinning up a thread to download each listing file and they are running all at once.  Upon each request the foreman code checks the http_proxy setting to see if it should apply an http proxy to the GET request.  It loads the setting from the database.  As many as ~10-15 requests could happen at once.  With a timeout of 5 seconds I would not expect this to cause a problem, as this setting should easily be able to be loaded 15 times within 5 seconds.  Its possible this is worsened becuase all this is running inside dynflow, but even then I wouldn't expect this to occur... yet it is.  

This is why increasing the pool size helps, the queries for this setting are not waiting on pool slots to open up in order to load the setting.  

Possible solutions to the problem:  

1) spawn fewer threads at at time (leading to slower responses for this data)
2) include some way to ignore this http proxy request for certain requests, since we already have a way to configure this proxy
3) Figure out why this is timing out?  When i looked at the postgresql slow query logging, nothing showed up as being slower than 500ms.

Comment 6 Brad Buckingham 2018-10-26 14:36:00 UTC

*** Bug 1642205 has been marked as a duplicate of this bug. ***

Comment 7 Brad Buckingham 2018-10-26 21:03:25 UTC

*** Bug 1642925 has been marked as a duplicate of this bug. ***

Comment 9 Stephen Benjamin 2018-11-01 15:07:11 UTC

Created redmine issue https://projects.theforeman.org/issues/25366 from this bug

Comment 10 Satellite Program 2018-11-01 16:02:58 UTC

Upstream bug assigned to stbenjam

Comment 11 Satellite Program 2018-11-01 16:03:03 UTC

Upstream bug assigned to stbenjam

Comment 12 Satellite Program 2018-11-04 17:02:48 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25366 has been resolved.

Comment 13 vkaushik 2019-05-21 15:02:09 UTC

Verified

@Satellite 6.6.0 snap-3
@Foreman 1.22.0-0.5


Verification Steps:
1. Import a manifest with some valid subscriptions in it (non-empty).
2. Check the available repositories in Redhat repositories from UI. Search for Red Hat Enterprise Linux 6 Server (RPMs)
2. Also check through hammer cli:
# hammer repository-set available-repositories --product-id 132 --organization-id 2 --id 168
3. Observe the available repositories.


Observation:
1. All available repositories are seen on UI.
2. Hammer cli output also shows all available repositories for the particular repository set.

# hammer repository-set available-repositories --product-id 132 --organization-id 2 --id 168
-----------------------------------------|--------|---------|--------
NAME                                     | ARCH   | RELEASE | ENABLED
-----------------------------------------|--------|---------|--------
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6Server | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.10    | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.9     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.8     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.7     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.6     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.5     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.4     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.3     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.2     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | x86_64 | 6.1     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6Server | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.10    | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.9     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.8     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.7     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.6     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.5     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.4     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.3     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.2     | no     
Red Hat Enterprise Linux 6 Server (RPMs) | i386   | 6.1     | no     
-----------------------------------------|--------|---------|--------

Comment 17 errata-xmlrpc 2019-10-22 12:46:44 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3172