Bug 2033593

Summary: fact_values api performance issues when loading a large number of facts
Product: Red Hat Satellite Reporter: Kenny Tordeurs <ktordeur>
Component: FactAssignee: Tomer Brisker <tbrisker>
Status: CLOSED ERRATA QA Contact: Jan Hutaƙ <jhutar>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.9.7CC: jhutar, lvrtelov, mhulan, pcreech, peter.vreman, zhunting
Target Milestone: 6.11.0Keywords: Performance, Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-05 14:31:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenny Tordeurs 2021-12-17 11:31:23 UTC
Description of problem:
The fact_values api called loaded a lot of un-needed data from the database and used a sub-optimal way of converting the data to a
hash for display.

Version-Release number of selected component (if applicable):
Satellite 6.9.7

How reproducible:
100%

Steps to Reproduce:
Load all facts from hosts.
https://sat-fqdn/apidoc/v2/fact_values

Actual results:
Sat6 instance with 300 hosts
~~~
2021-11-25T17:51:57 [I|app|d7f0a4e0] Started GET "/api/v2/fact_values?per_page=9999999" for 10.208.172.104 at 2021-11-25 17:51:57 +0000
2021-11-25T17:51:57 [I|app|d7f0a4e0] Processing by Api::V2::FactValuesController#index as JSON
2021-11-25T17:51:57 [I|app|d7f0a4e0]   Parameters: {"per_page"=>"9999999", "apiv"=>"v2", "fact_value"=>{}}
2021-11-25T17:51:57 [I|app|d7f0a4e0] Authorized user testing(testing )
2021-11-25T17:52:35 [I|app|d7f0a4e0]   Rendering api/v2/fact_values/index.json.rabl within api/v2/layouts/index_layout
2021-11-25T17:52:36 [I|app|d7f0a4e0]   Rendered api/v2/fact_values/index.json.rabl within api/v2/layouts/index_layout (Duration: 206.9ms | Allocations: 237490)
2021-11-25T17:52:36 [I|app|d7f0a4e0] Completed 200 OK in 38748ms (Views: 219.0ms | ActiveRecord: 421.2ms | Allocations: 5964354)
~~~

Sat6 instance with 75 hosts same can be observed, althugh 6 seconds instead of 40 seconds
~~~
2021-11-25T17:52:36 [I|app|99aecc01] Started GET "/api/v2/fact_values?per_page=9999999" for 10.208.172.104 at 2021-11-25 17:52:36 +0000
2021-11-25T17:52:36 [I|app|99aecc01] Processing by Api::V2::FactValuesController#index as JSON
2021-11-25T17:52:36 [I|app|99aecc01]   Parameters: {"per_page"=>"9999999", "apiv"=>"v2", "fact_value"=>{}}
2021-11-25T17:52:36 [I|app|99aecc01] Authorized user testing(testing )
2021-11-25T17:52:42 [I|app|99aecc01]   Rendering api/v2/fact_values/index.json.rabl within api/v2/layouts/index_layout
2021-11-25T17:52:42 [I|app|99aecc01]   Rendered api/v2/fact_values/index.json.rabl within api/v2/layouts/index_layout (Duration: 67.5ms | Allocations: 89770)
2021-11-25T17:52:42 [I|app|99aecc01] Completed 200 OK in 6062ms (Views: 76.6ms | ActiveRecord: 259.8ms | Allocations: 2226318)
~~~

Expected results:
Better performance

Additional info:
All facts are requested in a single query for performance reasons.

Comment 1 Kenny Tordeurs 2021-12-17 12:12:39 UTC
Before the patch from  https://github.com/theforeman/foreman/pull/9000.patch

# curl -X GET -s -k -u admin:pass https://ktordeur-satellite.example.lan/api/v2/fact_values\?per_page\=99999999 | python -m json.tool 
~~~
2021-12-17T12:45:47 [I|app|85ed1fcd] Started GET "/api/v2/fact_values?per_page=99999999" for 10.40.194.212 at 2021-12-17 12:45:47 +0100
2021-12-17T12:45:47 [I|app|85ed1fcd] Processing by Api::V2::FactValuesController#index as JSON
2021-12-17T12:45:47 [I|app|85ed1fcd]   Parameters: {"per_page"=>"99999999", "apiv"=>"v2"}
2021-12-17T12:45:47 [I|app|85ed1fcd] Authorized user admin(Admin User)
2021-12-17T12:45:48 [I|app|85ed1fcd]   Rendering api/v2/fact_values/index.json.rabl within api/v2/layouts/index_layout
2021-12-17T12:45:48 [I|app|85ed1fcd]   Rendered api/v2/fact_values/index.json.rabl within api/v2/layouts/index_layout (Duration: 23.2ms | Allocations: 12917)
2021-12-17T12:45:48 [I|app|85ed1fcd] Completed 200 OK in 650ms (Views: 42.1ms | ActiveRecord: 62.3ms | Allocations: 212369)
~~~

After the patch:
~~~
2021-12-17T13:11:47 [I|app|82f50291] Started GET "/api/v2/fact_values?per_page=99999999" for 10.44.129.52 at 2021-12-17 13:11:47 +0100
2021-12-17T13:11:47 [I|app|82f50291] Processing by Api::V2::FactValuesController#index as JSON
2021-12-17T13:11:47 [I|app|82f50291]   Parameters: {"per_page"=>"99999999", "apiv"=>"v2"}
2021-12-17T13:11:47 [I|app|82f50291] Authorized user admin(Admin User)
2021-12-17T13:11:47 [I|app|82f50291]   Rendering api/v2/fact_values/index.json.rabl within api/v2/layouts/index_layout
2021-12-17T13:11:47 [I|app|82f50291]   Rendered api/v2/fact_values/index.json.rabl within api/v2/layouts/index_layout (Duration: 11.0ms | Allocations: 12880)
2021-12-17T13:11:47 [I|app|82f50291] Completed 200 OK in 243ms (Views: 18.1ms | ActiveRecord: 32.5ms | Allocations: 138916)
~~~

Comment 2 Bryan Kearney 2021-12-18 20:06:58 UTC
Upstream bug assigned to tbrisker

Comment 3 Bryan Kearney 2021-12-18 20:06:59 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34160 has been resolved.

Comment 4 Peter Vreman 2021-12-20 17:04:49 UTC
Thanks for the quick follow-up and preparing a code fix

I applied the patch on 2 sat6 instances one smaller running 6.9.7 and larger one running 6.9.6 and can confirm it gives a big improvement:

The 6.9.7 instance has 45000 facts goes from ~7sec -> ~1.5sec   allocations  ~6000000 -> 4000000
The 6.9.6 instance has 125000 facts goes from ~40sec -> ~5sec   allocations 2200000 -> 1500000

Comment 10 errata-xmlrpc 2022-07-05 14:31:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498