Bug 2087115

Summary: Foreman inventory sync with Ansible Tower is five times slower when non-admin user with custom role\permission is used and Reports API is enabled
Product: Red Hat Satellite Reporter: Sayan Das <saydas>
Component: ReportingAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED MIGRATED QA Contact: Lukáš Hellebrandt <lhellebr>
Severity: high Docs Contact:
Priority: medium    
Version: 6.9.9CC: ahumbe, egolov, mhulan, myllynen, paji, paul.porcelli, rlavi
Target Milestone: UnspecifiedKeywords: MigratedToJIRA, Performance, Reopened, Triaged
Target Release: Unused   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-06-06 12:21:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sayan Das 2022-05-17 11:12:49 UTC
Description of problem:

Following these resources, we can configure Ansible Tower to Talk to satellite and use the satellite data as the Inventory source to create dynamic inventory.

Satellite and Ansible Tower integration part 1: Inventory integration
https://www.redhat.com/en/blog/satellite-and-ansible-tower-integration-part-1-inventory-integration

Use Satellite 6 as an Inventory Source in Ansible Tower
https://www.ansible.com/blog/use-satellite-6-as-an-inventory-source-in-ansible-tower

Now, if I configure everything and set up a non-admin user in Satellite with correct roles and permissions, then Ansible Tower will be able to use that user in the Source of Foreman Inventory but will take a huge amount of time to Load and Import all Host and Hostgroups information from Satellite.

The same thing can be done with "admin" or an Administrator-level user of Satellite within five times less amount of time.

This is a significant decrease in performance and It seems the non-admin user is taking a huge amount of time to generate the report from the "Ansible - Ansible Inventory" template


Version-Release number of selected component (if applicable):

Satellite 6.9.9 

How reproducible:

Always with Customer DB restored 

Steps to Reproduce:

Check internal comments for the reproducer details.


Actual results:

With non-admin user:

** Default i.e. Same config as the customer where batch_size is set to 250

~~~
 1489.542 INFO     Processing JSON output...
 1489.999 INFO     Loaded 306 groups, 3565 hosts
 1529.132 INFO     Inventory import completed for sat69_inventory_source in 1524.8s
~~~
 
 i.e. 25.4 minutes


** the same config as the customer but the batch_size is increased to 500

~~~
 1464.983 INFO     Loaded 306 groups, 3565 hosts
 1497.332 INFO     Inventory import completed for sat69_inventory_source in 1492.6s
~~~
 
 i.e. 24.87 minutes

Obviously, the batch_size is irrelevant in the case of Reports API but the duration stays between 25 - 27 mins to complete the sync.


Now, With the "admin" user of the satellite:

~~~
  241.946 INFO     Processing JSON output...
  242.422 INFO     Loaded 306 groups, 3565 hosts
  275.385 INFO     Inventory import completed for sat69_inventory_source in 271.1s
~~~

4.5 mins which are five times less than what it took for the non-admin user. 



Expected results:

Non-Admin users with correct roles\permissions should be able to get the Inventory Report generated within 4 - 5 mins


Additional info:

NA

Comment 3 Evgeni Golov 2022-05-17 14:51:49 UTC
Here is a slightly simpler reproducer, even if with way lower times (most probably due to less data attached to the generated hosts).

1. Deploy a Satellite (duh!)
2. Install ansible-collection-redhat-satelite (this is not installed by default on older setups)
3. Create an "ansible" user for pulling the report
   hammer user create --login ansible --password ansible --location 'Default Location' --organization 'Default Organization' --auth-source Internal --mail ansible@localhost
   hammer user add-role --login ansible --role 'Ansible Tower Inventory Reader'
   hammer user add-role --login ansible --role 'View hosts'
   hammer user add-role --login ansible --role 'Viewer'
4. Create 2000 fake hosts
   foreman-rake console

os = Operatingsystem.first
arch = os.architectures.first
domain = Domain.first
org = Organization.first
loc = Location.first
owner = User.anonymous_admin

2000.times.each do |_|
  h = Host::Managed.new(operatingsystem: os, architecture: arch, domain: domain, organization: org, location: loc, owner: owner)
  h.save
end

5. Create an inventory configuration my.foreman.yml (or similar, needs to end in "foreman.yaml")
plugin: redhat.satellite.foreman
url: https://satellite.example.com
validate_certs: false
user: ansible
password: ansible

use_reports_api: true
max_timeout: 3600 
group_prefix: hg_
want_hostcollections: true
want_params: true
legacy_hostvars: true

6. Call "time ansible-inventory -i my.foreman.yaml --list"

On my setup, I get ~45sec for requesting this with "admin" and ~95sec with "ansible" users.
It seems to be stable across 6.9, 6.10 and 6.11 that I have tested here.

Comment 4 Evgeni Golov 2022-05-17 14:54:56 UTC
> It seems to be stable across 6.9, 6.10 and 6.11 that I have tested here.

6.11 on EL8 does perform better (26s/47s), but it does run on different (albeit actually less performant) HW.

Comment 6 Evgeni Golov 2022-05-19 13:50:58 UTC
I've merged a small improvement in https://github.com/theforeman/foreman/pull/9223 for the "as admin, it still feels slow"

Comment 16 Eric Helms 2024-06-06 12:21:45 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "SAT-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.