Bug 2025658
| Summary: | RFE - IDM does not provide a way to add a large quantity of users in 1-2 hours because the time to load a user increase significantly as the number of users increases. | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | toasty <wrydberg> |
| Component: | ipa | Assignee: | Florence Blanc-Renaud <frenaud> |
| Status: | CLOSED MIGRATED | QA Contact: | ipa-qe <ipa-qe> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.4 | CC: | abokovoy, dchen, dhan, jamison.bennett, pasik, rcritten, tscherf |
| Target Milestone: | rc | Keywords: | FutureFeature, MigratedToJIRA, Triaged |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-09-18 18:40:10 UTC | Type: | Story |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
toasty
2021-11-22 17:18:35 UTC
I'd recommend configuring the additional servers *after* the bulk load to save on all the replication traffic and overhead. I think that disabling the memberof plugin, and probably schema compat, will also improve load times. I started on an option to not do the final entry lookup as a general purpose option but it was full of corner cases and has been put on hold. Bulk load of other entry types (hosts, groups, sudo rules, DNS) have also been requested in the past so adding a skip just for user would be insufficient. Bulk loading is probably best done via an LDIF but getting it right can be tricky, particularly since an IPA user entry may be different depending on the system configuration. So there is no universal "plug your users in here" template we can provide currently. Is it possible to share the load_test.py script? Maybe additional improvements can be made there. > I'd recommend configuring the additional servers *after* the bulk load to save on all the replication traffic and overhead. This might be possible for the initial loading, but it would still be a challenge in cases where we have a bulk load at a later time. > I think that disabling the memberof plugin, and probably schema compat, will also improve load times. We have not tried that yet. We will have to evaluate this. > Bulk loading is probably best done via an LDIF but getting it right can be tricky, particularly since an IPA user entry may be different depending on the system configuration. So there is no universal "plug your users in here" template we can provide currently. We tried to load the users completely via LDIF but that didn't seem to be a supported method. The stage user can be added via LDIF, but then the activation appeared to require the API to be called rather than via LDIF (I think it does things like allocate IDs, generate keytabs, etc). > Is it possible to share the load_test.py script? Yes, the script is on github https://github.com/red-tux/perf-scripts/blob/7dc4d167896fba71afc79a28740d6482afbd4c4f/idm/load_test.py This script is just a simple simulation tool that we are using to evaluate performance and we have an application that uses FreeIPA APIs in a similar manner. Thanks We've investigated the memberof plugin and it appears to perform some actions inefficiently for a bulk loading case. 1. The members are sorted each time a set of members are added to determine which users have changed and need to be updated. This leads to sorting each time we call group-add-member, especially if the number of users added per request is small. 2. The memberOf attributes of each affected user are regenerated. Regeneration gets slower as the number of groups each user is a memberOf increases. These inefficiencies could potentially be avoided by creating an LDIF that updates both the groups member attribute and the users memberOf attribute. I haven't looked into the schema-compat plugin. This creates secondary user and group objects right? If we disable the schema-compat plugin then should our LDIF also create these compat user and group objects? While I know you said you don't have a template that we can plug our users into, we'd appreciate any guidance you can give on crafting appropriate LDIF. As Jamison mentioned, it appears that user-add or stageuser-activate perform many tasks that we want to make sure are covered by any approach we take. schema-compat plugin is read-only -- it reacts to actions done on the primary tree after the fact and regenerates its virtual views. If you are loading data, chances are that this particular server should not respond to compat tree requests at this time anyway, so disabling schema compat plugin is recommended. So your LDIF should not include anything for schema-compat. Once you'd re-enable the plugin, it will perform initialization and re-populate own tree. Again, this happens on every DS instance restart if the plugin is enabled because the tree represented by the plugin is virtual and has no storage other than in-memory. If you are not using cn=compat,$SUFFIX anywhere in your applications, you can disable the plugin permanently. The only issue that might bring is with SUDO using netgroups on systems that don't use relatively recent SSSD (1.14.2 or later, 2016+) because ou=sudoers,$SUFFIX is also emulated by the schema compat plugin. Both user_add and stageuser_activate operations require some modifications to disable adding to the default group. I think this particular part of the discussion can be separated into an upstream contribution that adds a deployment-wide configuration setting to skip the addition. However, user_add/stageuser_activate also perform a number of other changes that depend on the behavior of LDAP server plugins. If you want to repeat those in your LDIF, I'd recommend first to enable audit logging in 389-ds and then observe LDAP ADD operations these two commands issue. They'll have some attributes with specific values to trigger auto-generation by the LDAP server plugins FreeIPA provides. I do not have comments on memberof plugin behavior -- any optimization work for that plugin should be tracked with a separate bug to 389-ds itself. Alexander, Thanks for the information. Regarding the schema-compat plugin, how does it behave w.r.t. replication? In our use-case, we may need to make bulk user changes while the system is running, after the initial provisioning of users into the system. It sounds like, since schema-compat will run individually on each DS instance, schema-compat will be updating it's tree as the data is replicated and may slow down replication. Is this the case? We want to make sure that replication does not lag too badly. You recommend separating the user_add and stageuser_activate changes into a separate upstream contribution. What's the process for separating that work out? Should I create a new bug here? We'd prefer to go through these APIs if they perform well enough so that we can use supported mechanisms for add/mod/deleting users to future-proof us against any changes in idm, freeipa, or their plugins. Can the memberof plugin be disabled on a per-DS level or does it have to be disabled across the board? We will likely investigate FreeIPA topologies where user sync work communicates with one node (i.e., hidden master). Even if we manage the member and memberof attributes manually through LDIF, we would likely want to have the regular behavior available for users managed through other mechanisms. We have a handful of users that would fall into this category and want to evaluate the feasibility of using this type of topology. I'd appreciate some feedback on my questions above. On my list of things to do are: 1) Start investigating if we use cn=compat,$SUFFIX anywhere. 2) Enabling audit logging in 389-ds to see what other attributes are set when a user is added. 3) Attempt to craft LDIF for managing member and memberof attributes 4) Disabling the schema-compat and memberof plugins and evaluating user loading performance. David David,
schema-compat performs independently on each server. It maintains a virtual tree so there is a pure reaction to a change in the primary tree if a particular change affects any of the entries referenced in schema-compat configuration. This will increase amount of work done per each replicated operation. So if you'll find out you don't need schema compatibility plugin's work, it is worth to disable it. I think we will default to that upstream next year.
For user_add/stageuser_activate changes, we need a ticket upstream that will be connected to this bug. I'll open one later this week.
memberof plugin configuration is per-server. memberof changes aren't replicated by default, below is the full list of attributes which excluded from the replication:
# List of attributes that need to be excluded from replication initialization.
TOTAL_EXCLUDES = ('entryusn',
'krblastsuccessfulauth',
'krblastfailedauth',
'krbloginfailedcount')
# List of attributes that need to be excluded from normal replication.
EXCLUDES = ('memberof', 'idnssoaserial') + TOTAL_EXCLUDES
as a result, memberof plugin is always used to 'fix-up' the entries at each replica. If it is disabled, then it needs to be re-enabled after bulk load happened to perform the fixup.
I think your to-do list is sound, please let us know your results.
I created https://pagure.io/freeipa/issue/9064 to track automatic `ipausers` group membership upstream. Alexander, Thanks for the response. I'm going to be out through the holidays. I'll follow up again in January. David (In reply to toasty from comment #0) > Setup > > Create a 3 instance cluster of IDM with bi-directiional replication between > all instances.. For this example AWS instances were used using m5.large > instances (8 GB RAM, 2 VCPUs, gp2 SSD 100 GB size with 300 iops sustained > and a burst capacity of 3000 iops. The burst capacity was not depleted > during these tests nor was it depleted during the 100k user tests.) > It would be ideal if we could pass in an API paramter to the ipa CLI > tool/ipa API such that the default group memberhsip was skipped (this is not > needed) and the user is not looked up upon user activation (this also takes > significant time as the number of users grows). This would allow the bulk > users loading of 100k users to happen in 1-2 hours. According to: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/installing_identity_management/preparing-the-system-for-ipa-server-installation_installing-identity-management#hardware-recomendations_preparing-the-system-for-ipa-server-installation The recommend hardware for that is: For 100,000 users and 50,000 groups: at least 16 GB of RAM and 4 GB of swap space So they need to increase the RAM to 16 GB The issue is more complex than memory exhaustion. The idea of disabling memberof, compat and default users groups during bulk user loading are all ideas we want to explore. As the original case is closed, we are unable to obtain the mentioned load_test.py Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |