Bug 1440825
| Summary: | Fact import code consumes lot of memory | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Chris Roberts <chrobert> |
| Component: | Puppet | Assignee: | satellite6-bugs <satellite6-bugs> |
| Status: | CLOSED ERRATA | QA Contact: | Katello QA List <katello-qa-list> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.2.8 | CC: | inecas, jcallaha, lzap, oshtaier |
| Target Milestone: | Unspecified | Keywords: | Triaged |
| Target Release: | Unused | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| URL: | http://projects.theforeman.org/issues/9016 | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-02-21 17:03:49 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Chris Roberts
2017-04-10 14:30:39 UTC
Thanks, fixed upstream, we need to backport. I can do the work, I made the upstream patch. QA NOTES: To reproduce slow fact loading, create million (or more) fact names. Use foreman-rake console to do this:
FactName.transaction do
(0..1000000).each do |x|
FactName.connection.execute
"INSERT INTO fact_names (name) values ('rand_fact_name_#{x}')"
end
end
Then upload facts using both discovery (discover node) or using puppet (register puppet client and perform puppet run). In the production.log the processing time should dropped from several (or tens) seconds to under a second. Process will also consume much less memory over time.
Typical fact import session is around a second:
2017-04-11 03:28:58 [app] [I] Started POST "/api/hosts/facts" for 10.8.106.1 at
2017-04-11 03:28:58 [audit] [I] [cruicble.toledo.satellite.lab.eng.rdu2.redhat.com] deleted 0 (38.3ms)
2017-04-11 03:28:58 [audit] [I] [cruicble.toledo.satellite.lab.eng.rdu2.redhat.com] updated 9 (251.6ms)
2017-04-11 03:28:58 [audit] [I] [cruicble.toledo.satellite.lab.eng.rdu2.redhat.com] added 0 (4.2ms)
2017-04-11 03:28:58 [app] [I] Import facts for 'cruicble.toledo.satellite.lab.eng.rdu2.redhat.com' completed. Added: 0, Updated: 9, Deleted 0 facts
2017-04-11 03:28:59 [app] [I] Completed 201 Created in 712ms (Views: 14.1ms | ActiveRecord: 135.8ms)
The way this was designed requires quite complex transaction, do not expect this to drop down to fractions of second. We'd need to redesign how facts are stored.
QA NOTES 2: To test how fact import performs do this command: # grep -A1 'Import facts for' /var/log/foreman/production.log | grep Completed The command does not show all hits but at least some. When testing this rember to fill database with fake fact names as in comment 4. This is how it looks before the patch: 2017-04-10 03:42:36 [app] [I] Completed 201 Created in 264ms (Views: 4.1ms | ActiveRecord: 93.6ms) 2017-04-10 03:50:37 [app] [I] Completed 201 Created in 2645ms (Views: 19.2ms | ActiveRecord: 377.8ms) 2017-04-10 03:54:15 [app] [I] Completed 201 Created in 5722ms (Views: 22.0ms | ActiveRecord: 920.4ms) 2017-04-10 03:56:46 [app] [I] Completed 201 Created in 1734ms (Views: 18.2ms | ActiveRecord: 110.8ms) 2017-04-10 03:59:18 [app] [I] Completed 201 Created in 2070ms (Views: 19.9ms | ActiveRecord: 156.3ms) 2017-04-10 04:08:15 [app] [I] Completed 200 OK in 7040ms (Views: 0.9ms | ActiveRecord: 1.0ms) 2017-04-10 04:26:13 [app] [I] Completed 201 Created in 21071ms (Views: 97.7ms | ActiveRecord: 3828.8ms) 2017-04-10 04:29:44 [app] [I] Completed 201 Created in 3627ms (Views: 24.3ms | ActiveRecord: 3008.5ms) 2017-04-10 04:35:01 [app] [I] Completed 201 Created in 373ms (Views: 16.0ms | ActiveRecord: 99.9ms) 2017-04-10 04:35:08 [app] [I] Completed 201 Created in 420ms (Views: 3.2ms | ActiveRecord: 262.5ms) After all requests should be less than a second *on average*: 2017-04-11 03:19:19 [app] [I] Completed 201 Created in 1132ms (Views: 18.4ms | ActiveRecord: 385.8ms) 2017-04-11 03:22:09 [app] [I] Completed 201 Created in 575ms (Views: 6.2ms | ActiveRecord: 127.4ms) 2017-04-11 03:28:59 [app] [I] Completed 201 Created in 712ms (Views: 14.1ms | ActiveRecord: 135.8ms) 2017-04-11 03:34:55 [app] [I] Completed 201 Created in 599ms (Views: 6.9ms | ActiveRecord: 111.3ms) 2017-04-11 03:35:02 [app] [I] Completed 201 Created in 439ms (Views: 7.2ms | ActiveRecord: 76.7ms) Chris, I deleted all interfaces on your libvirt host, it had thousands of them. This fixed the problem: Host.find(27).interfaces.destroy_all I suggest you to use "Ignore interfaces with matching identifier" to prevent this again. The system has currently 78 interfaces, I assume this will grow as you will be adding more VMs or dockers there (what kind of hypervisor is this?): Host.find(27).interfaces.map(&:identifier) => ["cfeature_el6", "dcapsule_el6", "dcapsule_el7", "downstream_el6", "downstream_el7", "em1.205", "em1.206", "em1.207", "em1.208", "em1.209", "em1.210", "em1.211", "em1.212", "em1.213", "em1.214", "em1.222", "em1.223", "em1.224", "em1.225", "em1.226", "em1.227", "em1.228", "em1.229", "em1.250", "em1.251", "em1.252", "em1.253", "em1.254", "em1.255", "em2", "em3", "em4", "internal_el6", "internal_el7", "katello_latest", "katello_nightly", "libvirt1.205", "libvirt1.206", "libvirt1.207", "libvirt1.208", "libvirt1.209", "libvirt1.210", "libvirt1.211", "libvirt1.212", "libvirt1.213", "libvirt1.214", "libvirt1.222", "libvirt1.223", "libvirt1.224", "libvirt1.225", "libvirt1.226", "libvirt1.227", "libvirt1.228", "libvirt1.229", "libvirt1.250", "libvirt1.251", "libvirt1.252", "libvirt1.253", "libvirt1.254", "libvirt1.255", "nightly_el6", "nightly_el7", "released_el6", "released_el7", "rhci", "sat6major_el6", "sat6major_el7", "sfeature_el6", "testingsat6_el6", "testingsat6_el7", "upgrade_el6", "upgrade_el7", "virbr0", "virbr0_nic", "virbr2", "virbr2_nic", "zstream_el6", "zstream_el7"] And I have reported this upstream: http://projects.theforeman.org/issues/19243 and http://projects.theforeman.org/issues/19244 These are trackers to fix this in more sane way. The patch for this BZ speeds things up and prevents memory from growing in other case that you experienced, not sure if we should backport. I guess not. Also part of the 6.3 there will be this issue: http://projects.theforeman.org/issues/20024 which optimized RHSM importer code to be more efficient on both memory and CPU. QA NOTES: MINIMUM TEST SCOPE: Verify there are no regression in Puppet and RHSM scope. BEST TEST SCOPE: Verify import of puppet and RHSM both works under heavy load. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:0336 |