Bug 1436918
| Summary: | Ruby, passenger-foreman memory is growing huge at scale. | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Pradeep Kumar Surisetty <psuriset> |
| Component: | Performance | Assignee: | satellite6-bugs <satellite6-bugs> |
| Status: | CLOSED DUPLICATE | QA Contact: | |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 6.2.8 | CC: | cduryee, inecas, jhutar, mmccune, pmoravec |
| Target Milestone: | Unspecified | Keywords: | Performance |
| Target Release: | Unused | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | scale_lab | ||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-03-30 14:55:55 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Attachments: | |||
|
Description
Pradeep Kumar Surisetty
2017-03-29 02:28:17 UTC
Created attachment 1267212 [details]
ruby memory growth while running 9k hosts
Created attachment 1267214 [details]
swapping while running 9k hosts
Created attachment 1267216 [details]
dynflow_executor memory growth while running 9k hosts
Created attachment 1267217 [details]
passenger-foreman memory growth while running 9k hosts
Created attachment 1267218 [details]
mem growth (date && ps aux --sort -rss | head -n20)
Created attachment 1267222 [details]
pgsql memory growth while running 9k hosts
For dynflow memory growth, please follow https://bugzilla.redhat.com/show_bug.cgi?id=1412307#c25 to collect more data about the garbage collection runs and memory allocation For passenger growth, I'm curious if this might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1434040 and if we could compare the behaviour after applying the patches from there. Also, I would like to ask for export of the tasks related data to csv, using the script available in https://bugzilla.redhat.com/show_bug.cgi?id=1412307#c22, also for future uploads, as it contains very valuable data for further analysis. While investigating some tasks that took too long (host update), I've noticed we were receiving a lot of docker interfaces from the hosts that run the containers and we spend a lot of time and memory when processing them. This might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1373903 Can you try adding 'veth*' to list of parsed interfaces in Provisioning -> "Ignore interfaces with matching identifier" (ignored_interface_identifiers) Also I recommend cleaning the docker interfaces from database: cat <<RUBY | foreman-rake console Nic::Interface.where("identifier like ?", 'veth%').each { |i| i.delete } RUBY I would be interested into seeing how the memory behaves after that. Another thing that might be related to this is https://bugzilla.redhat.com/show_bug.cgi?id=1435370 That magic with veth* interfaces worked nicely. Ruby RSS changed from about 34G to 19G. Thank you! Given we've seen the improvement, I'm closing this bug as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1373903 and will propose for getting that for 6.2, to prevent this kind of hick-ups on customers running docker with sat *** This bug has been marked as a duplicate of bug 1373903 *** |