Description of problem: Ruby, passenger-foreman memory is growing higher at scale. We have registered till 9K with katello-agent. have plans to go larger scale Ruby @ 50G, passenger-foreman @ 46G, Dynflow_executor @ 9G As shown in attached screenshots satellite is running on KVM/Libvirt VM. (24 vcpu, 64GB) PassengerMaxPoolSize 24 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Register hosts to satelite 2. 3. Actual results: Expected results: Additional info:
Created attachment 1267212 [details] ruby memory growth while running 9k hosts
Created attachment 1267214 [details] swapping while running 9k hosts
Created attachment 1267216 [details] dynflow_executor memory growth while running 9k hosts
Created attachment 1267217 [details] passenger-foreman memory growth while running 9k hosts
Created attachment 1267218 [details] mem growth (date && ps aux --sort -rss | head -n20)
Created attachment 1267222 [details] pgsql memory growth while running 9k hosts
For dynflow memory growth, please follow https://bugzilla.redhat.com/show_bug.cgi?id=1412307#c25 to collect more data about the garbage collection runs and memory allocation For passenger growth, I'm curious if this might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1434040 and if we could compare the behaviour after applying the patches from there.
Also, I would like to ask for export of the tasks related data to csv, using the script available in https://bugzilla.redhat.com/show_bug.cgi?id=1412307#c22, also for future uploads, as it contains very valuable data for further analysis.
While investigating some tasks that took too long (host update), I've noticed we were receiving a lot of docker interfaces from the hosts that run the containers and we spend a lot of time and memory when processing them. This might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1373903 Can you try adding 'veth*' to list of parsed interfaces in Provisioning -> "Ignore interfaces with matching identifier" (ignored_interface_identifiers) Also I recommend cleaning the docker interfaces from database: cat <<RUBY | foreman-rake console Nic::Interface.where("identifier like ?", 'veth%').each { |i| i.delete } RUBY I would be interested into seeing how the memory behaves after that. Another thing that might be related to this is https://bugzilla.redhat.com/show_bug.cgi?id=1435370
That magic with veth* interfaces worked nicely. Ruby RSS changed from about 34G to 19G. Thank you!
Given we've seen the improvement, I'm closing this bug as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1373903 and will propose for getting that for 6.2, to prevent this kind of hick-ups on customers running docker with sat *** This bug has been marked as a duplicate of bug 1373903 ***