Description of problem: Searching for hosts by organization (and optionally specifying more filters for the search) does not terminate. Rather "Passenger RackApp: /usr/share/foreman" process consumes more and more memory until OOM-killed. This is a regression between 6.2.8 (search works fine) and 6.2.9 (search doesnt work). Version-Release number of selected component (if applicable): Sat 6.2.9 How reproducible: 100% Steps to Reproduce: 1. curl -u admin:redhat "https://$(hostname -f)/api/v2/hosts?organization_id=1&page=1&search=&sort_by=name&sort_order=ASC" (or in WebUI, goto Hosts -> All hosts and use filter "organization = Organization_Name") (you can try more complex query as well that shall fail the same way, like: curl -u admin:redhat "https://$(hostname -f)/api/v2/hosts?organization_id=1&page=1&search=subscription_status+%3D+invalid&sort_by=name&sort_order=ASC" ) 2. Wait a minute 3. Monitor RSS usage of "Passenger RackApp: /usr/share/foreman" process. Actual results: The query never finishes. RSS usage of the foreman process grows forever. Expected results: The query to finish in reasonable time. No huge RSS increase. Additional info: per tbrisker, the regression is suspected due to https://bugzilla.redhat.com/show_bug.cgi?id=1419667
Created redmine issue http://projects.theforeman.org/issues/19461 from this bug
FYI - those repeater steps failed for us. We were able to repeat this by running: curl -k -u <username>:<password> "<satelliteserver>/api/hosts?search=<hostname>" Applying this patch has resolved this issue for us.
I can confirm that searching for hostname on the host page kills Satellite 6.2.9. The patch fixed the issue for me too.
Hello, patch worked also for on Satellite 6.2.9 having the same issue
I've had similar issues with 6.2.9 and Ruby OOMing. Other manifestations were that I couldn't sort by "Last Checkin" with Content Hosts. The page would just remain blank.
(In reply to Ben from comment #17) > I've had similar issues with 6.2.9 and Ruby OOMing. Other manifestations > were that I couldn't sort by "Last Checkin" with Content Hosts. The page > would just remain blank. Unofficial, hence _unsupported_ hotfix procedure: wget https://patch-diff.githubusercontent.com/raw/Katello/katello/pull/6774.patch -O /tmp/6774.patch cksum /tmp/6774.patch (it shall return: 3957324473 1573 /tmp/6774.patch ) cd /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.124 cat /tmp/6774.patch | patch -p1 service httpd restart service foreman-tasks restart To revert it: cd /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.124 cat /tmp/6774.patch | patch -R -p1 service httpd restart service foreman-tasks restart
HOTFIX INSTRUCTIONS - SATELLITE 6.2.9 Install instructions: 1) Download the attached tar for your appropriate version. 2) Extract files xvf Hotfix-1447958-el*.tar.bz2 3) Stop services with katello-service stop 4) CD into the unzipped folder and yum localinstall * 5) Start services with katello-service start MD5 Sums for both tars contents are: EL6: 560ce4877e4cb98f46e0b446b816678a tfm-rubygem-foreman_openscap-0.5.3.19-2.BZ1447958.el6sat.noarch.rpm 6bdc9863c5a820d6f720c97e05a2b625 tfm-rubygem-foreman_openscap-doc-0.5.3.19-2.BZ1447958.el6sat.noarch.rpm 600cd611daca2902f5549fe1597886da tfm-rubygem-katello-3.0.0.125-1.el6sat.noarch.rpm d934aad6c482e0d8540808fdf912b87e tfm-rubygem-katello-doc-3.0.0.125-1.el6sat.noarch.rpm 6b07c447060e00bc497be099e4408eae tfm-rubygem-katello_ostree-3.0.0.125-1.el6sat.noarch.rpm EL7: c9eea0ef8283efbf93cab87c0922305c tfm-rubygem-foreman_openscap-0.5.3.19-2.BZ1447958.el7sat.noarch.rpm 7d0440806333fe59ff0217437494bd5b tfm-rubygem-foreman_openscap-doc-0.5.3.19-2.BZ1447958.el7sat.noarch.rpm 43cc9cd73f57c345570f2a8efbc4fba9 tfm-rubygem-katello-3.0.0.125-1.el7sat.noarch.rpm e5fb484f154350c2432ec6f3ea7f3426 tfm-rubygem-katello-doc-3.0.0.125-1.el7sat.noarch.rpm 0d2db3b936128dda4b5947b2fa65d879 tfm-rubygem-katello_ostree-3.0.0.125-1.el7sat.noarch.rpm
Created attachment 1277451 [details] EL6 Hotfix
Created attachment 1277452 [details] EL7 Hotfix
In order to improve passenger processes with leaked memory detection, I created small script which gracefully terminates all passenger processes with higher RSS memory consumption (2 GB by default): https://gist.github.com/lzap/8dddbe66ec8d43cbd4277c1de7045c17 This should prevent situations when Satellite Server goes crazy with dozen of Passenger processes consuming 10 GB memory each. We are considering merging this upstream, but in the meantime feel free to test the script at customers if you want. It is Satellite 6.2 compatible. Report bugs to me or here.
Verified in Satellite 6.2.9 Async Previously, this request would never completed and consumed ~28G ram (on the host satellite), before I killed it. Now, it successfully completes within 14s on a system with over 100K content hosts. -bash-4.2# time curl -k -u admin:<password> "https://rhsm-qe-1.rhq.lab.eng.bos.redhat.com/api/hosts?search=abcde" { "total": 101913, "subtotal": 5, "page": 1, "per_page": 20, "search": "abcde", "sort": { "by": null, "order": null }, "results": [{"ip":null,"environment_id":null,"environment_name":null,"last_report":null,"mac":null,"realm_id":null,"realm_name":null,"sp_mac":null,"sp_ip":null,"sp_name":null,"domain_id":null,"domain_name":null,"architecture_id":1,"architecture_name":"x86_64","operatingsystem_id":2,"operatingsystem_name":"RedHat 6.6","subnet_id":null,"subnet_name":null,"sp_subnet_id":null,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"build":false,"comment":null,"disk":null,"installed_at":null,"model_id":null,"hostgroup_id":null,"owner_id":1,"owner_type":"User","enabled":true,"puppet_ca_proxy_id":null,"managed":false,"use_image":null,"image_file":"","uuid":null,"compute_resource_id":null,"compute_resource_name":null,"compute_profile_id":null,"compute_profile_name":null,"capabilities":["build"],"provision_method":"build","puppet_proxy_id":null,"certname":"abcde","image_id":null,"image_name":null,"created_at":"2016-12-11 02:03:18 UTC","updated_at":"2016-12-11 02:04:53 UTC","last_compile":"2016-12-11 02:03:18 UTC","global_status":2,"global_status_label":"Error","organization_id":1,"organization_name":"Default Organization","location_id":2,"location_name":"Default Location","puppet_status":0,"model_name":null,"errata_status":3,"errata_status_label":"Security errata applicable","subscription_status":2,"subscription_status_label":"Unentitled","name":"abcde","id":100590,"hostgroup_name":null,"hostgroup_title":null,"content_facet_attributes":{"id":970,"uuid":"79037426-0a02-4cf0-ab69-30dcfd12724e","content_view_id":55,"content_view_name":"6 tools","lifecycle_environment_id":1,"lifecycle_environment_name":"Library","content_view":{"id":55,"name":"6 tools"},"lifecycle_environment":{"id":1,"name":"Library"},"errata_counts":{"security":0,"bugfix":0,"enhancement":0,"total":0}},"subscription_facet_attributes":{"id":100592,"uuid":"79037426-0a02-4cf0-ab69-30dcfd12724e","last_checkin":"2016-12-11 02:03:46 UTC","service_level":null,"release_version":null,"autoheal":true,"registered_at":"2016-12-11 02:03:20 UTC"},"content_host_id":100622},{"ip":null,"environment_id":null,"environment_name":null,"last_report":null,"mac":null,"realm_id":null,"realm_name":null,"sp_mac":null,"sp_ip":null,"sp_name":null,"domain_id":null,"domain_name":null,"architecture_id":1,"architecture_name":"x86_64","operatingsystem_id":2,"operatingsystem_name":"RedHat 6.6","subnet_id":null,"subnet_name":null,"sp_subnet_id":null,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"build":false,"comment":null,"disk":null,"installed_at":null,"model_id":null,"hostgroup_id":null,"owner_id":1,"owner_type":"User","enabled":true,"puppet_ca_proxy_id":null,"managed":false,"use_image":null,"image_file":"","uuid":null,"compute_resource_id":null,"compute_resource_name":null,"compute_profile_id":null,"compute_profile_name":null,"capabilities":["build"],"provision_method":"build","puppet_proxy_id":null,"certname":"abcdef","image_id":null,"image_name":null,"created_at":"2016-12-11 02:10:05 UTC","updated_at":"2016-12-11 02:11:43 UTC","last_compile":"2016-12-11 02:10:05 UTC","global_status":2,"global_status_label":"Error","organization_id":1,"organization_name":"Default Organization","location_id":2,"location_name":"Default Location","puppet_status":0,"model_name":null,"errata_status":3,"errata_status_label":"Security errata applicable","subscription_status":2,"subscription_status_label":"Unentitled","name":"abcdef","id":100591,"hostgroup_name":null,"hostgroup_title":null,"content_facet_attributes":{"id":971,"uuid":"07285514-3508-4b76-a2be-3a4f8e5428c6","content_view_id":55,"content_view_name":"6 tools","lifecycle_environment_id":1,"lifecycle_environment_name":"Library","content_view":{"id":55,"name":"6 tools"},"lifecycle_environment":{"id":1,"name":"Library"},"errata_counts":{"security":0,"bugfix":0,"enhancement":0,"total":0}},"subscription_facet_attributes":{"id":100593,"uuid":"07285514-3508-4b76-a2be-3a4f8e5428c6","last_checkin":"2016-12-11 02:10:33 UTC","service_level":null,"release_version":null,"autoheal":true,"registered_at":"2016-12-11 02:10:07 UTC"},"content_host_id":100623},{"ip":null,"environment_id":null,"environment_name":null,"last_report":null,"mac":null,"realm_id":null,"realm_name":null,"sp_mac":null,"sp_ip":null,"sp_name":null,"domain_id":null,"domain_name":null,"architecture_id":1,"architecture_name":"x86_64","operatingsystem_id":2,"operatingsystem_name":"RedHat 6.6","subnet_id":null,"subnet_name":null,"sp_subnet_id":null,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"build":false,"comment":null,"disk":null,"installed_at":null,"model_id":null,"hostgroup_id":null,"owner_id":1,"owner_type":"User","enabled":true,"puppet_ca_proxy_id":null,"managed":false,"use_image":null,"image_file":"","uuid":null,"compute_resource_id":null,"compute_resource_name":null,"compute_profile_id":null,"compute_profile_name":null,"capabilities":["build"],"provision_method":"build","puppet_proxy_id":null,"certname":"abcdefg","image_id":null,"image_name":null,"created_at":"2016-12-11 05:00:45 UTC","updated_at":"2017-01-16 21:49:52 UTC","last_compile":"2017-01-16 21:49:52 UTC","global_status":2,"global_status_label":"Error","organization_id":1,"organization_name":"Default Organization","location_id":2,"location_name":"Default Location","puppet_status":0,"model_name":null,"errata_status":3,"errata_status_label":"Security errata applicable","subscription_status":2,"subscription_status_label":"Unentitled","name":"abcdefg","id":100592,"hostgroup_name":null,"hostgroup_title":null,"content_facet_attributes":{"id":972,"uuid":"fe00d96d-9eb4-4896-b6a7-4c4cb23e7b24","content_view_id":55,"content_view_name":"6 tools","lifecycle_environment_id":1,"lifecycle_environment_name":"Library","content_view":{"id":55,"name":"6 tools"},"lifecycle_environment":{"id":1,"name":"Library"},"errata_counts":{"security":0,"bugfix":0,"enhancement":0,"total":0}},"subscription_facet_attributes":{"id":100594,"uuid":"fe00d96d-9eb4-4896-b6a7-4c4cb23e7b24","last_checkin":"2016-12-11 05:01:04 UTC","service_level":null,"release_version":null,"autoheal":true,"registered_at":"2016-12-11 05:00:47 UTC"},"content_host_id":100624},{"ip":null,"environment_id":null,"environment_name":null,"last_report":null,"mac":null,"realm_id":null,"realm_name":null,"sp_mac":null,"sp_ip":null,"sp_name":null,"domain_id":null,"domain_name":null,"architecture_id":null,"architecture_name":null,"operatingsystem_id":null,"operatingsystem_name":null,"subnet_id":null,"subnet_name":null,"sp_subnet_id":null,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"build":false,"comment":null,"disk":null,"installed_at":null,"model_id":null,"hostgroup_id":null,"owner_id":1,"owner_type":"User","enabled":true,"puppet_ca_proxy_id":null,"managed":false,"use_image":null,"image_file":"","uuid":null,"compute_resource_id":null,"compute_resource_name":null,"compute_profile_id":null,"compute_profile_name":null,"capabilities":["build"],"provision_method":"build","puppet_proxy_id":null,"certname":"virt-who-6bfd49c0.cce1.4503.b7a5.9abcde5ecb3e-1","image_id":null,"image_name":null,"created_at":"2016-12-10 16:42:21 UTC","updated_at":"2016-12-10 16:42:21 UTC","last_compile":null,"global_status":0,"global_status_label":"OK","organization_id":1,"organization_name":"Default Organization","location_id":2,"location_name":"Default Location","puppet_status":0,"model_name":null,"subscription_status":0,"subscription_status_label":"Fully entitled","name":"virt-who-6bfd49c0.cce1.4503.b7a5.9abcde5ecb3e-1","id":96117,"hostgroup_name":null,"hostgroup_title":null,"subscription_facet_attributes":{"id":96119,"uuid":"535b6737-845c-4f6d-acfa-2e6767e83e88","last_checkin":null,"service_level":null,"release_version":null,"autoheal":false,"registered_at":"2016-11-28 17:18:36 UTC"},"content_host_id":96149},{"ip":null,"environment_id":null,"environment_name":null,"last_report":null,"mac":null,"realm_id":null,"realm_name":null,"sp_mac":null,"sp_ip":null,"sp_name":null,"domain_id":null,"domain_name":null,"architecture_id":null,"architecture_name":null,"operatingsystem_id":null,"operatingsystem_name":null,"subnet_id":null,"subnet_name":null,"sp_subnet_id":null,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"build":false,"comment":null,"disk":null,"installed_at":null,"model_id":null,"hostgroup_id":null,"owner_id":1,"owner_type":"User","enabled":true,"puppet_ca_proxy_id":null,"managed":false,"use_image":null,"image_file":"","uuid":null,"compute_resource_id":null,"compute_resource_name":null,"compute_profile_id":null,"compute_profile_name":null,"capabilities":["build"],"provision_method":"build","puppet_proxy_id":null,"certname":"virt-who-e4af8830.dd09.4b21.9a42.44e82a17cb61-1","image_id":null,"image_name":null,"created_at":"2016-12-10 09:38:13 UTC","updated_at":"2016-12-10 09:38:13 UTC","last_compile":null,"global_status":0,"global_status_label":"OK","organization_id":1,"organization_name":"Default Organization","location_id":2,"location_name":"Default Location","puppet_status":0,"model_name":null,"subscription_status":0,"subscription_status_label":"Fully entitled","name":"virt-who-e4af8830.dd09.4b21.9a42.44e82a17cb61-1","id":65278,"hostgroup_name":null,"hostgroup_title":null,"subscription_facet_attributes":{"id":65280,"uuid":"2d5af70f-93c5-4397-bcf8-abcde6dc1b65","last_checkin":null,"service_level":null,"release_version":null,"autoheal":false,"registered_at":"2016-11-28 17:18:36 UTC"},"content_host_id":65310}] } real 0m14.050s user 0m0.065s sys 0m0.069s Much faster on a system with only a few hosts. -bash-4.2# time curl -k -u admin:<hosts> "https://ibm-x3550m3-10.lab.eng.brq.redhat.com/api/hosts?search=docker" { "total": 2, "subtotal": 1, "page": 1, "per_page": 20, "search": "docker", "sort": { "by": null, "order": null }, "results": [{"ip":null,"environment_id":null,"environment_name":null,"last_report":null,"mac":null,"realm_id":null,"realm_name":null,"sp_mac":null,"sp_ip":null,"sp_name":null,"domain_id":null,"domain_name":null,"architecture_id":1,"architecture_name":"x86_64","operatingsystem_id":2,"operatingsystem_name":"RedHat 7.2","subnet_id":null,"subnet_name":null,"sp_subnet_id":null,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"build":false,"comment":null,"disk":null,"installed_at":null,"model_id":null,"hostgroup_id":null,"owner_id":1,"owner_type":"User","enabled":true,"puppet_ca_proxy_id":null,"managed":false,"use_image":null,"image_file":"","uuid":null,"compute_resource_id":null,"compute_resource_name":null,"compute_profile_id":null,"compute_profile_name":null,"capabilities":["build"],"provision_method":"build","puppet_proxy_id":null,"certname":"docker","image_id":null,"image_name":null,"created_at":"2017-05-15 14:52:08 UTC","updated_at":"2017-05-15 14:52:24 UTC","last_compile":"2017-05-15 14:52:11 UTC","global_status":1,"global_status_label":"Warning","organization_id":1,"organization_name":"Default Organization","location_id":2,"location_name":"Default Location","puppet_status":0,"model_name":null,"errata_status":1,"errata_status_label":"Could not calculate errata status, ensure host is registered and katello-agent is installed","subscription_status":0,"subscription_status_label":"Fully entitled","name":"docker","id":2,"hostgroup_name":null,"hostgroup_title":null,"content_facet_attributes":{"id":1,"uuid":"9e38d8b3-32dd-4f3e-8a62-495335a95422","content_view_id":1,"content_view_name":"Default Organization View","lifecycle_environment_id":1,"lifecycle_environment_name":"Library","content_view":{"id":1,"name":"Default Organization View"},"lifecycle_environment":{"id":1,"name":"Library"},"errata_counts":{"security":0,"bugfix":0,"enhancement":0,"total":0}},"subscription_facet_attributes":{"id":1,"uuid":"9e38d8b3-32dd-4f3e-8a62-495335a95422","last_checkin":"2017-05-15 14:52:22 UTC","service_level":null,"release_version":null,"autoheal":true,"registered_at":"2017-05-15 14:52:08 UTC"},"content_host_id":1}] } real 0m0.985s user 0m0.054s sys 0m0.045s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1234
*** Bug 1544447 has been marked as a duplicate of this bug. ***