Bug 1102739
| Summary: | pcsd extensive memory usage | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Radek Steiger <rsteiger> |
| Component: | pcs | Assignee: | Chris Feist <cfeist> |
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | cluster-qe <cluster-qe> |
| Severity: | medium | Docs Contact: | |
| Priority: | low | ||
| Version: | 7.0 | CC: | cluster-maint, rsteiger, tojeline |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-01 09:53:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I've done a couple more tests on a freshly set up clusters with 300 running resources and a browser left open for ~16 hours. Setup was following: 1) created a 4-node cluster 2) disabled stonith (as no fencing was configured) 3) rebooted all nodes to get all machines into a fresh state 4) logged into the GUI and opened a Resources tab 5) created 300 dummy resources using CLI Situation after ~16 hours with the Resources tab open all the time: Cluster I (physical machines): [root@west-13 /]# top -b -n1 -p $(pidof ruby),$(pidof cib),$(pidof corosync) -c | grep "^ " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 785 root 20 0 1702220 714064 5152 S 0.0 8.8 256:56.58 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb 1266 haclust+ 20 0 135844 45964 11476 R 92.3 0.6 80:30.15 /usr/libexec/pacemaker/cib 1032 root 20 0 198956 48616 34176 S 0.0 0.6 10:09.07 corosync [root@west-14 ~]# top -b -n1 -p $(pidof ruby),$(pidof cib),$(pidof corosync) -c | grep "^ " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 782 root 20 0 699844 289600 5016 S 32.4 3.6 234:43.48 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb 1255 haclust+ 20 0 126036 38360 9828 S 0.0 0.5 81:31.55 /usr/libexec/pacemaker/cib 1012 root 20 0 197964 48368 34176 S 0.0 0.6 6:48.71 corosync [root@west-15 ~]# top -b -n1 -p $(pidof ruby),$(pidof cib),$(pidof corosync) -c | grep "^ " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 795 root 20 0 969908 387320 5016 S 0.0 4.8 239:07.46 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb 1284 haclust+ 20 0 126036 38356 9828 R 0.0 0.5 81:53.15 /usr/libexec/pacemaker/cib 1030 root 20 0 196948 47584 34176 S 6.5 0.6 6:42.59 corosync [root@west-16 ~]# top -b -n1 -p $(pidof ruby),$(pidof cib),$(pidof corosync) -c | grep "^ " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 760 root 20 0 1166508 346200 5016 S 97.4 4.3 238:05.64 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb 1279 haclust+ 20 0 128092 40412 9828 S 0.0 0.5 82:00.64 /usr/libexec/pacemaker/cib 1018 root 20 0 195920 46160 34176 S 0.0 0.6 5:24.32 corosync Cluster II (virtual machines): [root@virt-021 ~]# top -b -n1 -p $(pidof ruby),$(pidof cib),$(pidof corosync) -c | grep "^ " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 582 root 20 0 990584 269752 5132 S 0.0 26.5 198:05.99 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb 901 haclust+ 20 0 128536 41240 9832 S 0.0 4.1 53:19.67 /usr/libexec/pacemaker/cib 715 root 20 0 198960 49708 34184 S 0.0 4.9 5:24.46 corosync [root@virt-022 ~]# top -b -n1 -p $(pidof ruby),$(pidof cib),$(pidof corosync) -c | grep "^ " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 585 root 20 0 831596 291148 4996 S 0.0 28.6 191:33.56 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb 898 haclust+ 20 0 126028 38732 9832 S 0.0 3.8 53:01.23 /usr/libexec/pacemaker/cib 717 root 20 0 197972 49560 34176 S 0.0 4.9 4:12.36 corosync [root@virt-023 ~]# top -b -n1 -p $(pidof ruby),$(pidof cib),$(pidof corosync) -c | grep "^ " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 577 root 20 0 762968 291416 4996 S 0.0 28.6 192:00.85 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb 896 haclust+ 20 0 126032 38392 9864 S 0.0 3.8 51:43.90 /usr/libexec/pacemaker/cib 712 root 20 0 197948 47676 34176 S 0.0 4.7 4:04.95 corosync [root@virt-024 ~]# top -b -n1 -p $(pidof ruby),$(pidof cib),$(pidof corosync) -c | grep "^ " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 575 root 20 0 700516 197612 4996 S 0.0 19.4 198:12.50 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb 900 haclust+ 20 0 128432 40760 9832 S 0.0 4.0 53:27.15 /usr/libexec/pacemaker/cib 714 root 20 0 195900 47148 34176 S 0.0 4.6 4:14.26 corosync In all cases above the memory usage seem to have stopped growing at some point - pcsd was using ~400M after 30 minutes on west-13, while after a long period of time the usage did go just beyond 700M, as displayed above. When I first encountered this issue it was much more significant - 5-7G memory used on 3 of 4 nodes, but since then I performed a system upgrade to a more recent RHEL7 snapshot, including upgrading pacemaker from pacemaker-1.1.10-27.el7 to pacemaker-1.1.10-29.el7 (no change in pcs version though). Although the memory usage seems to be not as harsh, pcsd is still consuming much more than any other process of the cluster stack. I'm closing this based on comment 14 from QA: There were no reports regarding this problem recently nor were we able to reproduce it. If there are concerns about pcsd memory usage, we can reopen the bug. |
Description of problem: Under certain circumstances using the GUI may cause pcsd to start using a extensive amount of memory. This can be relatively easily triggered by just opening details of a cluster with a huge CIB (a 4-node cluster with 1000 resources seems to be safely enough). If the browser is left open, memory usage of the ruby process will slowly go up on all nodes, ruby/cib processes might also show higher CPU usage. Most nodes will be likely shown as offline in the GUI. Version-Release number of selected component (if applicable): pcs-0.9.115-32.el7.x86_64 How reproducible: Easily Steps to Reproduce: 1. Set up a 4-node cluster, including pcs authentication 2. Create 1000 dummy resource (they don't have to be running): for i in `seq 1 1000`; do pcs resource create adummyresourcewithanunusuallylongname${i} Dummy meta target-role=Stopped ; done 3. Log into the GUI, add the cluster and open it, leave the browser running Actual results: After an hour the ruby process looks like this: # top -b -n1 -p $(pidof ruby) -c | grep "pcsd\|RES" PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9853 root 20 0 1333328 630592 1516 S 99.9 61.9 22:26.97 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb Expected results: No extensive memory/cpu usage.