Red Hat Bugzilla – Bug 1571235
Job thread in all tendrl components consumes lot of cpu and memory utilization
Last modified: 2018-09-04 03:05:56 EDT
Description of problem: The number of import and unmanage cluster flows are increasing the number of jobs. So when the number of jobs count is increased then job thread in each component in each module consumes a lot of CPU and memory. Actually, percent of CPU and memory consumption will keep increasing if we keep doing import and unmanage. Version-Release number of selected component (if applicable): How reproducible: Keep doing import and unmanage and check CPU and memory usage by tendrl components in server and storage nodes. Steps to Reproduce: 1. Import cluster 2. unmanage cluster 3. Keep doing the same Actual results: Memory and CPU utilization is kept increasing Expected results: Tendrl component consumes CPU and memory resource properly. Additional info:
Reproduced on: RHGS WA Server: collectd-5.7.2-3.1.el7rhgs.x86_64 collectd-ping-5.7.2-3.1.el7rhgs.x86_64 etcd-3.2.7-1.el7.x86_64 libcollectdclient-5.7.2-3.1.el7rhgs.x86_64 python-etcd-0.4.5-2.el7rhgs.noarch rubygem-etcd-0.3.0-2.el7rhgs.noarch tendrl-ansible-1.6.3-2.el7rhgs.noarch tendrl-api-1.6.3-1.el7rhgs.noarch tendrl-api-httpd-1.6.3-1.el7rhgs.noarch tendrl-commons-1.6.3-2.el7rhgs.noarch tendrl-grafana-plugins-1.6.3-1.el7rhgs.noarch tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch tendrl-monitoring-integration-1.6.3-1.el7rhgs.noarch tendrl-node-agent-1.6.3-2.el7rhgs.noarch tendrl-notifier-1.6.3-2.el7rhgs.noarch tendrl-selinux-1.5.4-2.el7rhgs.noarch tendrl-ui-1.6.3-1.el7rhgs.noarch Gluster Storage Server: collectd-5.7.2-3.1.el7rhgs.x86_64 collectd-ping-5.7.2-3.1.el7rhgs.x86_64 glusterfs-3.12.2-12.el7rhgs.x86_64 glusterfs-api-3.12.2-12.el7rhgs.x86_64 glusterfs-cli-3.12.2-12.el7rhgs.x86_64 glusterfs-client-xlators-3.12.2-12.el7rhgs.x86_64 glusterfs-events-3.12.2-12.el7rhgs.x86_64 glusterfs-fuse-3.12.2-12.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-12.el7rhgs.x86_64 glusterfs-libs-3.12.2-12.el7rhgs.x86_64 glusterfs-rdma-3.12.2-12.el7rhgs.x86_64 glusterfs-server-3.12.2-12.el7rhgs.x86_64 gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch libcollectdclient-5.7.2-3.1.el7rhgs.x86_64 libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.5.x86_64 python2-gluster-3.12.2-12.el7rhgs.x86_64 python-etcd-0.4.5-2.el7rhgs.noarch tendrl-collectd-selinux-1.5.4-2.el7rhgs.noarch tendrl-commons-1.6.3-2.el7rhgs.noarch tendrl-gluster-integration-1.6.3-2.el7rhgs.noarch tendrl-node-agent-1.6.3-2.el7rhgs.noarch tendrl-selinux-1.5.4-2.el7rhgs.noarch vdsm-gluster-4.19.43-2.3.el7rhgs.noarch Tested and Verified on: RHGS WA Server: collectd-5.7.2-3.1.el7rhgs.x86_64 collectd-ping-5.7.2-3.1.el7rhgs.x86_64 etcd-3.2.7-1.el7.x86_64 libcollectdclient-5.7.2-3.1.el7rhgs.x86_64 python-etcd-0.4.5-2.el7rhgs.noarch rubygem-etcd-0.3.0-2.el7rhgs.noarch tendrl-ansible-1.6.3-4.el7rhgs.noarch tendrl-api-1.6.3-3.el7rhgs.noarch tendrl-api-httpd-1.6.3-3.el7rhgs.noarch tendrl-commons-1.6.3-6.el7rhgs.noarch tendrl-grafana-plugins-1.6.3-4.el7rhgs.noarch tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch tendrl-monitoring-integration-1.6.3-4.el7rhgs.noarch tendrl-node-agent-1.6.3-6.el7rhgs.noarch tendrl-notifier-1.6.3-3.el7rhgs.noarch tendrl-selinux-1.5.4-2.el7rhgs.noarch tendrl-ui-1.6.3-3.el7rhgs.noarch Gluster Storage Server: collectd-5.7.2-3.1.el7rhgs.x86_64 collectd-ping-5.7.2-3.1.el7rhgs.x86_64 glusterfs-3.12.2-12.el7rhgs.x86_64 glusterfs-api-3.12.2-12.el7rhgs.x86_64 glusterfs-cli-3.12.2-12.el7rhgs.x86_64 glusterfs-client-xlators-3.12.2-12.el7rhgs.x86_64 glusterfs-events-3.12.2-12.el7rhgs.x86_64 glusterfs-fuse-3.12.2-12.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-12.el7rhgs.x86_64 glusterfs-libs-3.12.2-12.el7rhgs.x86_64 glusterfs-rdma-3.12.2-12.el7rhgs.x86_64 glusterfs-server-3.12.2-12.el7rhgs.x86_64 gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch libcollectdclient-5.7.2-3.1.el7rhgs.x86_64 libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.5.x86_64 python2-gluster-3.12.2-12.el7rhgs.x86_64 python-etcd-0.4.5-2.el7rhgs.noarch tendrl-collectd-selinux-1.5.4-2.el7rhgs.noarch tendrl-commons-1.6.3-6.el7rhgs.noarch tendrl-gluster-integration-1.6.3-4.el7rhgs.noarch tendrl-node-agent-1.6.3-6.el7rhgs.noarch tendrl-selinux-1.5.4-2.el7rhgs.noarch vdsm-gluster-4.19.43-2.3.el7rhgs.noarch With more than 40 jobs (20 cycles of manage and un-manage) the load of Tendrl components is still significantly lower on the new packages, than it was with older packages with 20 jobs (after 10 cycles of manage and un-manage). >> VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2616