Bug 1571235 - Job thread in all tendrl components consumes lot of cpu and memory utilization
Summary: Job thread in all tendrl components consumes lot of cpu and memory utilization
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: web-admin-tendrl-commons
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: RHGS 3.4.0
Assignee: Jeff Brown
QA Contact: Daniel Horák
URL:
Whiteboard:
Depends On:
Blocks: 1503137
TreeView+ depends on / blocked
 
Reported: 2018-04-24 11:26 UTC by gowtham
Modified: 2018-09-04 07:05 UTC (History)
5 users (show)

Fixed In Version: tendrl-commons-1.6.3-3.el7rhgs
Doc Type: Bug Fix
Doc Text:
Cause: RHGSWA job processing consumes too much resources Consequence: RHGS-WA jobs consumes more than expected resources. Fix: Check for job lock info before creating job worker thread Result: Less resource consumption of RHGSWA jobs processor thread per service
Clone Of:
Environment:
Last Closed: 2018-09-04 07:04:50 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github Tendrl commons issues 930 0 None None None 2018-04-24 11:27:33 UTC
Red Hat Product Errata RHSA-2018:2616 0 None None None 2018-09-04 07:05:55 UTC

Description gowtham 2018-04-24 11:26:19 UTC
Description of problem:
The number of import and unmanage cluster flows are increasing the number of jobs. So when the number of jobs count is increased then job thread in each component in each module consumes a lot of CPU and memory. Actually, percent of CPU and memory consumption will keep increasing if we keep doing import and unmanage.

Version-Release number of selected component (if applicable):


How reproducible:
Keep doing import and unmanage and check CPU and memory usage by tendrl components in server and storage nodes.

Steps to Reproduce:
1. Import cluster
2. unmanage cluster
3. Keep doing the same 

Actual results:
Memory and CPU utilization is kept increasing 

Expected results:
Tendrl component consumes CPU and memory resource properly.

Additional info:

Comment 4 Daniel Horák 2018-06-06 14:06:04 UTC
Reproduced on:
  RHGS WA Server:
  collectd-5.7.2-3.1.el7rhgs.x86_64
  collectd-ping-5.7.2-3.1.el7rhgs.x86_64
  etcd-3.2.7-1.el7.x86_64
  libcollectdclient-5.7.2-3.1.el7rhgs.x86_64
  python-etcd-0.4.5-2.el7rhgs.noarch
  rubygem-etcd-0.3.0-2.el7rhgs.noarch
  tendrl-ansible-1.6.3-2.el7rhgs.noarch
  tendrl-api-1.6.3-1.el7rhgs.noarch
  tendrl-api-httpd-1.6.3-1.el7rhgs.noarch
  tendrl-commons-1.6.3-2.el7rhgs.noarch
  tendrl-grafana-plugins-1.6.3-1.el7rhgs.noarch
  tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-monitoring-integration-1.6.3-1.el7rhgs.noarch
  tendrl-node-agent-1.6.3-2.el7rhgs.noarch
  tendrl-notifier-1.6.3-2.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-ui-1.6.3-1.el7rhgs.noarch

  Gluster Storage Server:
  collectd-5.7.2-3.1.el7rhgs.x86_64
  collectd-ping-5.7.2-3.1.el7rhgs.x86_64
  glusterfs-3.12.2-12.el7rhgs.x86_64
  glusterfs-api-3.12.2-12.el7rhgs.x86_64
  glusterfs-cli-3.12.2-12.el7rhgs.x86_64
  glusterfs-client-xlators-3.12.2-12.el7rhgs.x86_64
  glusterfs-events-3.12.2-12.el7rhgs.x86_64
  glusterfs-fuse-3.12.2-12.el7rhgs.x86_64
  glusterfs-geo-replication-3.12.2-12.el7rhgs.x86_64
  glusterfs-libs-3.12.2-12.el7rhgs.x86_64
  glusterfs-rdma-3.12.2-12.el7rhgs.x86_64
  glusterfs-server-3.12.2-12.el7rhgs.x86_64
  gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
  gluster-nagios-common-0.2.4-1.el7rhgs.noarch
  libcollectdclient-5.7.2-3.1.el7rhgs.x86_64
  libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.5.x86_64
  python2-gluster-3.12.2-12.el7rhgs.x86_64
  python-etcd-0.4.5-2.el7rhgs.noarch
  tendrl-collectd-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-commons-1.6.3-2.el7rhgs.noarch
  tendrl-gluster-integration-1.6.3-2.el7rhgs.noarch
  tendrl-node-agent-1.6.3-2.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  vdsm-gluster-4.19.43-2.3.el7rhgs.noarch

Tested and Verified on:
  RHGS WA Server:
  collectd-5.7.2-3.1.el7rhgs.x86_64
  collectd-ping-5.7.2-3.1.el7rhgs.x86_64
  etcd-3.2.7-1.el7.x86_64
  libcollectdclient-5.7.2-3.1.el7rhgs.x86_64
  python-etcd-0.4.5-2.el7rhgs.noarch
  rubygem-etcd-0.3.0-2.el7rhgs.noarch
  tendrl-ansible-1.6.3-4.el7rhgs.noarch
  tendrl-api-1.6.3-3.el7rhgs.noarch
  tendrl-api-httpd-1.6.3-3.el7rhgs.noarch
  tendrl-commons-1.6.3-6.el7rhgs.noarch
  tendrl-grafana-plugins-1.6.3-4.el7rhgs.noarch
  tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-monitoring-integration-1.6.3-4.el7rhgs.noarch
  tendrl-node-agent-1.6.3-6.el7rhgs.noarch
  tendrl-notifier-1.6.3-3.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-ui-1.6.3-3.el7rhgs.noarch

  Gluster Storage Server:
  collectd-5.7.2-3.1.el7rhgs.x86_64
  collectd-ping-5.7.2-3.1.el7rhgs.x86_64
  glusterfs-3.12.2-12.el7rhgs.x86_64
  glusterfs-api-3.12.2-12.el7rhgs.x86_64
  glusterfs-cli-3.12.2-12.el7rhgs.x86_64
  glusterfs-client-xlators-3.12.2-12.el7rhgs.x86_64
  glusterfs-events-3.12.2-12.el7rhgs.x86_64
  glusterfs-fuse-3.12.2-12.el7rhgs.x86_64
  glusterfs-geo-replication-3.12.2-12.el7rhgs.x86_64
  glusterfs-libs-3.12.2-12.el7rhgs.x86_64
  glusterfs-rdma-3.12.2-12.el7rhgs.x86_64
  glusterfs-server-3.12.2-12.el7rhgs.x86_64
  gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
  gluster-nagios-common-0.2.4-1.el7rhgs.noarch
  libcollectdclient-5.7.2-3.1.el7rhgs.x86_64
  libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.5.x86_64
  python2-gluster-3.12.2-12.el7rhgs.x86_64
  python-etcd-0.4.5-2.el7rhgs.noarch
  tendrl-collectd-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-commons-1.6.3-6.el7rhgs.noarch
  tendrl-gluster-integration-1.6.3-4.el7rhgs.noarch
  tendrl-node-agent-1.6.3-6.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  vdsm-gluster-4.19.43-2.3.el7rhgs.noarch

With more than 40 jobs (20 cycles of manage and un-manage) the load of Tendrl
components is still significantly lower on the new packages, than it was with
older packages with 20 jobs (after 10 cycles of manage and un-manage).

>> VERIFIED

Comment 7 errata-xmlrpc 2018-09-04 07:04:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2616


Note You need to log in before you can comment on or make changes to this bug.