Bug 1571235

Summary: Job thread in all tendrl components consumes lot of cpu and memory utilization
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: gowtham <gshanmug>
Component: web-admin-tendrl-commonsAssignee: Jeff Brown <jefbrown>
Status: CLOSED ERRATA QA Contact: Daniel Horák <dahorak>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: dahorak, mbukatov, nthomas, rhs-bugs, sankarshan
Target Milestone: ---   
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tendrl-commons-1.6.3-3.el7rhgs Doc Type: Bug Fix
Doc Text:
Cause: RHGSWA job processing consumes too much resources Consequence: RHGS-WA jobs consumes more than expected resources. Fix: Check for job lock info before creating job worker thread Result: Less resource consumption of RHGSWA jobs processor thread per service
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 07:04:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1503137    

Description gowtham 2018-04-24 11:26:19 UTC 
Description of problem:
The number of import and unmanage cluster flows are increasing the number of jobs. So when the number of jobs count is increased then job thread in each component in each module consumes a lot of CPU and memory. Actually, percent of CPU and memory consumption will keep increasing if we keep doing import and unmanage.

Version-Release number of selected component (if applicable):


How reproducible:
Keep doing import and unmanage and check CPU and memory usage by tendrl components in server and storage nodes.

Steps to Reproduce:
1. Import cluster
2. unmanage cluster
3. Keep doing the same 

Actual results:
Memory and CPU utilization is kept increasing 

Expected results:
Tendrl component consumes CPU and memory resource properly.

Additional info:
Comment 4 Daniel Horák 2018-06-06 14:06:04 UTC 
Reproduced on:
  RHGS WA Server:
  collectd-5.7.2-3.1.el7rhgs.x86_64
  collectd-ping-5.7.2-3.1.el7rhgs.x86_64
  etcd-3.2.7-1.el7.x86_64
  libcollectdclient-5.7.2-3.1.el7rhgs.x86_64
  python-etcd-0.4.5-2.el7rhgs.noarch
  rubygem-etcd-0.3.0-2.el7rhgs.noarch
  tendrl-ansible-1.6.3-2.el7rhgs.noarch
  tendrl-api-1.6.3-1.el7rhgs.noarch
  tendrl-api-httpd-1.6.3-1.el7rhgs.noarch
  tendrl-commons-1.6.3-2.el7rhgs.noarch
  tendrl-grafana-plugins-1.6.3-1.el7rhgs.noarch
  tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-monitoring-integration-1.6.3-1.el7rhgs.noarch
  tendrl-node-agent-1.6.3-2.el7rhgs.noarch
  tendrl-notifier-1.6.3-2.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-ui-1.6.3-1.el7rhgs.noarch

  Gluster Storage Server:
  collectd-5.7.2-3.1.el7rhgs.x86_64
  collectd-ping-5.7.2-3.1.el7rhgs.x86_64
  glusterfs-3.12.2-12.el7rhgs.x86_64
  glusterfs-api-3.12.2-12.el7rhgs.x86_64
  glusterfs-cli-3.12.2-12.el7rhgs.x86_64
  glusterfs-client-xlators-3.12.2-12.el7rhgs.x86_64
  glusterfs-events-3.12.2-12.el7rhgs.x86_64
  glusterfs-fuse-3.12.2-12.el7rhgs.x86_64
  glusterfs-geo-replication-3.12.2-12.el7rhgs.x86_64
  glusterfs-libs-3.12.2-12.el7rhgs.x86_64
  glusterfs-rdma-3.12.2-12.el7rhgs.x86_64
  glusterfs-server-3.12.2-12.el7rhgs.x86_64
  gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
  gluster-nagios-common-0.2.4-1.el7rhgs.noarch
  libcollectdclient-5.7.2-3.1.el7rhgs.x86_64
  libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.5.x86_64
  python2-gluster-3.12.2-12.el7rhgs.x86_64
  python-etcd-0.4.5-2.el7rhgs.noarch
  tendrl-collectd-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-commons-1.6.3-2.el7rhgs.noarch
  tendrl-gluster-integration-1.6.3-2.el7rhgs.noarch
  tendrl-node-agent-1.6.3-2.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  vdsm-gluster-4.19.43-2.3.el7rhgs.noarch

Tested and Verified on:
  RHGS WA Server:
  collectd-5.7.2-3.1.el7rhgs.x86_64
  collectd-ping-5.7.2-3.1.el7rhgs.x86_64
  etcd-3.2.7-1.el7.x86_64
  libcollectdclient-5.7.2-3.1.el7rhgs.x86_64
  python-etcd-0.4.5-2.el7rhgs.noarch
  rubygem-etcd-0.3.0-2.el7rhgs.noarch
  tendrl-ansible-1.6.3-4.el7rhgs.noarch
  tendrl-api-1.6.3-3.el7rhgs.noarch
  tendrl-api-httpd-1.6.3-3.el7rhgs.noarch
  tendrl-commons-1.6.3-6.el7rhgs.noarch
  tendrl-grafana-plugins-1.6.3-4.el7rhgs.noarch
  tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-monitoring-integration-1.6.3-4.el7rhgs.noarch
  tendrl-node-agent-1.6.3-6.el7rhgs.noarch
  tendrl-notifier-1.6.3-3.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-ui-1.6.3-3.el7rhgs.noarch

  Gluster Storage Server:
  collectd-5.7.2-3.1.el7rhgs.x86_64
  collectd-ping-5.7.2-3.1.el7rhgs.x86_64
  glusterfs-3.12.2-12.el7rhgs.x86_64
  glusterfs-api-3.12.2-12.el7rhgs.x86_64
  glusterfs-cli-3.12.2-12.el7rhgs.x86_64
  glusterfs-client-xlators-3.12.2-12.el7rhgs.x86_64
  glusterfs-events-3.12.2-12.el7rhgs.x86_64
  glusterfs-fuse-3.12.2-12.el7rhgs.x86_64
  glusterfs-geo-replication-3.12.2-12.el7rhgs.x86_64
  glusterfs-libs-3.12.2-12.el7rhgs.x86_64
  glusterfs-rdma-3.12.2-12.el7rhgs.x86_64
  glusterfs-server-3.12.2-12.el7rhgs.x86_64
  gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
  gluster-nagios-common-0.2.4-1.el7rhgs.noarch
  libcollectdclient-5.7.2-3.1.el7rhgs.x86_64
  libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.5.x86_64
  python2-gluster-3.12.2-12.el7rhgs.x86_64
  python-etcd-0.4.5-2.el7rhgs.noarch
  tendrl-collectd-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-commons-1.6.3-6.el7rhgs.noarch
  tendrl-gluster-integration-1.6.3-4.el7rhgs.noarch
  tendrl-node-agent-1.6.3-6.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  vdsm-gluster-4.19.43-2.3.el7rhgs.noarch

With more than 40 jobs (20 cycles of manage and un-manage) the load of Tendrl
components is still significantly lower on the new packages, than it was with
older packages with 20 jobs (after 10 cycles of manage and un-manage).

>> VERIFIED
Comment 7 errata-xmlrpc 2018-09-04 07:04:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2616