Bug 1559415

Summary: Provisioner node re-election happens almost continuously
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rohan Kanade <rkanade>
Component: web-admin-tendrl-commonsAssignee: Nishanth Thomas <nthomas>
Status: CLOSED ERRATA QA Contact: Daniel Horák <dahorak>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: dahorak, mbukatov, nthomas, rhs-bugs
Target Milestone: ---   
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tendrl-commons-1.6.1-3.el7rhgs Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 07:02:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1503137    

Description Rohan Kanade 2018-03-22 13:52:28 UTC 
Description of problem:

Provisioner node re-election happens almost continuously


Which triggers the 'configure Monitoring' jobs for new Provisioner node and floods the job Queue.

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Start tendrl-node-agent service
2. Watch tendrl central store key "/indexes/tags/provisioner/:integration_id"

Actual results:
1. The value of above key changes every few minutes due to re-election


Expected results:
1 The value of above key should nit change until tendrl-node-agent service restart

Additional info:
Comment 3 Rohan Kanade 2018-04-02 12:09:02 UTC 
Fixed in version:
tendrl-commons-1.6.1-3.el7rhgs
Comment 6 Daniel Horák 2018-06-26 11:32:29 UTC 
Reproduced on:
  Red Hat Enterprise Linux Server release 7.5 (Maipo)
  tendrl-ansible-1.6.1-2.el7rhgs.noarch
  tendrl-api-1.6.1-1.el7rhgs.noarch
  tendrl-api-httpd-1.6.1-1.el7rhgs.noarch
  tendrl-commons-1.6.1-1.el7rhgs.noarch
  tendrl-grafana-plugins-1.6.1-1.el7rhgs.noarch
  tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-monitoring-integration-1.6.1-1.el7rhgs.noarch
  tendrl-node-agent-1.6.1-1.el7rhgs.noarch
  tendrl-notifier-1.6.0-1.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-ui-1.6.1-1.el7rhgs.noarch

Using following command:
  # while true; do 
    date; etcdctl \
      ${TLS_PARAMETERS} \
      --endpoints https://${HOSTNAME}:2379 ls /indexes/tags/provisioner;
    sleep 10;
  done

In the output, there should be repeated date and particular path from etcd
database, but for some times, there is no path from etcd database printed,
(it is not present).

For example:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Tue Jun 26 13:22:58 CEST 2018
  /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58
  Tue Jun 26 13:23:08 CEST 2018
  /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58
  Tue Jun 26 13:23:18 CEST 2018
  /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58
  Tue Jun 26 13:23:28 CEST 2018
  /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58
  Tue Jun 26 13:23:38 CEST 2018
  /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58
  Tue Jun 26 13:23:48 CEST 2018
  Tue Jun 26 13:23:58 CEST 2018
  Tue Jun 26 13:24:08 CEST 2018
  Tue Jun 26 13:24:18 CEST 2018
  Tue Jun 26 13:24:28 CEST 2018
  Tue Jun 26 13:24:38 CEST 2018
  Tue Jun 26 13:24:48 CEST 2018
  /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58
  Tue Jun 26 13:24:58 CEST 2018
  /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58
  Tue Jun 26 13:25:08 CEST 2018
  /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58
  Tue Jun 26 13:25:19 CEST 2018
  /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Comment 7 Daniel Horák 2018-06-26 11:37:10 UTC 
Tested and Verified using the same approach as described in comment 6 and the path with provisioner
id is always present (watched for more than 2 hours).

Example (truncated) output:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Tue Jun 26 12:58:29 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  Tue Jun 26 12:58:39 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  Tue Jun 26 12:58:49 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  Tue Jun 26 12:58:59 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  Tue Jun 26 12:59:10 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  Tue Jun 26 12:59:20 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  Tue Jun 26 12:59:30 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  Tue Jun 26 12:59:40 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  Tue Jun 26 12:59:50 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  Tue Jun 26 13:00:00 CEST 2018
  /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Tested on:
  Red Hat Enterprise Linux Server release 7.5 (Maipo)
  tendrl-ansible-1.6.3-5.el7rhgs.noarch
  tendrl-api-1.6.3-3.el7rhgs.noarch
  tendrl-api-httpd-1.6.3-3.el7rhgs.noarch
  tendrl-commons-1.6.3-7.el7rhgs.noarch
  tendrl-grafana-plugins-1.6.3-5.el7rhgs.noarch
  tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-monitoring-integration-1.6.3-5.el7rhgs.noarch
  tendrl-node-agent-1.6.3-7.el7rhgs.noarch
  tendrl-notifier-1.6.3-4.el7rhgs.noarch
  tendrl-selinux-1.5.4-2.el7rhgs.noarch
  tendrl-ui-1.6.3-4.el7rhgs.noarch

>> VERIFIED
Comment 9 errata-xmlrpc 2018-09-04 07:02:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2616