Description of problem: Provisioner node re-election happens almost continuously Which triggers the 'configure Monitoring' jobs for new Provisioner node and floods the job Queue. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Start tendrl-node-agent service 2. Watch tendrl central store key "/indexes/tags/provisioner/:integration_id" Actual results: 1. The value of above key changes every few minutes due to re-election Expected results: 1 The value of above key should nit change until tendrl-node-agent service restart Additional info:
Fixed in version: tendrl-commons-1.6.1-3.el7rhgs
Reproduced on: Red Hat Enterprise Linux Server release 7.5 (Maipo) tendrl-ansible-1.6.1-2.el7rhgs.noarch tendrl-api-1.6.1-1.el7rhgs.noarch tendrl-api-httpd-1.6.1-1.el7rhgs.noarch tendrl-commons-1.6.1-1.el7rhgs.noarch tendrl-grafana-plugins-1.6.1-1.el7rhgs.noarch tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch tendrl-monitoring-integration-1.6.1-1.el7rhgs.noarch tendrl-node-agent-1.6.1-1.el7rhgs.noarch tendrl-notifier-1.6.0-1.el7rhgs.noarch tendrl-selinux-1.5.4-2.el7rhgs.noarch tendrl-ui-1.6.1-1.el7rhgs.noarch Using following command: # while true; do date; etcdctl \ ${TLS_PARAMETERS} \ --endpoints https://${HOSTNAME}:2379 ls /indexes/tags/provisioner; sleep 10; done In the output, there should be repeated date and particular path from etcd database, but for some times, there is no path from etcd database printed, (it is not present). For example: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tue Jun 26 13:22:58 CEST 2018 /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58 Tue Jun 26 13:23:08 CEST 2018 /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58 Tue Jun 26 13:23:18 CEST 2018 /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58 Tue Jun 26 13:23:28 CEST 2018 /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58 Tue Jun 26 13:23:38 CEST 2018 /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58 Tue Jun 26 13:23:48 CEST 2018 Tue Jun 26 13:23:58 CEST 2018 Tue Jun 26 13:24:08 CEST 2018 Tue Jun 26 13:24:18 CEST 2018 Tue Jun 26 13:24:28 CEST 2018 Tue Jun 26 13:24:38 CEST 2018 Tue Jun 26 13:24:48 CEST 2018 /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58 Tue Jun 26 13:24:58 CEST 2018 /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58 Tue Jun 26 13:25:08 CEST 2018 /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58 Tue Jun 26 13:25:19 CEST 2018 /indexes/tags/provisioner/aac5e400-3c11-4b1b-b6d6-6f88b118ac58 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tested and Verified using the same approach as described in comment 6 and the path with provisioner id is always present (watched for more than 2 hours). Example (truncated) output: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tue Jun 26 12:58:29 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 Tue Jun 26 12:58:39 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 Tue Jun 26 12:58:49 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 Tue Jun 26 12:58:59 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 Tue Jun 26 12:59:10 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 Tue Jun 26 12:59:20 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 Tue Jun 26 12:59:30 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 Tue Jun 26 12:59:40 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 Tue Jun 26 12:59:50 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 Tue Jun 26 13:00:00 CEST 2018 /indexes/tags/provisioner/119c8298-8009-45df-82fc-af8acd64ec61 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tested on: Red Hat Enterprise Linux Server release 7.5 (Maipo) tendrl-ansible-1.6.3-5.el7rhgs.noarch tendrl-api-1.6.3-3.el7rhgs.noarch tendrl-api-httpd-1.6.3-3.el7rhgs.noarch tendrl-commons-1.6.3-7.el7rhgs.noarch tendrl-grafana-plugins-1.6.3-5.el7rhgs.noarch tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch tendrl-monitoring-integration-1.6.3-5.el7rhgs.noarch tendrl-node-agent-1.6.3-7.el7rhgs.noarch tendrl-notifier-1.6.3-4.el7rhgs.noarch tendrl-selinux-1.5.4-2.el7rhgs.noarch tendrl-ui-1.6.3-4.el7rhgs.noarch >> VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2616