Description of problem: During testing of BZ 1630344 there appeared errors in journal logs for tendrl-node-agent: ... Nov 07 10:20:04 fbalak-server tendrl-node-agent[3228]: Starting new HTTPS connection (4): fbalak-server Nov 07 10:20:04 fbalak-server tendrl-node-age[3228]: Starting new HTTPS connection (4): fbalak-server Nov 07 10:20:04 fbalak-server tendrl-node-age[3228]: Request to server https://fbalak-server:2379 failed: MaxRetryError("HTTPSConnectionPool(host='fbalak-server', port=2379): Max retries exceeded with url: /v2/keys/queue?quorum=true (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))",) Nov 07 10:20:04 fbalak-server tendrl-node-agent[3228]: Request to server https://fbalak-server:2379 failed: MaxRetryError("HTTPSConnectionPool(host='fbalak-server', port=2379): Max retries exceeded with url: /v2/keys/queue?quorum=true (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))",) Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Error connecting to central store (etcd), trying again... Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Exception in thread Thread-2: Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Traceback (most recent call last): Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.run() Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 765, in run Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.__target(*self.__args, **self.__kwargs) Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py", line 95, in watch Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: for change in NS._int.client.eternal_watch(key): Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: local_index = response.modifiedIndex + 1 Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Exception in thread Thread-13: Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Traceback (most recent call last): Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.run() Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 765, in run Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.__target(*self.__args, **self.__kwargs) Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py", line 95, in watch Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: for change in NS._int.client.eternal_watch(key): Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: local_index = response.modifiedIndex + 1 Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Exception in thread Thread-10: Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Traceback (most recent call last): Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.run() Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 765, in run Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.__target(*self.__args, **self.__kwargs) Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py", line 95, in watch Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: for change in NS._int.client.eternal_watch(key): Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: local_index = response.modifiedIndex + 1 Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Nov 07 10:31:04 fbalak-server systemd[1]: Started A python agent local to every managed storage node in the sds cluster. Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: Creating namespace.node_agent from source tendrl.node_agent Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: namespace.node_agent created! Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: Finding objects in namespace.node_agent.objects Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: Registering object namespace.node_agent.objects.ClusterMessage ... It seems that this happens when etcd is down and service is unable to connect there. Version-Release number of selected component (if applicable): tendrl-monitoring-integration-1.6.3-15.el7rhgs.noarch tendrl-selinux-1.5.4-2.el7rhgs.noarch tendrl-api-1.6.3-8.el7rhgs.noarch tendrl-api-httpd-1.6.3-8.el7rhgs.noarch tendrl-grafana-plugins-1.6.3-15.el7rhgs.noarch tendrl-ansible-1.6.3-9.el7rhgs.noarch tendrl-commons-1.6.3-13.el7rhgs.noarch tendrl-node-agent-1.6.3-11.el7rhgs.noarch tendrl-ui-1.6.3-12.el7rhgs.noarch tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch tendrl-notifier-1.6.3-4.el7rhgs.noarch How reproducible: 100% Steps to Reproduce: 1. log into WA server 2. # systemctl stop etcd 3. etcd stops and after few minutes node-agent and monitoring-integation will go down too 4. # systemctl start etcd 5. # systemctl start tendrl-node-agent (don't use restart) 7. # systemctl start tendrl-monitoring-integration 8. # journalctl -u tendrl-node-agent Actual results: There is a python traceback. Expected results: The error in traceback should be handled and provide clear log. Additional info: Similar Traceback happens for tendrl-node-agent service (BZ 1647386)
Is this something same as reported at BZ#1647386?