Bug 1647393

Summary:	AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Traceback
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Filip Balák <fbalak>
Component:	web-admin-tendrl-node-agent	Assignee:	gowtham <gshanmug>
Status:	CLOSED WONTFIX	QA Contact:	sds-qe-bugs
Severity:	medium	Docs Contact:
Priority:	medium
Version:	rhgs-3.4	CC:	gshanmug, rhs-bugs, sankarshan, shtripat
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-05-08 19:02:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Filip Balák 2018-11-07 11:49:26 UTC

Description of problem:
During testing of BZ 1630344 there appeared errors in journal logs for tendrl-node-agent:
...
Nov 07 10:20:04 fbalak-server tendrl-node-agent[3228]: Starting new HTTPS connection (4): fbalak-server
Nov 07 10:20:04 fbalak-server tendrl-node-age[3228]: Starting new HTTPS connection (4): fbalak-server
Nov 07 10:20:04 fbalak-server tendrl-node-age[3228]: Request to server https://fbalak-server:2379 failed: MaxRetryError("HTTPSConnectionPool(host='fbalak-server', port=2379): Max retries exceeded with url: /v2/keys/queue?quorum=true (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))",)
Nov 07 10:20:04 fbalak-server tendrl-node-agent[3228]: Request to server https://fbalak-server:2379 failed: MaxRetryError("HTTPSConnectionPool(host='fbalak-server', port=2379): Max retries exceeded with url: /v2/keys/queue?quorum=true (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))",)
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Error connecting to central store (etcd), trying again...
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Exception in thread Thread-2:
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Traceback (most recent call last):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.run()
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py", line 95, in watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: local_index = response.modifiedIndex + 1
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Exception in thread Thread-13:
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Traceback (most recent call last):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.run()
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py", line 95, in watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: local_index = response.modifiedIndex + 1
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Exception in thread Thread-10:
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Traceback (most recent call last):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.run()
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py", line 95, in watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: local_index = response.modifiedIndex + 1
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:31:04 fbalak-server systemd[1]: Started A python agent local to every managed storage node in the sds cluster.
Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: Creating namespace.node_agent from source tendrl.node_agent
Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: namespace.node_agent created!
Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: Finding objects in namespace.node_agent.objects
Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: Registering object namespace.node_agent.objects.ClusterMessage
...

It seems that this happens when etcd is down and service is unable to connect there.

Version-Release number of selected component (if applicable):
tendrl-monitoring-integration-1.6.3-15.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-api-1.6.3-8.el7rhgs.noarch
tendrl-api-httpd-1.6.3-8.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-15.el7rhgs.noarch
tendrl-ansible-1.6.3-9.el7rhgs.noarch
tendrl-commons-1.6.3-13.el7rhgs.noarch
tendrl-node-agent-1.6.3-11.el7rhgs.noarch
tendrl-ui-1.6.3-12.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch

How reproducible:
100%

Steps to Reproduce:
  1. log into WA server
  2. # systemctl stop etcd
  3. etcd stops and after few minutes node-agent and monitoring-integation will go down too
  4. # systemctl start etcd
  5. # systemctl start tendrl-node-agent
     (don't use restart)
  7. # systemctl start tendrl-monitoring-integration
  8. # journalctl -u tendrl-node-agent

Actual results:
There is a python traceback.

Expected results:
The error in traceback should be handled and provide clear log.

Additional info:
Similar Traceback happens for tendrl-node-agent service (BZ 1647386)

Comment 3 Shubhendu Tripathi 2018-11-19 13:16:28 UTC

Is this something same as reported at BZ#1647386?