Bug 1647386

Summary:	AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Traceback
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Filip Balák <fbalak>
Component:	web-admin-tendrl-monitoring-integration	Assignee:	gowtham <gshanmug>
Status:	CLOSED WONTFIX	QA Contact:	sds-qe-bugs
Severity:	medium	Docs Contact:
Priority:	medium
Version:	rhgs-3.4	CC:	rhs-bugs, sankarshan
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-05-08 16:03:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Filip Balák 2018-11-07 11:39:08 UTC

Description of problem:
During testing of BZ 1630344 there appeared errors in journal logs for tendrl-monitoring-integration:
...
Nov 07 10:51:23 fbalak-server[9229]: Error connecting to central store (etcd), trying again...
Nov 07 10:51:25 fbalak-server[9229]: Error connecting to central store (etcd), trying again...
Nov 07 10:51:25 fbalak-server[9229]: Error connecting to central store (etcd), trying again...
Nov 07 10:51:25 fbalak-server[9229]: Exception in thread Thread-18:
Nov 07 10:51:25 fbalak-server[9229]: Traceback (most recent call last):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:51:25 fbalak-server[9229]: self.run()
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:51:25 fbalak-server[9229]: Traceback (most recent call last):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:51:25 fbalak-server[9229]: self.run()
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:51:25 fbalak-server[9229]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py",
Nov 07 10:51:25 fbalak-server[9229]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:51:25 fbalak-server[9229]: local_index = response.modifiedIndex + 1
Nov 07 10:51:25 fbalak-server[9229]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:51:25 fbalak-server[9229]: Exception in thread Thread-16:
Nov 07 10:51:25 fbalak-server[9229]: Traceback (most recent call last):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:51:25 fbalak-server[9229]: self.run()
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:51:25 fbalak-server[9229]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py",
Nov 07 10:51:25 fbalak-server[9229]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:51:25 fbalak-server[9229]: local_index = response.modifiedIndex + 1
Nov 07 10:51:25 fbalak-server[9229]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:51:25 fbalak-server[9229]: Exception in thread Thread-17:
Nov 07 10:51:25 fbalak-server[9229]: Traceback (most recent call last):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:51:25 fbalak-server[9229]: self.run()
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:51:25 fbalak-server[9229]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py",
Nov 07 10:51:25 fbalak-server[9229]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:51:25 fbalak-server[9229]: local_index = response.modifiedIndex + 1
Nov 07 10:51:25 fbalak-server[9229]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:51:25 fbalak-server[9229]: [07/Nov/2018:10:51:25] ENGINE Bus STOPPING
Nov 07 10:51:25 fbalak-server[9229]: [07/Nov/2018:10:51:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('10.
Nov 07 10:51:25 fbalak-server[9229]: [07/Nov/2018:10:51:25] ENGINE Stopped thread '_TimeoutMonitor'.
...

It seems that this happens when etcd is down and service is unable to connect there.

Version-Release number of selected component (if applicable):
tendrl-monitoring-integration-1.6.3-15.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-api-1.6.3-8.el7rhgs.noarch
tendrl-api-httpd-1.6.3-8.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-15.el7rhgs.noarch
tendrl-ansible-1.6.3-9.el7rhgs.noarch
tendrl-commons-1.6.3-13.el7rhgs.noarch
tendrl-node-agent-1.6.3-11.el7rhgs.noarch
tendrl-ui-1.6.3-12.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch

How reproducible:
100%

Steps to Reproduce:
  1. log into WA server
  2. # systemctl stop etcd
  3. etcd stops and after few minutes node-agent and monitoring-integation will go down too
  4. # systemctl start etcd
  5. # systemctl start tendrl-node-agent
     (don't use restart)
  7. # systemctl start tendrl-monitoring-integration

Actual results:
There is a python traceback.

Expected results:
The error in traceback should be handled and provide clear log.

Additional info:
Similar Traceback happens for tendrl-node-agent service

Comment 3 gowtham 2018-11-14 08:39:55 UTC

This error will come when storage node services unable to connect with etcd, is etcd is up?