Bug 1647386 - AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Traceback
Summary: AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Traceback
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: web-admin-tendrl-monitoring-integration
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: gowtham
QA Contact: sds-qe-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-07 11:39 UTC by Filip Balák
Modified: 2019-05-08 18:06 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-08 16:03:05 UTC
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1622461 0 low CLOSED tendrl-node-agent (and gluster-integration) crash if WA Server (etcd database) not available 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1630344 0 unspecified CLOSED Somtimes node-agent message socket file "message.sock" is missing 2022-07-09 10:16:41 UTC
Red Hat Bugzilla 1647393 0 medium CLOSED AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Traceback 2021-02-22 00:41:40 UTC

Internal Links: 1622461 1630344 1647393

Description Filip Balák 2018-11-07 11:39:08 UTC 
Description of problem:
During testing of BZ 1630344 there appeared errors in journal logs for tendrl-monitoring-integration:
...
Nov 07 10:51:23 fbalak-server[9229]: Error connecting to central store (etcd), trying again...
Nov 07 10:51:25 fbalak-server[9229]: Error connecting to central store (etcd), trying again...
Nov 07 10:51:25 fbalak-server[9229]: Error connecting to central store (etcd), trying again...
Nov 07 10:51:25 fbalak-server[9229]: Exception in thread Thread-18:
Nov 07 10:51:25 fbalak-server[9229]: Traceback (most recent call last):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:51:25 fbalak-server[9229]: self.run()
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:51:25 fbalak-server[9229]: Traceback (most recent call last):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:51:25 fbalak-server[9229]: self.run()
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:51:25 fbalak-server[9229]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py",
Nov 07 10:51:25 fbalak-server[9229]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:51:25 fbalak-server[9229]: local_index = response.modifiedIndex + 1
Nov 07 10:51:25 fbalak-server[9229]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:51:25 fbalak-server[9229]: Exception in thread Thread-16:
Nov 07 10:51:25 fbalak-server[9229]: Traceback (most recent call last):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:51:25 fbalak-server[9229]: self.run()
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:51:25 fbalak-server[9229]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py",
Nov 07 10:51:25 fbalak-server[9229]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:51:25 fbalak-server[9229]: local_index = response.modifiedIndex + 1
Nov 07 10:51:25 fbalak-server[9229]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:51:25 fbalak-server[9229]: Exception in thread Thread-17:
Nov 07 10:51:25 fbalak-server[9229]: Traceback (most recent call last):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:51:25 fbalak-server[9229]: self.run()
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:51:25 fbalak-server[9229]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py",
Nov 07 10:51:25 fbalak-server[9229]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:51:25 fbalak-server[9229]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:51:25 fbalak-server[9229]: local_index = response.modifiedIndex + 1
Nov 07 10:51:25 fbalak-server[9229]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:51:25 fbalak-server[9229]: [07/Nov/2018:10:51:25] ENGINE Bus STOPPING
Nov 07 10:51:25 fbalak-server[9229]: [07/Nov/2018:10:51:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('10.
Nov 07 10:51:25 fbalak-server[9229]: [07/Nov/2018:10:51:25] ENGINE Stopped thread '_TimeoutMonitor'.
...

It seems that this happens when etcd is down and service is unable to connect there.

Version-Release number of selected component (if applicable):
tendrl-monitoring-integration-1.6.3-15.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-api-1.6.3-8.el7rhgs.noarch
tendrl-api-httpd-1.6.3-8.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-15.el7rhgs.noarch
tendrl-ansible-1.6.3-9.el7rhgs.noarch
tendrl-commons-1.6.3-13.el7rhgs.noarch
tendrl-node-agent-1.6.3-11.el7rhgs.noarch
tendrl-ui-1.6.3-12.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch

How reproducible:
100%

Steps to Reproduce:
  1. log into WA server
  2. # systemctl stop etcd
  3. etcd stops and after few minutes node-agent and monitoring-integation will go down too
  4. # systemctl start etcd
  5. # systemctl start tendrl-node-agent
     (don't use restart)
  7. # systemctl start tendrl-monitoring-integration

Actual results:
There is a python traceback.

Expected results:
The error in traceback should be handled and provide clear log.

Additional info:
Similar Traceback happens for tendrl-node-agent service
Comment 3 gowtham 2018-11-14 08:39:55 UTC 
This error will come when storage node services unable to connect with etcd, is etcd is up?
Note You need to log in before you can comment on or make changes to this bug.