Bug 1647393 - AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Traceback
Summary: AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Traceback
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: web-admin-tendrl-node-agent
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: gowtham
QA Contact: sds-qe-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-07 11:49 UTC by Filip Balák
Modified: 2019-05-08 19:02 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-08 19:02:30 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1622461 0 low CLOSED tendrl-node-agent (and gluster-integration) crash if WA Server (etcd database) not available 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1630344 0 unspecified CLOSED Somtimes node-agent message socket file "message.sock" is missing 2022-07-09 10:16:41 UTC
Red Hat Bugzilla 1647386 0 medium CLOSED AttributeError: 'NoneType' object has no attribute 'modifiedIndex' Traceback 2021-02-22 00:41:40 UTC

Internal Links: 1622461 1630344 1647386

Description Filip Balák 2018-11-07 11:49:26 UTC
Description of problem:
During testing of BZ 1630344 there appeared errors in journal logs for tendrl-node-agent:
...
Nov 07 10:20:04 fbalak-server tendrl-node-agent[3228]: Starting new HTTPS connection (4): fbalak-server
Nov 07 10:20:04 fbalak-server tendrl-node-age[3228]: Starting new HTTPS connection (4): fbalak-server
Nov 07 10:20:04 fbalak-server tendrl-node-age[3228]: Request to server https://fbalak-server:2379 failed: MaxRetryError("HTTPSConnectionPool(host='fbalak-server', port=2379): Max retries exceeded with url: /v2/keys/queue?quorum=true (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))",)
Nov 07 10:20:04 fbalak-server tendrl-node-agent[3228]: Request to server https://fbalak-server:2379 failed: MaxRetryError("HTTPSConnectionPool(host='fbalak-server', port=2379): Max retries exceeded with url: /v2/keys/queue?quorum=true (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))",)
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Error connecting to central store (etcd), trying again...
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Exception in thread Thread-2:
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Traceback (most recent call last):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.run()
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py", line 95, in watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: local_index = response.modifiedIndex + 1
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Exception in thread Thread-13:
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Traceback (most recent call last):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.run()
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py", line 95, in watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: local_index = response.modifiedIndex + 1
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Exception in thread Thread-10:
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: Traceback (most recent call last):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.run()
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib64/python2.7/threading.py", line 765, in run
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: self.__target(*self.__args, **self.__kwargs)
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/tendrl/commons/utils/central_store/utils.py", line 95, in watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: for change in NS._int.client.eternal_watch(key):
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: File "/usr/lib/python2.7/site-packages/etcd/client.py", line 795, in eternal_watch
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: local_index = response.modifiedIndex + 1
Nov 07 10:20:05 fbalak-server tendrl-node-agent[3228]: AttributeError: 'NoneType' object has no attribute 'modifiedIndex'
Nov 07 10:31:04 fbalak-server systemd[1]: Started A python agent local to every managed storage node in the sds cluster.
Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: Creating namespace.node_agent from source tendrl.node_agent
Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: namespace.node_agent created!
Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: Finding objects in namespace.node_agent.objects
Nov 07 10:31:05 fbalak-server tendrl-node-agent[6811]: Registering object namespace.node_agent.objects.ClusterMessage
...

It seems that this happens when etcd is down and service is unable to connect there.

Version-Release number of selected component (if applicable):
tendrl-monitoring-integration-1.6.3-15.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-api-1.6.3-8.el7rhgs.noarch
tendrl-api-httpd-1.6.3-8.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-15.el7rhgs.noarch
tendrl-ansible-1.6.3-9.el7rhgs.noarch
tendrl-commons-1.6.3-13.el7rhgs.noarch
tendrl-node-agent-1.6.3-11.el7rhgs.noarch
tendrl-ui-1.6.3-12.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch

How reproducible:
100%

Steps to Reproduce:
  1. log into WA server
  2. # systemctl stop etcd
  3. etcd stops and after few minutes node-agent and monitoring-integation will go down too
  4. # systemctl start etcd
  5. # systemctl start tendrl-node-agent
     (don't use restart)
  7. # systemctl start tendrl-monitoring-integration
  8. # journalctl -u tendrl-node-agent

Actual results:
There is a python traceback.

Expected results:
The error in traceback should be handled and provide clear log.

Additional info:
Similar Traceback happens for tendrl-node-agent service (BZ 1647386)

Comment 3 Shubhendu Tripathi 2018-11-19 13:16:28 UTC
Is this something same as reported at BZ#1647386?


Note You need to log in before you can comment on or make changes to this bug.