1398324 – metricd exceptions: You can't write against a read only slave

Bug 1398324 - metricd exceptions: You can't write against a read only slave

Summary: metricd exceptions: You can't write against a read only slave

Keywords:
Status:	CLOSED DUPLICATE of bug 1305549
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-gnocchi
Sub Component:
Version:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	10.0 (Newton)
Assignee:	Pradeep Kilambi
QA Contact:	Yurii Prokulevych
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-11-24 12:53 UTC by Yurii Prokulevych
Modified:	2016-11-24 13:18 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-11-24 13:18:42 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Yurii Prokulevych 2016-11-24 12:53:13 UTC

Description of problem:
-----------------------
Scaling out a compute node after upgrade from RHOS-9 to RHOS-10.
Traceback in metricd.log:
    2016-11-24 12:22:32.594 66009 ERROR futurist.periodics ToozError: Command # 1 (HDEL _tooz_group:alarm_evaluator b7230d32-11d5-41ae-8dfa-8ec777988ba1 b4f2e43e-e5a1-47a9-bb49-49d5c78dcc1d e834547b-b164-4667-9bbc-c
    d30330b4e1c) of pipeline caused error: Command # 1 (HDEL _tooz_group:alarm_evaluator b7230d32-11d5-41ae-8dfa-8ec777988ba1 b4f2e43e-e5a1-47a9-bb49-49d5c78dcc1d e834547b-b164-4667-9bbc-cd30330b4e1c) of pipeline ca
    used error: You can't write against a read only slave.
    2016-11-24 12:22:32.594 66009 ERROR futurist.periodics
    2016-11-24 12:23:02.585 66009 DEBUG futurist.periodics [-] Submitting periodic function 'gnocchi.cli.run_watchers' _process_scheduled /usr/lib/python2.7/site-packages/futurist/periodics.py:614
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics [-] Failed to call periodic 'gnocchi.cli.run_watchers' (it runs every 30.00 seconds)
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics Traceback (most recent call last):
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 267, in run
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     cb(*args, **kwargs)
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 155, in decorator
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     return f(*args, **kwargs)
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 189, in run_watchers
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     self._coord.run_watchers()
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 776, in run_watchers
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     result = super(RedisDriver, self).run_watchers(timeout=timeout)
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 580, in run_watchers
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     timeout=w.leftover(return_none=True))
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 794, in get
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     return self._fut.result(timeout=timeout)
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     self.gen.throw(type, value, traceback)
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 54, in _translate_failures
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     cause=e)
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in raise_with_cause
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics   File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics     raise value
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics ToozError: Command # 1 (HDEL _tooz_group:alarm_evaluator b7230d32-11d5-41ae-8dfa-8ec777988ba1 b4f2e43e-e5a1-47a9-bb49-49d5c78dcc1d e834547b-b164-4667-9bbc-c
    d30330b4e1c) of pipeline caused error: Command # 1 (HDEL _tooz_group:alarm_evaluator b7230d32-11d5-41ae-8dfa-8ec777988ba1 b4f2e43e-e5a1-47a9-bb49-49d5c78dcc1d e834547b-b164-4667-9bbc-cd30330b4e1c) of pipeline ca
    used error: You can't write against a read only slave.
    2016-11-24 12:23:02.612 66009 ERROR futurist.periodics
    2016-11-24 12:23:14.109 66015 ERROR root [-] Unexpected exception occurred 60 time(s)... retrying.
    2016-11-24 12:23:14.109 66015 ERROR root Traceback (most recent call last):
    2016-11-24 12:23:14.109 66015 ERROR root   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 250, in wrapper
    2016-11-24 12:23:14.109 66015 ERROR root     return infunc(*args, **kwargs)
    2016-11-24 12:23:14.109 66015 ERROR root   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 176, in _beat_forever_until_stopped
    2016-11-24 12:23:14.109 66015 ERROR root     wait_until_next_beat = self._driver.heartbeat()
    2016-11-24 12:23:14.109 66015 ERROR root   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 503, in heartbeat
    2016-11-24 12:23:14.109 66015 ERROR root     value=self.STILL_ALIVE)
    2016-11-24 12:23:14.109 66015 ERROR root   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
    2016-11-24 12:23:14.109 66015 ERROR root     self.gen.throw(type, value, traceback)
    2016-11-24 12:23:14.109 66015 ERROR root   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 54, in _translate_failures
    2016-11-24 12:23:14.109 66015 ERROR root     cause=e)
    2016-11-24 12:23:14.109 66015 ERROR root   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in raise_with_cause
    2016-11-24 12:23:14.109 66015 ERROR root     excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
    2016-11-24 12:23:14.109 66015 ERROR root   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
    2016-11-24 12:23:14.109 66015 ERROR root     six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
    2016-11-24 12:23:14.109 66015 ERROR root   File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
    2016-11-24 12:23:14.109 66015 ERROR root     raise value
    2016-11-24 12:23:14.109 66015 ERROR root ToozError: You can't write against a read only slave.

Redis is accessible:
--------------------
redis-cli -h fd00:fd00:fd00:2000::11
[fd00:fd00:fd00:2000::11]:6379> AUTH 
(error) ERR wrong number of arguments for 'auth' command
[fd00:fd00:fd00:2000::11]:6379> AUTH g3Ywkm6mM4BK2gnuuvjhJP2qt
OK
[fd00:fd00:fd00:2000::11]:6379> ping
PONG
[fd00:fd00:fd00:2000::11]:6379> QUIT

Odd thing is that gnocchi-metricd is working:
---------------------------------------------
2016-11-24 12:46:17.697 57260 ERROR root ToozError: You can't write against a read only slave.
2016-11-24 12:46:17.697 57260 ERROR root 
2016-11-24 12:46:43.742 57265 INFO gnocchi.cli [-] 3 measurements bundles across 3 metrics wait to be processed.
2016-11-24 12:46:44.152 57260 DEBUG gnocchi.cli [-] 3 metrics scheduled for processing. _run_job /usr/lib/python2.7/site-packages/gnocchi/cli.py:231
2016-11-24 12:46:44.195 57261 DEBUG gnocchi.storage [-] Processing new measures process_background_tasks /usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py:185
2016-11-24 12:46:44.196 57260 DEBUG futurist.periodics [-] Submitting periodic function 'gnocchi.cli.run_watchers' _process_scheduled /usr/lib/python2.7/site-packages/futurist/periodics.py:614
2016-11-24 12:46:44.206 57261 DEBUG gnocchi.storage._carbonara [-] Processing measures for f92cff0e-569e-406e-9250-edff35987ff3 process_new_measures /usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py
:490
2016-11-24 12:46:44.211 57261 DEBUG gnocchi.storage._carbonara [-] Retrieve unaggregated measures for f92cff0e-569e-406e-9250-edff35987ff3 in 0.00s _get_unaggregated_timeserie_and_unserialize /usr/lib/python2.7/
site-packages/gnocchi/storage/_carbonara.py:110
2016-11-24 12:46:44.400 57261 DEBUG gnocchi.storage._carbonara [-] Computed new metric f92cff0e-569e-406e-9250-edff35987ff3 with 1 new measures in 0.19 seconds (2422 points/s, 127 measures/s) process_new_measure
s /usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:576
2016-11-24 12:46:44.405 57261 DEBUG gnocchi.storage._carbonara [-] Metric f92cff0e-569e-406e-9250-edff35987ff3 locked during 0.20 seconds process_new_measures /usr/lib/python2.7/site-packages/gnocchi/storage/_ca
rbonara.py:582
2016-11-24 12:46:44.407 57261 DEBUG gnocchi.storage._carbonara [-] Processing measures for f96511cc-ceb3-453e-bfd9-233d8bcb0e09 process_new_measures /usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py
:490
2016-11-24 12:46:44.411 57261 DEBUG gnocchi.storage._carbonara [-] Retrieve unaggregated measures for f96511cc-ceb3-453e-bfd9-233d8bcb0e09 in 0.00s _get_unaggregated_timeserie_and_unserialize /usr/lib/python2.7/
site-packages/gnocchi/storage/_carbonara.py:110
2016-11-24 12:46:44.592 57261 DEBUG gnocchi.storage._carbonara [-] Computed new metric f96511cc-ceb3-453e-bfd9-233d8bcb0e09 with 1 new measures in 0.18 seconds (2541 points/s, 133 measures/s) process_new_measure
s /usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:576
2016-11-24 12:46:44.596 57261 DEBUG gnocchi.storage._carbonara [-] Metric f96511cc-ceb3-453e-bfd9-233d8bcb0e09 locked during 0.19 seconds process_new_measures /usr/lib/python2.7/site-packages/gnocchi/storage/_ca
rbonara.py:582
2016-11-24 12:46:44.598 57261 DEBUG gnocchi.storage._carbonara [-] Processing measures for fc4eaa2f-ef96-452b-9b17-c63792f7a4cf process_new_measures /usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py
:490
2016-11-24 12:46:44.603 57261 DEBUG gnocchi.storage._carbonara [-] Retrieve unaggregated measures for fc4eaa2f-ef96-452b-9b17-c63792f7a4cf in 0.00s _get_unaggregated_timeserie_and_unserialize /usr/lib/python2.7/
site-packages/gnocchi/storage/_carbonara.py:110
2016-11-24 12:46:44.805 57261 DEBUG gnocchi.storage._carbonara [-] Computed new metric fc4eaa2f-ef96-452b-9b17-c63792f7a4cf with 1 new measures in 0.20 seconds (2267 points/s, 119 measures/s) process_new_measure
s /usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py:576
2016-11-24 12:46:44.810 57261 DEBUG gnocchi.storage._carbonara [-] Metric fc4eaa2f-ef96-452b-9b17-c63792f7a4cf locked during 0.21 seconds process_new_measures /usr/lib/python2.7/site-packages/gnocchi/storage/_ca
rbonara.py:582
2016-11-24 12:47:14.197 57260 DEBUG futurist.periodics [-] Submitting periodic function 'gnocchi.cli.run_watchers' _process_scheduled /usr/lib/python2.7/site-packages/futurist/periodics.py:614
2016-11-24 12:47:17.794 57267 ERROR root [-] Unexpected exception occurred 60 time(s)... retrying.
2016-11-24 12:47:17.794 57267 ERROR root Traceback (most recent call last):
2016-11-24 12:47:17.794 57267 ERROR root   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 250, in wrapper
2016-11-24 12:47:17.794 57267 ERROR root     return infunc(*args, **kwargs)
2016-11-24 12:47:17.794 57267 ERROR root   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 176, in _beat_forever_until_stopped
2016-11-24 12:47:17.794 57267 ERROR root     wait_until_next_beat = self._driver.heartbeat()
2016-11-24 12:47:17.794 57267 ERROR root   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 503, in heartbeat
2016-11-24 12:47:17.794 57267 ERROR root     value=self.STILL_ALIVE)
2016-11-24 12:47:17.794 57267 ERROR root   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2016-11-24 12:47:17.794 57267 ERROR root     self.gen.throw(type, value, traceback)
2016-11-24 12:47:17.794 57267 ERROR root   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 54, in _translate_failures
2016-11-24 12:47:17.794 57267 ERROR root     cause=e)
2016-11-24 12:47:17.794 57267 ERROR root   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in raise_with_cause
2016-11-24 12:47:17.794 57267 ERROR root     excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2016-11-24 12:47:17.794 57267 ERROR root   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2016-11-24 12:47:17.794 57267 ERROR root     six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2016-11-24 12:47:17.794 57267 ERROR root   File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
2016-11-24 12:47:17.794 57267 ERROR root     raise value
2016-11-24 12:47:17.794 57267 ERROR root ToozError: You can't write against a read only slave.


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
puppet-gnocchi-9.4.1-1.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-3.0.2-1.el7ost.noarch
python-gnocchi-3.0.2-1.el7ost.noarch
openstack-gnocchi-statsd-3.0.2-1.el7ost.noarch
python-gnocchiclient-2.6.0-1.el7ost.noarch
openstack-gnocchi-common-3.0.2-1.el7ost.noarch
openstack-gnocchi-metricd-3.0.2-1.el7ost.noarch
openstack-gnocchi-carbonara-3.0.2-1.el7ost.noarch
openstack-gnocchi-api-3.0.2-1.el7ost.noarch

Steps to Reproduce:
1. Upgrade from 9 -> 10
2. Scale out compute
3. Assert gnocchi/metricd.log

Comment 1 Julien Danjou 2016-11-24 13:12:10 UTC

Redis is in sentinel mode and it seems that metricd has been connected to a redis that became slave?

Can you dig into redis log to see what/how/why this happened?

Comment 2 Mehdi ABAAKOUK 2016-11-24 13:18:42 UTC


*** This bug has been marked as a duplicate of bug 1305549 ***

Note You need to log in before you can comment on or make changes to this bug.