DescriptionPradeep Kilambi
2017-08-14 15:30:55 UTC
+++ This bug was initially created as a clone of Bug #1478555 +++
Description of problem:
It is possibly to exceed the capacity of a Tripleo deployed cloud's Telemetry service such that some metrics will never be processed and thus have measures grow unbounded. When this occurs there is several major failure conditions which will occur based on the speed of the unbounded growth. This bug's purpose is to support a change to the Gnocchi v3 scheduling of unprocessed data to a "round-robin" function which would reduce the likely hood of a metric being starved and thus growing unbounded.
Version-Release number of selected component (if applicable):
Gnocchi/3.0 - Newton
Gnocchi/3.1 - Ocata
How reproducible:
With ~100 instances a 3 controller cloud with 24 logical cores/machine (thus 6 workers per controller, 18 total metricd processing workers) will exceed capacity and eventually fail an osd if Ceph. Swift long term failure conditions are not well know but it is suspected that Swift will eventually slow down when a single container is overloaded with many small objects.
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
One potential fix [0] that has been proposed is to sort the Gnocchi backlog from oldest to newest and process the oldest backlog first. This prevents some metrics from being starved and growing their number of measures in the Ceph omap and Ceph metrics pool as 16-byte objects. It may be possible for Ceph to do the sorting beforehand though that route needs to be investigated.
[0] https://github.com/gnocchixyz/gnocchi/pull/266