Red Hat Bugzilla – Bug 1478555
Prevent starved metrics from growing measures unbounded (Ceph and Swift)
Last modified: 2017-09-28 12:37:35 EDT
Description of problem:
It is possibly to exceed the capacity of a Tripleo deployed cloud's Telemetry service such that some metrics will never be processed and thus have measures grow unbounded. When this occurs there is several major failure conditions which will occur based on the speed of the unbounded growth. This bug's purpose is to support a change to the Gnocchi v3 scheduling of unprocessed data to a "round-robin" function which would reduce the likely hood of a metric being starved and thus growing unbounded.
Version-Release number of selected component (if applicable):
Gnocchi/3.0 - Newton
Gnocchi/3.1 - Ocata
With ~100 instances a 3 controller cloud with 24 logical cores/machine (thus 6 workers per controller, 18 total metricd processing workers) will exceed capacity and eventually fail an osd if Ceph. Swift long term failure conditions are not well know but it is suspected that Swift will eventually slow down when a single container is overloaded with many small objects.
Steps to Reproduce:
One potential fix  that has been proposed is to sort the Gnocchi backlog from oldest to newest and process the oldest backlog first. This prevents some metrics from being starved and growing their number of measures in the Ceph omap and Ceph metrics pool as 16-byte objects. It may be possible for Ceph to do the sorting beforehand though that route needs to be investigated.
*** Bug 1481327 has been marked as a duplicate of this bug. ***
rebased rdo to 3.0.14:
*** Bug 1426552 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.