Bug 1478555 - Prevent starved metrics from growing measures unbounded (Ceph and Swift)
Prevent starved metrics from growing measures unbounded (Ceph and Swift)
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-gnocchi (Show other bugs)
10.0 (Newton)
Unspecified Unspecified
high Severity urgent
: z5
: 10.0 (Newton)
Assigned To: Julien Danjou
Sasha Smolyak
: Triaged, ZStream
: 1426552 1481327 (view as bug list)
Depends On:
Blocks: 1389374 1481327 1481328
  Show dependency treegraph
Reported: 2017-08-04 15:18 EDT by Alex Krzos
Modified: 2018-02-27 07:03 EST (History)
9 users (show)

See Also:
Fixed In Version: openstack-gnocchi-3.0.14-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1481327 1481328 (view as bug list)
Last Closed: 2017-09-28 12:37:35 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2824 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 Bug Fix and Enhancement Advisory 2017-09-28 16:34:18 EDT

  None (edit)
Description Alex Krzos 2017-08-04 15:18:40 EDT
Description of problem:
It is possibly to exceed the capacity of a Tripleo deployed cloud's Telemetry service such that some metrics will never be processed and thus have measures grow unbounded.  When this occurs there is several major failure conditions which will occur based on the speed of the unbounded growth.  This bug's purpose is to support a change to the Gnocchi v3 scheduling of unprocessed data to a "round-robin" function which would reduce the likely hood of a metric being starved and thus growing unbounded.

Version-Release number of selected component (if applicable):
Gnocchi/3.0 - Newton 
Gnocchi/3.1 - Ocata

How reproducible:
With ~100 instances a 3 controller cloud with 24 logical cores/machine (thus 6 workers per controller, 18 total metricd processing workers) will exceed capacity and eventually fail an osd if Ceph.  Swift long term failure conditions are not well know but it is suspected that Swift will eventually slow down when a single container is overloaded with many small objects.

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

One potential fix [0] that has been proposed is to sort the Gnocchi backlog from oldest to newest and process the oldest backlog first.  This prevents some metrics from being starved and growing their number of measures in the Ceph omap and Ceph metrics pool as 16-byte objects.  It may be possible for Ceph to do the sorting beforehand though that route needs to be investigated.

[0] https://github.com/gnocchixyz/gnocchi/pull/266
Comment 4 Pradeep Kilambi 2017-08-29 10:27:02 EDT
*** Bug 1481327 has been marked as a duplicate of this bug. ***
Comment 5 Pradeep Kilambi 2017-08-30 09:02:18 EDT
rebased rdo to 3.0.14:

Comment 6 Julien Danjou 2017-09-04 03:30:13 EDT
*** Bug 1426552 has been marked as a duplicate of this bug. ***
Comment 13 errata-xmlrpc 2017-09-28 12:37:35 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.