Bug 1654707 - haproxy reports Gnocchi is no longer available
Summary: haproxy reports Gnocchi is no longer available
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: gnocchi
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Mehdi ABAAKOUK
QA Contact: Sasha Smolyak
URL:
Whiteboard: scale_lab
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-29 13:22 UTC by Joe Talerico
Modified: 2018-12-12 07:23 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-12 07:23:13 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Joe Talerico 2018-11-29 13:22:36 UTC
Description of problem:
During workload, I am seeing the following message on the controllers:

haproxy[99218]: proxy gnocchi has no server available!    

This workload consists of just launching nova guests (16 at a time). 

Version-Release number of selected component (if applicable):
4ca485c8ffa8        192.168.0.1:8787/rhosp14/openstack-gnocchi-api:2018-11-06.1                 "kolla_start"            41 hours ago        Up 41 hours (healthy)                         gnocchi_api
139739313b80        192.168.0.1:8787/rhosp14/openstack-gnocchi-metricd:2018-11-06.1             "kolla_start"            41 hours ago        Up 41 hours (healthy)                         gnocchi_metricd
a9a9e5ddb9d4        192.168.0.1:8787/rhosp14/openstack-gnocchi-statsd:2018-11-06.1              "kolla_start"            41 hours ago        Up 41 hours (healthy)                         gnocchi_statsd


How reproducible:
100%

Steps to Reproduce:
1. Launch 16 guests concurrently -- until you reach 150 guests.

Comment 2 Mehdi ABAAKOUK 2018-11-29 13:53:34 UTC
Can you describe your setup ? How many controllers do you have ? What is your Gnocchi backend ? If Swift or Ceph, how many dedicated Swift storage nodes do you have ? If file how do you share Gnocchi data between controllers ?

Comment 3 Joe Talerico 2018-11-29 19:10:30 UTC
(In reply to Mehdi ABAAKOUK from comment #2)
> Can you describe your setup ? How many controllers do you have ? What is
> your Gnocchi backend ? If Swift or Ceph, how many dedicated Swift storage
> nodes do you have ? If file how do you share Gnocchi data between
> controllers ?

3 Controllers, Swift as the backend, collocated on the controllers. These guests are very short lived. If you think it is due to the back end, i can deploy with a very simple Ceph backend.

Comment 4 Mehdi ABAAKOUK 2018-11-30 08:17:50 UTC
Swift collocated on controller can't handle any load. Ceph is always preferred is available.

Comment 5 Joe Talerico 2018-12-03 11:55:15 UTC
(In reply to Mehdi ABAAKOUK from comment #4)
> Swift collocated on controller can't handle any load. Ceph is always
> preferred is available.

Mehdi - Hm... I don't think we can blanket statement that. 

Yes, collocated isn't preferred, however putting Swift on a separate disk should alleviate some of the load.  Also, you mention Swift cannot handle the load, Gnocchi is the service "going down". 

Also, if WE know Collocation isn't preferred, do we deploy different retention/collection policies based on collocation? If not, why not?

Comment 6 Mehdi ABAAKOUK 2018-12-10 15:54:50 UTC
The current retention/collection policies is already very low because of this, because gather less data.


Note You need to log in before you can comment on or make changes to this bug.