Description of problem: During workload, I am seeing the following message on the controllers: haproxy[99218]: proxy gnocchi has no server available! This workload consists of just launching nova guests (16 at a time). Version-Release number of selected component (if applicable): 4ca485c8ffa8 192.168.0.1:8787/rhosp14/openstack-gnocchi-api:2018-11-06.1 "kolla_start" 41 hours ago Up 41 hours (healthy) gnocchi_api 139739313b80 192.168.0.1:8787/rhosp14/openstack-gnocchi-metricd:2018-11-06.1 "kolla_start" 41 hours ago Up 41 hours (healthy) gnocchi_metricd a9a9e5ddb9d4 192.168.0.1:8787/rhosp14/openstack-gnocchi-statsd:2018-11-06.1 "kolla_start" 41 hours ago Up 41 hours (healthy) gnocchi_statsd How reproducible: 100% Steps to Reproduce: 1. Launch 16 guests concurrently -- until you reach 150 guests.
Can you describe your setup ? How many controllers do you have ? What is your Gnocchi backend ? If Swift or Ceph, how many dedicated Swift storage nodes do you have ? If file how do you share Gnocchi data between controllers ?
(In reply to Mehdi ABAAKOUK from comment #2) > Can you describe your setup ? How many controllers do you have ? What is > your Gnocchi backend ? If Swift or Ceph, how many dedicated Swift storage > nodes do you have ? If file how do you share Gnocchi data between > controllers ? 3 Controllers, Swift as the backend, collocated on the controllers. These guests are very short lived. If you think it is due to the back end, i can deploy with a very simple Ceph backend.
Swift collocated on controller can't handle any load. Ceph is always preferred is available.
(In reply to Mehdi ABAAKOUK from comment #4) > Swift collocated on controller can't handle any load. Ceph is always > preferred is available. Mehdi - Hm... I don't think we can blanket statement that. Yes, collocated isn't preferred, however putting Swift on a separate disk should alleviate some of the load. Also, you mention Swift cannot handle the load, Gnocchi is the service "going down". Also, if WE know Collocation isn't preferred, do we deploy different retention/collection policies based on collocation? If not, why not?
The current retention/collection policies is already very low because of this, because gather less data.