Bug 1416546

Summary: mongodb fills with event type of "objectstore.http.request" with gnocchi driver=swift
Product: Red Hat OpenStack Reporter: Dave Wilson <dwilson>
Component: python-ceilometermiddlewareAssignee: Pradeep Kilambi <pkilambi>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: apevec, augol, david.costakos, djuran, dwilson, jschluet, lhh, mburns, nlevinki, pkilambi, rhel-osp-director-maint, tvignaud, vaggarwa
Target Milestone: Upstream M2Keywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-ceilometermiddleware-1.0.1-0.20170228134010.6ba005f.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1468223 1468225 (view as bug list) Environment:
Last Closed: 2017-12-13 21:04:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1468223, 1468225    

Description Dave Wilson 2017-01-25 19:10:44 UTC
Description of problem: 
mongodb fills disk with event data of "objectstore.http.request" with gnocchi driver=swift


Version-Release number of selected component (if applicable):


How reproducible: 100%


Steps to Reproduce:
1. Create overcloud (non ceph) 
2. Deploy a few vms

Actual results: 
mongodb fills with many "objectstore.http.request" filling up system disk


Expected results:


Additional info: 
Below is a breakdown of document count by event type in collection after a few days of running in overcloud with 4 compute nodes and 10 VMs

DB size : ceilometer event colletion size  12GB

Event Document Count
----------------------------------
"objectstore.http.request", 3505417
"identity.authenticate", 34043
"compute.instance.exists", 446
"compute.instance.update", 70
"identity.endpoint.created", 39
"identity.role_assignment.created", 15
"identity.service.created", 13
"identity.user.created", 12
"scheduler.select_destinations.start", 10
"scheduler.select_destinations.end", 10
"port.update.start", 10
"port.update.end", 10
"port.create.start", 10
"port.create.end", 10
"compute.instance.create.start", 10
"compute.instance.create.end", 10
"identity.role.created", 5
"image.send", 4
"identity.user.updated", 3
"image.update", 2
"identity.project.created", 2
"identity.domain.created", 2
"subnet.create.start", 1
"subnet.create.end", 1
"network.create.start", 1
"network.create.end", 1
"image.upload", 1
"image.prepare", 1
"image.create", 1
"image.activate", 1
"identity.region.created", 1
"identity.project.updated", 1

Comment 1 Julien Danjou 2017-03-13 15:37:23 UTC
Hi Dave,


Those objects are meant to be filtered by Ceilometer if they are generated by Gnocchi itself. Is it the case? You should have filter_service_activity=True in ceilometer.conf under the gnocchi dispatcher section.

How and why did you deploy both Gnocchi and MongoDB?

Comment 3 Julien Danjou 2017-03-14 09:39:36 UTC
Hi David,

Ok, I did not get it was event related. So the filtering only happens in the metric part anyway.

In this case, what should be checked if whetever "ignore_projects" is set in /etc/swift/proxy-server.conf. That should be set to the project id that gnocchi is using. Its default value is "gnocchi", so if the project gnocchi is using to access Swift is "gnocchi" that would also work.

Comment 6 Dave Wilson 2017-04-06 03:20:26 UTC
It's not set

[root@overcloud-controller-0 ~]# grep ignore /etc/swift/proxy-server.conf
[root@overcloud-controller-0 ~]#

Comment 10 Thierry Vignaud 2017-06-27 13:03:04 UTC
The relevant changes are in python-ceilometermiddleware-1.0.1-0.20170228134010.6ba005f.el7ost, openstack-tripleo-heat-templates-7.0.0-0.20170624014919.el7ost & puppet-swift-11.2.0-0.20170623014910.e8f765d.el7ost

Comment 13 Julien Danjou 2017-08-11 13:25:44 UTC
*** Bug 1449986 has been marked as a duplicate of this bug. ***

Comment 17 errata-xmlrpc 2017-12-13 21:04:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462