Bug 1511597

Summary: [RFE] Gnocchi creates too many directories, which can overflow limitations on certain storage systems
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: gnocchiAssignee: Julien Danjou <jdanjou>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: low Docs Contact:
Priority: low    
Version: 10.0 (Newton)CC: apannu, apevec, dhill, dvd, jdanjou, jjoyce, jschluet, lhh, lnatapov, mabaakou, marjones, pkilambi, slinaber, tvignaud
Target Milestone: Upstream M2Keywords: FutureFeature, Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: gnocchi-4.3.2-0.20180822205817.342a449.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1665165 (view as bug list) Environment:
Last Closed: 2019-01-11 11:48:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1640021    
Bug Blocks: 1665165    

Description David Hill 2017-11-09 16:37:19 UTC
What problem/issue/behavior are you having trouble with?  What do you expect to see?

There is a limitation on the VNX that each NFS share can only contain 65k directories

Since gnocchi creates a directory for each metric, at scale this can easily grow to over 65k directories, overwhelming the storage backend and causing gnocchi (and eventually redis) to lock up

For an example of how little scale it takes to reach this limitation, in this Verizon lab, the archive policy is set to retain 6 ten minute points per hour of data, for a relatively small number of VMs

[stack@hostname01~]$ openstack server list --all | grep ACTIVE | wc -l
175
[root@hostname01 ~]# ls /var/lib/gnocchi/ | wc -l
17781

Comment 1 Julien Danjou 2017-11-09 16:56:44 UTC
There's one directory per metric with the file driver. The archive policies have no influence over that.

Is there any limit on the number of files created on VNX NFS shares?

Comment 2 David Hill 2017-11-09 18:23:05 UTC
It would also appear that there's a limitation in number of files to 500k per directory as per this document [1].

[1] https://vmwiz.files.wordpress.com/2012/12/docu44573_vnx-operating-environment-for-file-release-notes-version-7-1-56-5.pdf

Comment 21 errata-xmlrpc 2019-01-11 11:48:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045