Bug 1511597 - [RFE] Gnocchi creates too many directories, which can overflow limitations on certain storage systems
Summary: [RFE] Gnocchi creates too many directories, which can overflow limitations on...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: gnocchi
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: Upstream M2
: 14.0 (Rocky)
Assignee: Julien Danjou
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On: 1640021
Blocks: 1665165
TreeView+ depends on / blocked
 
Reported: 2017-11-09 16:37 UTC by David Hill
Modified: 2019-01-11 11:49 UTC (History)
14 users (show)

Fixed In Version: gnocchi-4.3.2-0.20180822205817.342a449.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1665165 (view as bug list)
Environment:
Last Closed: 2019-01-11 11:48:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github gnocchixyz gnocchi pull 900 0 None closed file: allow to create a subdir to create less root directory 2020-04-06 14:45:01 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:49:03 UTC

Description David Hill 2017-11-09 16:37:19 UTC
What problem/issue/behavior are you having trouble with?  What do you expect to see?

There is a limitation on the VNX that each NFS share can only contain 65k directories

Since gnocchi creates a directory for each metric, at scale this can easily grow to over 65k directories, overwhelming the storage backend and causing gnocchi (and eventually redis) to lock up

For an example of how little scale it takes to reach this limitation, in this Verizon lab, the archive policy is set to retain 6 ten minute points per hour of data, for a relatively small number of VMs

[stack@hostname01~]$ openstack server list --all | grep ACTIVE | wc -l
175
[root@hostname01 ~]# ls /var/lib/gnocchi/ | wc -l
17781

Comment 1 Julien Danjou 2017-11-09 16:56:44 UTC
There's one directory per metric with the file driver. The archive policies have no influence over that.

Is there any limit on the number of files created on VNX NFS shares?

Comment 2 David Hill 2017-11-09 18:23:05 UTC
It would also appear that there's a limitation in number of files to 500k per directory as per this document [1].

[1] https://vmwiz.files.wordpress.com/2012/12/docu44573_vnx-operating-environment-for-file-release-notes-version-7-1-56-5.pdf

Comment 21 errata-xmlrpc 2019-01-11 11:48:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.