Bug 1848503 - cephfs: Provide alternatives to increase the total cephfs subvolume snapshot counts to greater than the current 400 across a Cephfs volume
Summary: cephfs: Provide alternatives to increase the total cephfs subvolume snapshot ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: CephFS
Version: 4.1
Hardware: All
OS: All
high
urgent
Target Milestone: z2
: 4.1
Assignee: Yan, Zheng
QA Contact: Hemanth Kumar
Aron Gunn
URL:
Whiteboard:
Depends On:
Blocks: 1816167 1854503
TreeView+ depends on / blocked
 
Reported: 2020-06-18 13:17 UTC by Shyamsundar
Modified: 2020-10-29 09:52 UTC (History)
15 users (show)

Fixed In Version: ceph-14.2.8-104.el8cp, ceph-14.2.8-104.el7cp
Doc Type: Bug Fix
Doc Text:
.Improved Ceph File System performance as the number of subvolume snapshots increases Previously, creating more than 400 subvolume snapshots was degrading the Ceph File System performance by slowing down file system operations. With this release, you can configure subvolumes to only support subvolume snapshots at the subvolume root directory, and you can prevent cross-subvolume links and renames. Doing this allows for the creation of higher numbers of subvolume snapshots, and does not degrade the Ceph File System performance.
Clone Of:
: 1854503 (view as bug list)
Environment:
Last Closed: 2020-09-30 17:25:42 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 47096 0 None None None 2020-08-27 21:23:03 UTC
Ceph Project Bug Tracker 47157 0 None None None 2020-08-27 03:14:03 UTC
Red Hat Product Errata RHBA-2020:4144 0 None None None 2020-09-30 17:26:08 UTC

Description Shyamsundar 2020-06-18 13:17:17 UTC
Issue is originally discussed here: https://github.com/ceph/ceph-csi/issues/1133

Ceph upstream tracker for the same is here: https://tracker.ceph.com/issues/46074

Snip from github discussion:
"For cephfs volume, we only create snapshots at volume root. we can disable the special handling for inodes with multiple links. If the special handling is disabled, that can help avoiding the 400 snapshot per-file-system limit"

For subvolumes, the intention is that these are used in isolation with each other, and hence hard links across subvolumes should not be a concern. Given this, the handling as discussed above can be relaxed for subvolumes.

This bug is filed to track the bug/feature is backported to required Ceph versions that would ship with OCS 4.6.

OCS 4.6 has a snapshot limit requirement of 512 as per: https://issues.redhat.com/browse/KNIP-661

Comment 3 Yaniv Kaul 2020-07-15 17:27:12 UTC
Can we get QA_ACK?

Comment 8 Hemanth Kumar 2020-09-02 08:33:47 UTC
Hi Shyam,

As part of the fix, I was able to reach 1k snapshots of a subvol on regular intervals with IO's on. I dont see any issues on the filesystem so far and the IO's were on for atleast 2 days now.
So, Do we still have any limit on the snapshot ? so that I can reach the max limit and see if we hit any issues.

Comment 19 Hemanth Kumar 2020-09-09 08:44:47 UTC
Hi Shyam, 

I have tested creating snapshots on subvolume path instead of a directory and it all works fine. No issues seen.

Moving to verified.

Comment 26 errata-xmlrpc 2020-09-30 17:25:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 4.1 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4144

Comment 27 Yaniv Kaul 2020-10-26 14:28:15 UTC
(In reply to Hemanth Kumar from comment #19)
> Hi Shyam, 
> 
> I have tested creating snapshots on subvolume path instead of a directory
> and it all works fine. No issues seen.
> 
> Moving to verified.

Was it tested at scale? How many snaps?


Note You need to log in before you can comment on or make changes to this bug.