Bug 1893740 - Improve scaling of snapshot mirror scheduler
Summary: Improve scaling of snapshot mirror scheduler
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RBD-Mirror
Version: 4.2
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.2
Assignee: Jason Dillaman
QA Contact: Harish Munjulur
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-02 13:45 UTC by Jason Dillaman
Modified: 2021-01-12 14:58 UTC (History)
7 users (show)

Fixed In Version: ceph-14.2.11-75.el8cp, ceph-14.2.11-75.el7cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-12 14:58:09 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 47827 0 None None None 2020-11-02 13:45:30 UTC
Red Hat Product Errata RHSA-2021:0081 0 None None None 2021-01-12 14:58:33 UTC

Description Jason Dillaman 2020-11-02 13:45:31 UTC
Description of problem:
Right now the rbd mirror snapshot scheduler (rbd_support mgr module) calls `mirror_image_create_snapshot` synchronously in one thread loop for all mirroring images that require a snapshot. Creating a snapshot is rather long operation (i.e. sometimes >1 second) and therefore means that the scheduler has rather low limit of snapshot creation rate.

E.g. if one creates a mirroring pool with 100 images in snapshot mirror mode, and configure global 1 minute interval snapshot schedule, it is most likely the snapshot schedule will be lagging.

It can be improved, e.g. by implementing async version of `mirror_image_create_snapshot` and modifying the scheduler to create snapshots for many images in parallel.

Version-Release number of selected component (if applicable):
RHCS 4.2

How reproducible:
100%

Steps to Reproduce:
1. Create hundred+ snapshot-based mirrored images
2. Schedule mirror snapshots to be created every minute

Actual results:
The "rbd_support" MGR module cannot keep up.

Expected results:
The "rbd_support" module can better scale. The goal is to improve the scaling by nearly an order of magnitude.

Additional info:

Comment 9 errata-xmlrpc 2021-01-12 14:58:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0081


Note You need to log in before you can comment on or make changes to this bug.