Bug 1893740

Summary: Improve scaling of snapshot mirror scheduler
Product: Red Hat Ceph Storage Reporter: Jason Dillaman <jdillama>
Component: RBD-MirrorAssignee: Jason Dillaman <jdillama>
Status: CLOSED ERRATA QA Contact: Harish Munjulur <hmunjulu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.2CC: asakthiv, ceph-eng-bugs, ceph-qe-bugs, hmunjulu, racpatel, tserlin, vereddy
Target Milestone: ---Keywords: Improvement, Performance
Target Release: 4.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-14.2.11-75.el8cp, ceph-14.2.11-75.el7cp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-12 14:58:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jason Dillaman 2020-11-02 13:45:31 UTC
Description of problem:
Right now the rbd mirror snapshot scheduler (rbd_support mgr module) calls `mirror_image_create_snapshot` synchronously in one thread loop for all mirroring images that require a snapshot. Creating a snapshot is rather long operation (i.e. sometimes >1 second) and therefore means that the scheduler has rather low limit of snapshot creation rate.

E.g. if one creates a mirroring pool with 100 images in snapshot mirror mode, and configure global 1 minute interval snapshot schedule, it is most likely the snapshot schedule will be lagging.

It can be improved, e.g. by implementing async version of `mirror_image_create_snapshot` and modifying the scheduler to create snapshots for many images in parallel.

Version-Release number of selected component (if applicable):
RHCS 4.2

How reproducible:
100%

Steps to Reproduce:
1. Create hundred+ snapshot-based mirrored images
2. Schedule mirror snapshots to be created every minute

Actual results:
The "rbd_support" MGR module cannot keep up.

Expected results:
The "rbd_support" module can better scale. The goal is to improve the scaling by nearly an order of magnitude.

Additional info:

Comment 9 errata-xmlrpc 2021-01-12 14:58:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0081