Bug 1779272 - Workload throughput rates slow as cluster ages with MS active
Summary: Workload throughput rates slow as cluster ages with MS active
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW-Multisite
Version: 3.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: 5.*
Assignee: Matt Benjamin (redhat)
QA Contact: Tejas
URL:
Whiteboard:
Depends On:
Blocks: 1727980
TreeView+ depends on / blocked
 
Reported: 2019-12-03 16:02 UTC by John Harrigan
Modified: 2020-05-11 18:55 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-11 18:55:59 UTC
Embargoed:

Attachments (Terms of Use)
fill cluster workload (940 bytes, text/plain)
2019-12-03 16:02 UTC, John Harrigan 		no flags Details
mixed operation workload (1.58 KB, text/plain)
2019-12-03 16:03 UTC, John Harrigan 		no flags Details
View All

Description John Harrigan 2019-12-03 16:02:30 UTC 
Created attachment 1641715 [details]
fill cluster workload

Description of problem:
Workload throughput rates (operations per second) degrade as cluster ages, through pool deletion/creation.

Version-Release number of selected component (if applicable):
12.2.12-79 = RHCS 3.3z1

How reproducible:
yes

Steps to Reproduce:
1. Deploy two Ceph clusters: 3x MON nodes, 8x OSD nodes (RGW collocated)
2. Configure and activate site1 and site2 multisite
3. Install COSbench
4. Execute cluster-fill (bb-fillWorkload) and mixed-operation (bb-hybridSS) workloads - attached. Record results.
5. Delete and recreate RGW pools
6. Rerun workloads, record results. Note throughput reduction in COSbench.
7. After five test cycles of pool deletion and creation, perf degrades ~40%

Actual results:
Cluster fill workload: w194(initial)=10659, w201(aged)=6309
Hybrid workload: w195(initial)=2631; w202(aged)=1474

Expected results:
Workload performance is roughly sustained (~10% variance) as cluster ages

Additional info:
Comment 1 RHEL Program Management 2019-12-03 16:02:36 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 2 John Harrigan 2019-12-03 16:03:30 UTC 
Created attachment 1641716 [details]
mixed operation workload
Note You need to log in before you can comment on or make changes to this bug.