Bug 1410923 - rbd export/export-diff commands not honoring '--rbd-concurrent-management-ops' option, option appears to have no effect on performance
Summary: rbd export/export-diff commands not honoring '--rbd-concurrent-management-ops...
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RBD
Version: 1.3.3
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: 3.0
Assignee: Jason Dillaman
QA Contact: Tejas
Bara Ancincova
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: 1494421
TreeView+ depends on / blocked
 
Reported: 2017-01-06 20:57 UTC by Kyle Squizzato
Modified: 2017-12-05 23:32 UTC (History)
8 users (show)

(edit)
.The `--rbd-concurrent-management-ops` option works with the `rbd export` command

The `--rbd-concurrent-management-ops` option ensures that image export or import work in parallel. Previously, when `--rbd-concurrent-management-ops` was used with the `rbd export` command, it had no effect on the command performance. The underlying source code has been modified, and `--rbd-concurrent-management-ops` works as expected when exporting images by using `rbd export`.
Clone Of:
(edit)
Last Closed: 2017-12-05 23:32:37 UTC


Attachments (Terms of Use)
brief sequential rbd write & read fio job file (764 bytes, text/plain)
2017-01-06 20:58 UTC, Kyle Squizzato
no flags Details
fio job output against rbds in the pool we are exporting from (12.37 KB, text/plain)
2017-01-06 20:59 UTC, Kyle Squizzato
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3387 normal SHIPPED_LIVE Red Hat Ceph Storage 3.0 bug fix and enhancement update 2017-12-06 03:03:45 UTC
Ceph Project Bug Tracker 19034 None None None 2017-03-01 13:55 UTC

Description Kyle Squizzato 2017-01-06 20:57:43 UTC
Description of problem:
When using either the 'rbd export' or 'rbd export-diff' commands with the '--rbd-concurrent-management-ops' flag no impact on performance is observed.

Sample output from current attempt:

# snapshot created for initial export
$ rbd snap ls rbd/nas7-rds-stg1
SNAPID NAME             SIZE 
  1106 161224-0935 102400 GB

$ rbd info rbd/foobar@161224-0935
rbd image 'foobar':
        size 102400 GB in 26214400 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.5f059d238e1f29
        format: 2
        features: layering
        flags: 
        protected: False

$ rbd export-diff --rbd-concurrent-management-ops 50 rbd/foobar@161224-0935 - | pv | rbd import-diff --rbd-concurrent-management-ops 50 - expandtest/foobar
Importing image diff: 1% complete...    1TiB 7:39:35 [37.2MiB/s]
Importing image diff: 2% complete...   2TiB 19:53:12 [36.7MiB/s]
Importing image diff: 3% complete...   3TiB 27:30:54 [43.2MiB/s]
Importing image diff: 4% complete...   4TiB 34:33:08 [7.97MiB/s]
Importing image diff: 5% complete...   5TiB 41:44:14 [42.4MiB/s]

Version-Release number of selected component (if applicable):
0.94.x (will request specific ceph -v) 

How reproducible:
Always 

Steps to Reproduce:
1. Create snapshot from an rbd 
2. Export/import and specify the '--rbd-concurrent-management-ops' flag, for example:

rbd export-diff --rbd-concurrent-management-ops 50 rbd/foobar@161224-0935 - | pv | rbd import-diff --rbd-concurrent-management-ops 50 - expandtest/foobar

Actual results:
Slow performance when performing an rbd export/export-diff

Expected results:
Performance should be higher when '--rbd-concurrent-management-ops' is set to a high value.  A ceph cluster should have a ton of parallelism available to it.

Additional info:
* Providing fio profile and job output which exhibits the concurrency expected in the cluster vs the image diff import above.
* Debug data is en route
* It's also worth noting that if 'export-diff' truly does not support 'rbd-concurrent-management-ops' even though it appears to be present in the code: 

ceph/src/tools/rbd/action/ExportDiff.cc:
168   ExportDiffContext edc(&image, fd, info.size,
169                         g_conf->rbd_concurrent_management_ops, no_progress);
170   r = image.diff_iterate2(fromsnapname, 0, info.size, true, whole_object,
171                           &C_ExportDiff::export_diff_cb, (void *)&edc);

The customer has also seen the same behavior on just 'export'.

Comment 2 Kyle Squizzato 2017-01-06 20:58 UTC
Created attachment 1238092 [details]
brief sequential rbd write & read fio job file

Comment 3 Kyle Squizzato 2017-01-06 20:59 UTC
Created attachment 1238093 [details]
fio job output against rbds in the pool we are exporting from

Comment 11 Jason Dillaman 2017-09-24 01:54:21 UTC
I would suggest adding "--debug-rbd 20" to the rbd CLI and utilize the "librbd::io::AioCompletion" logs to see if more than one instance of an AIO message is in-flight concurrently.

Comment 12 Tejas 2017-09-25 04:33:42 UTC
Thank you Jason!
Verified in ceph version 12.2.0-1.el7cp

Comment 18 errata-xmlrpc 2017-12-05 23:32:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3387


Note You need to log in before you can comment on or make changes to this bug.