RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1222101 - add IO statistics reporting to dmsetup
Summary: add IO statistics reporting to dmsetup
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.1
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Bryn M. Reeves
QA Contact: Bruno Goncalves
Steven J. Levine
URL:
Whiteboard:
Depends On: 1254801 1267632
Blocks: 1189089 1246193
TreeView+ depends on / blocked
 
Reported: 2015-05-15 18:39 UTC by Bryn M. Reeves
Modified: 2023-03-08 07:27 UTC (History)
13 users (show)

Fixed In Version: lvm2-2.02.130-1.el7
Doc Type: Enhancement
Doc Text:
New dmstats command to display and manage I/O statistics for regions of devices that use the device-mapper driver The `dmstats` command provides userspace support for device-mapper I/O statistics. This allows a user to create, manage and report I/O counters, metrics and latency histogram data for user-defined arbitrary regions of device-mapper devices. Statistics fields are now available in `dmsetup` reports and the `dmstats` command adds new specialized reporting modes designed for use with statistics information. For information on the `dmstats` command, see the dmstats(8) man page.
Clone Of:
: 1254801 1267664 (view as bug list)
Environment:
Last Closed: 2015-11-19 12:46:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2147 0 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2015-11-19 11:11:07 UTC

Description Bryn M. Reeves 2015-05-15 18:39:18 UTC
Description of problem:
The kernel has supported configurable IO statistic for device-mapper devices since 3.11. Enabling and using them currently requires manual use of the device-mapper message facility and returns values in terms of diskstats-style counter sets.

Adding statistics support to libdevmapper and dmsetup would simplify configuration, inspection and management of device-mapper IO statistics and provide common mechanisms for use in other device-mapper applications like lvm2 and multipath-tools.

Version-Release number of selected component (if applicable):
lvm2-2.02.105-14.el7, LVM2-2.02.119

How reproducible:
100%

Steps to Reproduce:
1. dmsetup --help

Actual results:
Np statistics commands.

Additional info:

dm-stats kernel interface: see Documentation/device-mapper/statistics.txt

Comment 2 Peter Rajnoha 2015-08-10 07:57:42 UTC
New "dmstats" commands has been added in recent lvm2 v2.02.127 upstream which is going to be included in RHEL 7.2 device-mapper subpackage (device-mapper v 1.02.104, a subpackage of lvm2).

See also man dmstats(8) for more info on usage.

Comment 4 Bryn M. Reeves 2015-08-11 08:17:44 UTC
A small number of correctness patches (from Coverity reports) that are worth including:

f072a76 libdm-stats: backtrace if fclose fails (Coverity)
856f9cc libdm: simplify stats nr_areas calculation (Coverity)
ec87e88 dmsetup: don't free handle if dm_stats_create fails (Coverity)
f9f5aac libdm: fix stats handle leak in dm_stats_create (Coverity)
1134de3 libdm: fix FILE leak in _program_id_from_proc() (Coverity)
3b74824 dmsetup: remove bogus !_report test in _stats_report (Coverity)

These are all minor or hard to trigger: i.e. tiny leaks (one FILE object per handle created with NULL or empty program_id) or bugs in hard-to-trigger error paths but they are simple and very low-risk changes that fix real problems.

Comment 5 Alasdair Kergon 2015-08-18 21:22:21 UTC
In the course of development we discovered that we were missing the ability for the kernel to report back to userspace the 'precise_statistics' and 'histogram' settings used for each region.  Without this information, the numbers that the kernel reports for these details cannot be interpreted.  If a single process is handling everything, that process knows what data it asked the kernel to collect, so things are OK, but if two separate processes are involved (i.e. two separate invocations of 'dmstats'), userspace has to record the configuration somewhere semi-persistent that they both can access.

Mikulas has provided a fix so that @stats_list reports this information back to userspace so that it doesn't have to be stored somewhere else:

https://www.redhat.com/archives/dm-devel/2015-August/msg00123.html

Comment 6 Bryn M. Reeves 2015-08-19 08:53:06 UTC
A viable workaround does exist: the kernel provides an "aux_data" storage facility for each registered region. This can be used to store the context information required to interpret the data for the region, e.g.:

  DMS_PRECISE,DMS_HIST=75000,100000,1000000,5000000,10000000|

Indicates a region with precise_timestamps and a 6-binned histogram.

The problem with this is that working around the lack of information from current kernels in this way would mean carrying this in libdevmapper userspace indefinitely. There are other planned uses for the feature however (e.g. grouping and aggregation) for which kernel support is not needed so the use of the aux_data area as a NVP store is likely to continue.

Comment 8 Jonathan Earl Brassow 2015-09-01 21:13:35 UTC
removing tech preview keyword.  We intend to fully support this feature in 7.2.

Comment 13 Bryn M. Reeves 2015-09-03 09:13:09 UTC
Most of the remaining histogram patches are now upstream (along with a couple of minor doc fixes and a cleanup to the stats report fields):

b86bd3b man: fix typo in dmstats.8.in
3414601 man: document --regionid in dmstats.8.in
d31c4e0 man: update dmstats.8.in examples
031cd2b dmstats: improve stats column names
3c0fc6f dmstats: add histogram support
a0cf3d4 libdm: add latency histogram support
c4f3732 libdm: reset report field widths in _destroy_rows()

The report patch is needed to ensure proper width calculation of histogram fields in repeating reports (since the first report is often very wide due to accumulated counts).

I'll be adding an additional patch today to control number unit formatting in histogram data which should make this feature complete.

Comment 14 Bryn M. Reeves 2015-09-03 21:17:46 UTC
49b5022 dmstats: support --noheadings for histogram fields
84d88cb dmstats: add --notimesuffix switch
f09e4f7 libdm: allow formatting histogram strings with no whitespace

Comment 15 Bryn M. Reeves 2015-09-08 09:19:22 UTC
There's one important histogram patch that landed after 1.02.107:

daa94eb libdm: fix display of nsec suffixes in histogram strings

This fixes a significant bug in the display of nsec values in histogram boundary strings.

Three other patches are cleanup and not essential:

d7f45eb libdm: clean up stats local variable use
5f99047 libdm: clean up _build_histogram_arg()
4bc7a86 libdm: only free the first histogram explicitly (Coverity)

Comment 17 Bruno Goncalves 2015-09-18 07:31:42 UTC
Verified using:
kernel-3.10.0-316.el7
lvm2-2.02.130-2.el7


# dmstats create --alldevices --precise
rhel_ibm--ls22--02-home: Created new region with 1 area(s) as region ID 0
rhel_ibm--ls22--02-swap: Created new region with 1 area(s) as region ID 0
rhel_ibm--ls22--02-root: Created new region with 1 area(s) as region ID 0

# dmstats list -o +precise
Name                    RgID RgSta RgSize #Areas ArSize ProgID Precise
rhel_ibm--ls22--02-home    0     0 77.54g      1 77.54g dmstats 1
rhel_ibm--ls22--02-swap    0     0  7.88g      1  7.88g dmstats 1
rhel_ibm--ls22--02-root    0     0 50.00g      1 50.00g dmstats 1

# dmstats create --start 10g --len 10g rhel_ibm--ls22--02-home
rhel_ibm--ls22--02-home: Created new region with 1 area(s) as region ID 1

# dmstats create --precise --bounds 10ms,20ms,30ms rhel_ibm--ls22--02-root
rhel_ibm--ls22--02-root: Created new region with 1 area(s) as region ID 1

# dmstats list --histogram
Name                    RgID RgStar RgSize #Areas ArSize #Bins     Histogram Bounds 
rhel_ibm--ls22--02-home    0      0 77.54g      1 77.54g 0                          
rhel_ibm--ls22--02-home    1 10.00g 10.00g      1 10.00g 0                          
rhel_ibm--ls22--02-swap    0      0  7.88g      1  7.88g 0                          
rhel_ibm--ls22--02-root    0      0 50.00g      1 50.00g 0                          
rhel_ibm--ls22--02-root    1      0 50.00g      1 50.00g 4         0s,10ms,20ms,30ms

# dmstats list -o +precise,hist_bins
Name                    RgID RgStar RgSize #Areas ArSize ProgID  Precise #Bins    
rhel_ibm--ls22--02-home    0      0 77.54g      1 77.54g dmstats 1       0        
rhel_ibm--ls22--02-home    1 10.00g 10.00g      1 10.00g dmstats 0       0        
rhel_ibm--ls22--02-swap    0      0  7.88g      1  7.88g dmstats 1       0        
rhel_ibm--ls22--02-root    0      0 50.00g      1 50.00g dmstats 1       0        
rhel_ibm--ls22--02-root    1      0 50.00g      1 50.00g dmstats 1       4      


# dmstats delete --regionid 1 rhel_ibm--ls22--02-home
#

# dmstats list --histogram
Name                    RgID RgSta RgSize #Areas ArSize #Bins     Histogram Bounds 
rhel_ibm--ls22--02-home    0     0 77.54g      1 77.54g 0                          
rhel_ibm--ls22--02-swap    0     0  7.88g      1  7.88g 0                          
rhel_ibm--ls22--02-root    0     0 50.00g      1 50.00g 0                          
rhel_ibm--ls22--02-root    1     0 50.00g      1 50.00g 4         0s,10ms,20ms,30ms

# dmstats delete --allregions rhel_ibm--ls22--02-home
#

# dmstats list --histogram
Name                    RgID RgSta RgSize #Areas ArSize #Bins     Histogram Bounds 
rhel_ibm--ls22--02-swap    0     0  7.88g      1  7.88g 0                          
rhel_ibm--ls22--02-root    0     0 50.00g      1 50.00g 0                          
rhel_ibm--ls22--02-root    1     0 50.00g      1 50.00g 4         0s,10ms,20ms,30ms


# dmstats delete --allregions --alldevices
#

# dmstats list --histogram
#

Comment 18 Steven J. Levine 2015-10-07 15:37:09 UTC
Updating the "doc text" field to prepare it for release notes. The way this now works is that the field is automatically picked up for the release notes build, which requires a single-sentence "title" summary followed by a blank line, then the description of the new feature. This is for "enhancements" specifically -- bug fixes have their standard bug fix formula.

Writers are now assigned the bugs for release notes to edit this text into that format (and to generally edit the texts, but Bryn needs no editing.)

So I added a title -- which is pretty much the first sentence of the Portal article draft -- and summarized what Bryn had written into one paragraph (which is his basic paragraph). I don't want to lose any information, of course, but the "reason" and "result" format made for some redundancy.

(This is by way of explanation.)

Comment 19 Steven J. Levine 2015-10-19 15:59:12 UTC
Editing title to be less than 120 characters.

Comment 20 errata-xmlrpc 2015-11-19 12:46:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2147.html


Note You need to log in before you can comment on or make changes to this bug.