RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1282563 - dmstats: add support for grouping and aggregation
Summary: dmstats: add support for grouping and aggregation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.2
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Bryn M. Reeves
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-11-16 18:22 UTC by Bryn M. Reeves
Modified: 2021-09-03 12:39 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-04 04:12:19 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1445 0 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2016-11-03 13:46:41 UTC

Description Bryn M. Reeves 2015-11-16 18:22:58 UTC 
Description of problem:
The current dmstats command (and corresponding library interfaces) do not support grouping or aggregation of device statistics regions.

It should be possible to specify a set of existing region IDs to be grouped, or to be aggregated via the command line and C API.

Grouping makes an additional (default hidden) field available containing a group identifier that can be used for sorting and selection operations.

Aggregation works similarly to grouping except that the rows of data for individual regions are suppressed and reported as a single aggregate row for the group.

Version-Release number of selected component (if applicable):
lvm2-2.02.130-*.el7

How reproducible:
100%

Steps to Reproduce:
1. man dmstats

Actual results:
No aggregation / grouping support.

Expected results:

Aggregation and grouping are documented and available in the command-line tools and C API.

Additional info:

Examples:

# dmstats group --regionid=1-3,5,7 vg_hex-lv_root
0

# dmstats report -o region_id,group_id
RgId    GrpId
1       0
2       0
3       0
5       0
7       0

# dmstats report -S group_id=0
Name             RgID ArID ArStart ArSize RMrg/s WMrg/s R/s  W/s    RSz/s WSz/s AvgRqSz QSize Util% AWait RdAWait WrAWait
vg_hex-lv_root      0    0       0 25.00g   0.00   0.00 0.00 209.00     0 1.09m   5.00k  0.48  3.10  2.30    0.00    2.30
vg_hex-lv_root      1    0       0 25.00g   0.00   0.00 0.00   0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
vg_hex-lv_root      2    0       0 25.00g   0.00   0.00 0.00   0.00     0     0       0  0.00  0.00  0.00    0.00    0.00


# dmstats group --aggregate --regionid=1-3,5,7 vg_hex-lv_root
1

# dmstats report -o region_id,group_id
RgId    GrpId
1       1
2       1
3       1
5       1
7       1

# dmstats report -S group_id=1
Name                                      RgID ArID ArStart ArSize  RMrg/s WMrg/s R/s   W/s    RSz/s  WSz/s   AvgRqSz QSize Util% AWait RdAWait WrAWait
vg_hex-lv_root                               0    0       0  25.00g   0.00   0.00 83.00 172.00  1.77m 740.00k   9.50k  0.38  4.20  1.47    0.28    2.05
Comment 3 Bryn M. Reeves 2016-06-20 17:27:08 UTC 
Basic functionality is working:

Name             Name             RgID GrpID RgSta RgSiz
vg_hex-lv_root   foo                 0     0     0 1.00g
vg_hex-lv_root   foo                 1     0 1.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root      2     - 2.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root      3     - 3.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root      4     - 4.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root      5     - 5.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root      6     - 6.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root      7     - 7.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root      8     - 8.00g 1.00g
vg_hex-lv_root   foo               0-1     0     0 2.00g

# dmstats group --regions 2-4,8 --alias qux vg_hex/lv_root

# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len
Name             Name             RgID GrpID RgSta RgSiz
vg_hex-lv_root   foo                  0     0     0 1.00g
vg_hex-lv_root   foo                  1     0 1.00g 1.00g
vg_hex-lv_root   qux                  2     2 2.00g 1.00g
vg_hex-lv_root   qux                  3     2 3.00g 1.00g
vg_hex-lv_root   qux                  4     2 4.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root       5     - 5.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root       6     - 6.00g 1.00g
vg_hex-lv_root   vg_hex-lv_root       7     - 7.00g 1.00g
vg_hex-lv_root   qux                  8     2 8.00g 1.00g
vg_hex-lv_root   foo                0-1     0     0 2.00g
vg_hex-lv_root   qux              2-4,8     2 2.00g 4.00g

Mapping a file with a python script:
# dmsfilemap.py /var/lib/libvirt/images/rhel7.0-1.qcow2 rhel7.0-1.qcow2 
File size of /var/lib/libvirt/images/rhel7.0-1.qcow2 is 2148073472 (4195456 blocks of 512 bytes)
Grouped regions 0-10 as group ID 0 on /dev/mapper/vg_hex-lv_images

# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len --statstype=group
Name             Name             RgID GrpID RgSta RgSiz
vg_hex-lv_root   foo                0-1     0     0 2.00g
vg_hex-lv_root   qux              2-4,8     2 2.00g 4.00g
vg_hex-lv_images rhel7.0-1.qcow2   0-10     0 8.90g 1.82m

There may still be some changes to command line options and field names but the aim is to start getting this upstream this week.
Comment 4 Bryn M. Reeves 2016-06-24 19:38:30 UTC 
Draft update to the man page covering groups:

  http://paste.fedoraproject.org/384279/14667947/
Comment 5 Bryn M. Reeves 2016-06-30 16:10:38 UTC 
New update to the man page:

https://paste.fedoraproject.org/386711/67298191/

The latest (in troff markup) can always be found at:

https://git.fedorahosted.org/cgit/lvm2.git/tree/man/dmstats.8.in?h=dev-bmr-dmstats-groups
Comment 6 Bryn M. Reeves 2016-06-30 16:12:02 UTC 
The main group support patches series is now just awaiting review:

doc: update dmstats.8.in for groups
dmstats: allow --statstype to override report defults
dmstats: report a list of members as a group's region_id
dmstats: report groups and region summaries
dmstats: do not walk regions if deleting a single id
dmstats: add stats_name field
dmstats: add group alias support
dmstats: add 'group' and 'ungroup' commands
dmstats: add group_id report field type
libdm: check for overlaps when grouping regions
libdm: add dm_stats_get_group_regions()
libdm: add stats group and region iterators and properties
libdm: increase buffer size for @stats_list
libdm: add dm_stats_set_region_aux_data()
libdm: add statistics groups
libdm: rename 'region' to 'skip_region' in _stats_walk_next
libdm: add enum based counter and metric calls
libdm: cache dm name in stats handle
libdm: rename dm_stats name, devno and uuid members
libdm: add dm_bitset_parse_list()
dmsetup: fix timestamp leak
dmstats: fix RgStart and RgSize field widths
docs: remove obsolete note from dmstats.8.in
Comment 8 Roman Bednář 2016-07-15 11:03:54 UTC 
The overall status seems good, the new documentation on groups is very well structured and understandable. Although I need more information on --statstype option (not working) and 'dmstats delete' which seems to be ungrouping everything if only one group member is removed. This might be intended behaviour. If so, please verify. For details see NEEDINFO section in my comment below.


Overview:

OK  	doc: update dmstats.8.in for groups
NOT OK	dmstats: allow --statstype to override report defults
OK	dmstats: report a list of members as a group's region_id
OK	dmstats: report groups and region summaries
OK?	dmstats: do not walk regions if deleting a single id
OK	dmstats: add stats_name field
OK	dmstats: add group alias support
OK	dmstats: add 'group' and 'ungroup' commands
OK	dmstats: add group_id report field type

===============PASSED===============
# lvs -a -S vg_name=vg
  LV      VG Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  test_lv vg -wi-ao---- 39.99g 
                                                   
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len
Name             Name             RgID GrpID RgStart RgSize
vg-test_lv       vg-test_lv          0     -       0  1.00g
vg-test_lv       vg-test_lv          1     -   1.00g  1.00g
vg-test_lv       vg-test_lv          2     -   2.00g  1.00g
vg-test_lv       vg-test_lv          3     -   3.00g  1.00g
vg-test_lv       vg-test_lv          4     -   4.00g  1.00g
vg-test_lv       vg-test_lv          5     -   5.00g  1.00g

# dmstats group --regions 1-3,5 --alias foo vg/test_lv
Grouped regions 1-3,5 as group ID 1 on vg/test_lv

# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len
Name             Name             RgID  GrpID RgStart RgSize
vg-test_lv       vg-test_lv           0     -       0  1.00g
vg-test_lv       foo                  1     1   1.00g  1.00g
vg-test_lv       foo                  2     1   2.00g  1.00g
vg-test_lv       foo                  3     1   3.00g  1.00g
vg-test_lv       vg-test_lv           4     -   4.00g  1.00g
vg-test_lv       foo                  5     1   5.00g  1.00g
vg-test_lv       foo              1-3,5     1   1.00g  4.00g

# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len -S group_id=1
Name             Name             RgID  GrpID RgStart RgSize
vg-test_lv       foo                  1     1   1.00g  1.00g
vg-test_lv       foo                  2     1   2.00g  1.00g
vg-test_lv       foo                  3     1   3.00g  1.00g
vg-test_lv       foo                  5     1   5.00g  1.00g
vg-test_lv       foo              1-3,5     1   1.00g  4.00g

# dmstats ungroup --groupid=1 vg/test_lv
Removed group ID 1 on vg/test_lv
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len 
Name             Name             RgID GrpID RgStart RgSize
vg-test_lv       vg-test_lv          0     -       0  1.00g
vg-test_lv       vg-test_lv          1     -   1.00g  1.00g
vg-test_lv       vg-test_lv          2     -   2.00g  1.00g
vg-test_lv       vg-test_lv          3     -   3.00g  1.00g
vg-test_lv       vg-test_lv          4     -   4.00g  1.00g
vg-test_lv       vg-test_lv          5     -   5.00g  1.00g


===============NEEDINFO===============
>>>Whole group gets ungrouped when only one "member" region is deleted:
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len 
Name             Name             RgID  GrpID RgStart RgSize
vg-test_lv       vg-test_lv           0     -       0  1.00g
vg-test_lv       foo                  1     1   1.00g  1.00g
vg-test_lv       foo                  2     1   2.00g  1.00g
vg-test_lv       foo                  3     1   3.00g  1.00g
vg-test_lv       vg-test_lv           4     -   4.00g  1.00g
vg-test_lv       foo                  5     1   5.00g  1.00g
vg-test_lv       foo              1-3,5     1   1.00g  4.00g

# dmstats delete --regionid=1 vg/test_lv
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len 
Name             Name             RgID GrpID RgStart RgSize
vg-test_lv       vg-test_lv          0     -       0  1.00g
vg-test_lv       vg-test_lv          2     -   2.00g  1.00g
vg-test_lv       vg-test_lv          3     -   3.00g  1.00g
vg-test_lv       vg-test_lv          4     -   4.00g  1.00g
vg-test_lv       vg-test_lv          5     -   5.00g  1.00g

>>>--statstype does not seem to work, if it's been removed/changed it needs to be reflected in manpages
# dmstats list -o name,stats_name,region_id,group_id,region_start,region_len --statstype=group
dmstats: unrecognized option '--statstype=group'
Couldn't process command line

=====================================================
3.10.0-460.el7.x86_64

lvm2-2.02.160-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
lvm2-libs-2.02.160-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
lvm2-cluster-2.02.160-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-1.02.130-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-libs-1.02.130-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-event-1.02.130-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-event-libs-1.02.130-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
device-mapper-persistent-data-0.6.2-1.el7    BUILT: Mon Jul 11 11:32:34 CEST 2016
cmirror-2.02.160-1.el7    BUILT: Wed Jul  6 18:16:47 CEST 2016
Comment 9 Bryn M. Reeves 2016-07-15 11:12:58 UTC 
> Although I need more information on --statstype option (not working)

Sorry - my fault. Where are you still seeing references to this?

NOT OK	dmstats: allow --statstype to override report defults

Should now be:

        dmstats: allow --area, --region, and --group to override report defaults

We dropped the --statstype={region,area,group} switch in favour of the separate --area, --region, and --group - they should function in almost the same way (presence of a type includes those objects in the list / report).

I'll get these updated today.

> 'dmstats delete' which seems to be ungrouping everything if only one group 
> member is removed

Yes, it's intended - I'll add this to the documentation. The first member of a group is special in that it stores the group metadata (so deleting that member will cause the entire group to vanish). This seemed better than trying to do complicated things like "moving" the group to the next region_id (although that could be done technically if it is thought to be more desirable).

Do you think a warning message or other clarification would be useful?
Comment 10 Roman Bednář 2016-07-15 12:13:08 UTC 
>Sorry - my fault. Where are you still seeing references to this?
In the manpages.
 
Functionality of the new options confirmed:

# dmstats report vg/test_lv --area --group --region
Name             GrpID RgID ObjType ArID ArStart ArSize RMrg/s WMrg/s R/s  W/s  RSz/s WSz/s AvgRqSz QSize Util% AWait RdAWait WrAWait
vg-test_lv           -    0 area       0       0  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
foo                  1    1 area       0   1.00g  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
foo                  1    2 area       0   2.00g  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
foo                  1    3 area       0   3.00g  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
vg-test_lv           -    4 area       0   4.00g  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
vg-test_lv           -    0 region     0       0  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
foo                  1    1 region     0   1.00g  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
foo                  1    2 region     0   2.00g  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
foo                  1    3 region     0   3.00g  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
vg-test_lv           -    4 region     0   4.00g  1.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
foo                  1  1-3 group      0   1.00g  3.00g   0.00   0.00 0.00 0.00     0     0       0  0.00  0.00  0.00    0.00    0.00
 

>Yes, it's intended - I'll add this to the documentation.

Perfect, thank you for explanation.

When a region without metadata is removed the grouping remains:

# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len 
Name             Name             RgID  GrpID RgStart RgSize
vg-test_lv       vg-test_lv           0     -       0  1.00g
vg-test_lv       foo                  1     1   1.00g  1.00g
vg-test_lv       foo                  2     1   2.00g  1.00g
vg-test_lv       foo                  3     1   3.00g  1.00g
vg-test_lv       vg-test_lv           4     -   4.00g  1.00g
vg-test_lv       foo                  5     1   5.00g  1.00g
vg-test_lv       foo              1-3,5     1   1.00g  4.00g
# dmstats delete --regionid=5 vg/test_lv
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len 
Name             Name             RgID GrpID RgStart RgSize
vg-test_lv       vg-test_lv          0     -       0  1.00g
vg-test_lv       foo                 1     1   1.00g  1.00g
vg-test_lv       foo                 2     1   2.00g  1.00g
vg-test_lv       foo                 3     1   3.00g  1.00g
vg-test_lv       vg-test_lv          4     -   4.00g  1.00g
vg-test_lv       foo               1-3     1   1.00g  3.00g



>Do you think a warning message or other clarification would be useful?

I believe a warning message or confirmation prompt would be a great addition so users don't end up deleting the whole group by accident. However this does not delete any 'real' data and the group can be recreated easily, so a warning message should be enough to let user know about group metadata removal/whole group removal instead of just doing that silently. Both solutions are fine.


Conclusion:

changes to be applied in the documentation: 
1) removal of --statstype references and adding --area --region --group options
2) explanation on groups metadata location and removal

optional:
3)add a warning or confirmation upon removing a member containing metadata (first member of a group)
Comment 12 Bryn M. Reeves 2016-08-10 10:12:56 UTC 
The following changes were committed to the dmstats.8 man page for LVM2-2.02.163:

commit dad02900f18d3ef9b85d78f2e42434d92e6d84c9
Author: Bryn M. Reeves <bmr>
Date:   Mon Aug 8 19:29:12 2016 +0100

    man: explain deletion of 1st group member in dmstats.8.in
    
    Although the use of the first region_id in a group to store the
    DMS_GROUP=... aux_data tag is an internal implementation detail,
    it has a user visible consequence in that deleting this region will
    cause the group to disappear: add an explanation of this to the
    'group' command and 'Regions, areas, and groups' section.

commit 4bcbcdb1a2f276b8139ae0ee7f66ffb342c03f23
Author: Bryn M. Reeves <bmr>
Date:   Mon Aug 8 19:06:40 2016 +0100

    man: minor fixes to dmstats.8.in
    
    Remove obsolete reference to '--target'.

They should show up in the latest build.
Comment 13 Roman Bednář 2016-08-11 13:59:32 UTC 
Marking as verified.

3.10.0-487.el7.x86_64

lvm2-2.02.163-1.el7    BUILT: Wed Aug 10 06:53:21 CDT 2016
lvm2-libs-2.02.163-1.el7    BUILT: Wed Aug 10 06:53:21 CDT 2016
lvm2-cluster-2.02.163-1.el7    BUILT: Wed Aug 10 06:53:21 CDT 2016
device-mapper-1.02.133-1.el7    BUILT: Wed Aug 10 06:53:21 CDT 2016
device-mapper-libs-1.02.133-1.el7    BUILT: Wed Aug 10 06:53:21 CDT 2016
device-mapper-event-1.02.133-1.el7    BUILT: Wed Aug 10 06:53:21 CDT 2016
device-mapper-event-libs-1.02.133-1.el7    BUILT: Wed Aug 10 06:53:21 CDT 2016
device-mapper-persistent-data-0.6.3-1.el7    BUILT: Fri Jul 22 05:29:13 CDT 2016
cmirror-2.02.163-1.el7    BUILT: Wed Aug 10 06:53:21 CDT 2016
Comment 15 errata-xmlrpc 2016-11-04 04:12:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1445.html
Note You need to log in before you can comment on or make changes to this bug.