Bug 1282563
| Summary: | dmstats: add support for grouping and aggregation | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Bryn M. Reeves <bmr> |
| Component: | lvm2 | Assignee: | Bryn M. Reeves <bmr> |
| lvm2 sub component: | libdevmapper | QA Contact: | cluster-qe <cluster-qe> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | agk, bmr, cmarthal, heinzm, jbrassow, msnitzer, prajnoha, prockai, rbednar, zkabelac |
| Version: | 7.2 | Keywords: | FutureFeature |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Enhancement | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-04 04:12:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Bryn M. Reeves
2015-11-16 18:22:58 UTC
Basic functionality is working: Name Name RgID GrpID RgSta RgSiz vg_hex-lv_root foo 0 0 0 1.00g vg_hex-lv_root foo 1 0 1.00g 1.00g vg_hex-lv_root vg_hex-lv_root 2 - 2.00g 1.00g vg_hex-lv_root vg_hex-lv_root 3 - 3.00g 1.00g vg_hex-lv_root vg_hex-lv_root 4 - 4.00g 1.00g vg_hex-lv_root vg_hex-lv_root 5 - 5.00g 1.00g vg_hex-lv_root vg_hex-lv_root 6 - 6.00g 1.00g vg_hex-lv_root vg_hex-lv_root 7 - 7.00g 1.00g vg_hex-lv_root vg_hex-lv_root 8 - 8.00g 1.00g vg_hex-lv_root foo 0-1 0 0 2.00g # dmstats group --regions 2-4,8 --alias qux vg_hex/lv_root # dmstats list -oname,stats_name,region_id,group_id,region_start,region_len Name Name RgID GrpID RgSta RgSiz vg_hex-lv_root foo 0 0 0 1.00g vg_hex-lv_root foo 1 0 1.00g 1.00g vg_hex-lv_root qux 2 2 2.00g 1.00g vg_hex-lv_root qux 3 2 3.00g 1.00g vg_hex-lv_root qux 4 2 4.00g 1.00g vg_hex-lv_root vg_hex-lv_root 5 - 5.00g 1.00g vg_hex-lv_root vg_hex-lv_root 6 - 6.00g 1.00g vg_hex-lv_root vg_hex-lv_root 7 - 7.00g 1.00g vg_hex-lv_root qux 8 2 8.00g 1.00g vg_hex-lv_root foo 0-1 0 0 2.00g vg_hex-lv_root qux 2-4,8 2 2.00g 4.00g Mapping a file with a python script: # dmsfilemap.py /var/lib/libvirt/images/rhel7.0-1.qcow2 rhel7.0-1.qcow2 File size of /var/lib/libvirt/images/rhel7.0-1.qcow2 is 2148073472 (4195456 blocks of 512 bytes) Grouped regions 0-10 as group ID 0 on /dev/mapper/vg_hex-lv_images # dmstats list -oname,stats_name,region_id,group_id,region_start,region_len --statstype=group Name Name RgID GrpID RgSta RgSiz vg_hex-lv_root foo 0-1 0 0 2.00g vg_hex-lv_root qux 2-4,8 2 2.00g 4.00g vg_hex-lv_images rhel7.0-1.qcow2 0-10 0 8.90g 1.82m There may still be some changes to command line options and field names but the aim is to start getting this upstream this week. Draft update to the man page covering groups: http://paste.fedoraproject.org/384279/14667947/ New update to the man page: https://paste.fedoraproject.org/386711/67298191/ The latest (in troff markup) can always be found at: https://git.fedorahosted.org/cgit/lvm2.git/tree/man/dmstats.8.in?h=dev-bmr-dmstats-groups The main group support patches series is now just awaiting review: doc: update dmstats.8.in for groups dmstats: allow --statstype to override report defults dmstats: report a list of members as a group's region_id dmstats: report groups and region summaries dmstats: do not walk regions if deleting a single id dmstats: add stats_name field dmstats: add group alias support dmstats: add 'group' and 'ungroup' commands dmstats: add group_id report field type libdm: check for overlaps when grouping regions libdm: add dm_stats_get_group_regions() libdm: add stats group and region iterators and properties libdm: increase buffer size for @stats_list libdm: add dm_stats_set_region_aux_data() libdm: add statistics groups libdm: rename 'region' to 'skip_region' in _stats_walk_next libdm: add enum based counter and metric calls libdm: cache dm name in stats handle libdm: rename dm_stats name, devno and uuid members libdm: add dm_bitset_parse_list() dmsetup: fix timestamp leak dmstats: fix RgStart and RgSize field widths docs: remove obsolete note from dmstats.8.in The overall status seems good, the new documentation on groups is very well structured and understandable. Although I need more information on --statstype option (not working) and 'dmstats delete' which seems to be ungrouping everything if only one group member is removed. This might be intended behaviour. If so, please verify. For details see NEEDINFO section in my comment below.
Overview:
OK doc: update dmstats.8.in for groups
NOT OK dmstats: allow --statstype to override report defults
OK dmstats: report a list of members as a group's region_id
OK dmstats: report groups and region summaries
OK? dmstats: do not walk regions if deleting a single id
OK dmstats: add stats_name field
OK dmstats: add group alias support
OK dmstats: add 'group' and 'ungroup' commands
OK dmstats: add group_id report field type
===============PASSED===============
# lvs -a -S vg_name=vg
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
test_lv vg -wi-ao---- 39.99g
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len
Name Name RgID GrpID RgStart RgSize
vg-test_lv vg-test_lv 0 - 0 1.00g
vg-test_lv vg-test_lv 1 - 1.00g 1.00g
vg-test_lv vg-test_lv 2 - 2.00g 1.00g
vg-test_lv vg-test_lv 3 - 3.00g 1.00g
vg-test_lv vg-test_lv 4 - 4.00g 1.00g
vg-test_lv vg-test_lv 5 - 5.00g 1.00g
# dmstats group --regions 1-3,5 --alias foo vg/test_lv
Grouped regions 1-3,5 as group ID 1 on vg/test_lv
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len
Name Name RgID GrpID RgStart RgSize
vg-test_lv vg-test_lv 0 - 0 1.00g
vg-test_lv foo 1 1 1.00g 1.00g
vg-test_lv foo 2 1 2.00g 1.00g
vg-test_lv foo 3 1 3.00g 1.00g
vg-test_lv vg-test_lv 4 - 4.00g 1.00g
vg-test_lv foo 5 1 5.00g 1.00g
vg-test_lv foo 1-3,5 1 1.00g 4.00g
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len -S group_id=1
Name Name RgID GrpID RgStart RgSize
vg-test_lv foo 1 1 1.00g 1.00g
vg-test_lv foo 2 1 2.00g 1.00g
vg-test_lv foo 3 1 3.00g 1.00g
vg-test_lv foo 5 1 5.00g 1.00g
vg-test_lv foo 1-3,5 1 1.00g 4.00g
# dmstats ungroup --groupid=1 vg/test_lv
Removed group ID 1 on vg/test_lv
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len
Name Name RgID GrpID RgStart RgSize
vg-test_lv vg-test_lv 0 - 0 1.00g
vg-test_lv vg-test_lv 1 - 1.00g 1.00g
vg-test_lv vg-test_lv 2 - 2.00g 1.00g
vg-test_lv vg-test_lv 3 - 3.00g 1.00g
vg-test_lv vg-test_lv 4 - 4.00g 1.00g
vg-test_lv vg-test_lv 5 - 5.00g 1.00g
===============NEEDINFO===============
>>>Whole group gets ungrouped when only one "member" region is deleted:
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len
Name Name RgID GrpID RgStart RgSize
vg-test_lv vg-test_lv 0 - 0 1.00g
vg-test_lv foo 1 1 1.00g 1.00g
vg-test_lv foo 2 1 2.00g 1.00g
vg-test_lv foo 3 1 3.00g 1.00g
vg-test_lv vg-test_lv 4 - 4.00g 1.00g
vg-test_lv foo 5 1 5.00g 1.00g
vg-test_lv foo 1-3,5 1 1.00g 4.00g
# dmstats delete --regionid=1 vg/test_lv
# dmstats list -oname,stats_name,region_id,group_id,region_start,region_len
Name Name RgID GrpID RgStart RgSize
vg-test_lv vg-test_lv 0 - 0 1.00g
vg-test_lv vg-test_lv 2 - 2.00g 1.00g
vg-test_lv vg-test_lv 3 - 3.00g 1.00g
vg-test_lv vg-test_lv 4 - 4.00g 1.00g
vg-test_lv vg-test_lv 5 - 5.00g 1.00g
>>>--statstype does not seem to work, if it's been removed/changed it needs to be reflected in manpages
# dmstats list -o name,stats_name,region_id,group_id,region_start,region_len --statstype=group
dmstats: unrecognized option '--statstype=group'
Couldn't process command line
=====================================================
3.10.0-460.el7.x86_64
lvm2-2.02.160-1.el7 BUILT: Wed Jul 6 18:16:47 CEST 2016
lvm2-libs-2.02.160-1.el7 BUILT: Wed Jul 6 18:16:47 CEST 2016
lvm2-cluster-2.02.160-1.el7 BUILT: Wed Jul 6 18:16:47 CEST 2016
device-mapper-1.02.130-1.el7 BUILT: Wed Jul 6 18:16:47 CEST 2016
device-mapper-libs-1.02.130-1.el7 BUILT: Wed Jul 6 18:16:47 CEST 2016
device-mapper-event-1.02.130-1.el7 BUILT: Wed Jul 6 18:16:47 CEST 2016
device-mapper-event-libs-1.02.130-1.el7 BUILT: Wed Jul 6 18:16:47 CEST 2016
device-mapper-persistent-data-0.6.2-1.el7 BUILT: Mon Jul 11 11:32:34 CEST 2016
cmirror-2.02.160-1.el7 BUILT: Wed Jul 6 18:16:47 CEST 2016
> Although I need more information on --statstype option (not working) Sorry - my fault. Where are you still seeing references to this? NOT OK dmstats: allow --statstype to override report defults Should now be: dmstats: allow --area, --region, and --group to override report defaults We dropped the --statstype={region,area,group} switch in favour of the separate --area, --region, and --group - they should function in almost the same way (presence of a type includes those objects in the list / report). I'll get these updated today. > 'dmstats delete' which seems to be ungrouping everything if only one group > member is removed Yes, it's intended - I'll add this to the documentation. The first member of a group is special in that it stores the group metadata (so deleting that member will cause the entire group to vanish). This seemed better than trying to do complicated things like "moving" the group to the next region_id (although that could be done technically if it is thought to be more desirable). Do you think a warning message or other clarification would be useful? >Sorry - my fault. Where are you still seeing references to this? In the manpages. Functionality of the new options confirmed: # dmstats report vg/test_lv --area --group --region Name GrpID RgID ObjType ArID ArStart ArSize RMrg/s WMrg/s R/s W/s RSz/s WSz/s AvgRqSz QSize Util% AWait RdAWait WrAWait vg-test_lv - 0 area 0 0 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 foo 1 1 area 0 1.00g 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 foo 1 2 area 0 2.00g 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 foo 1 3 area 0 3.00g 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 vg-test_lv - 4 area 0 4.00g 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 vg-test_lv - 0 region 0 0 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 foo 1 1 region 0 1.00g 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 foo 1 2 region 0 2.00g 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 foo 1 3 region 0 3.00g 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 vg-test_lv - 4 region 0 4.00g 1.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 foo 1 1-3 group 0 1.00g 3.00g 0.00 0.00 0.00 0.00 0 0 0 0.00 0.00 0.00 0.00 0.00 >Yes, it's intended - I'll add this to the documentation. Perfect, thank you for explanation. When a region without metadata is removed the grouping remains: # dmstats list -oname,stats_name,region_id,group_id,region_start,region_len Name Name RgID GrpID RgStart RgSize vg-test_lv vg-test_lv 0 - 0 1.00g vg-test_lv foo 1 1 1.00g 1.00g vg-test_lv foo 2 1 2.00g 1.00g vg-test_lv foo 3 1 3.00g 1.00g vg-test_lv vg-test_lv 4 - 4.00g 1.00g vg-test_lv foo 5 1 5.00g 1.00g vg-test_lv foo 1-3,5 1 1.00g 4.00g # dmstats delete --regionid=5 vg/test_lv # dmstats list -oname,stats_name,region_id,group_id,region_start,region_len Name Name RgID GrpID RgStart RgSize vg-test_lv vg-test_lv 0 - 0 1.00g vg-test_lv foo 1 1 1.00g 1.00g vg-test_lv foo 2 1 2.00g 1.00g vg-test_lv foo 3 1 3.00g 1.00g vg-test_lv vg-test_lv 4 - 4.00g 1.00g vg-test_lv foo 1-3 1 1.00g 3.00g >Do you think a warning message or other clarification would be useful? I believe a warning message or confirmation prompt would be a great addition so users don't end up deleting the whole group by accident. However this does not delete any 'real' data and the group can be recreated easily, so a warning message should be enough to let user know about group metadata removal/whole group removal instead of just doing that silently. Both solutions are fine. Conclusion: changes to be applied in the documentation: 1) removal of --statstype references and adding --area --region --group options 2) explanation on groups metadata location and removal optional: 3)add a warning or confirmation upon removing a member containing metadata (first member of a group) The following changes were committed to the dmstats.8 man page for LVM2-2.02.163:
commit dad02900f18d3ef9b85d78f2e42434d92e6d84c9
Author: Bryn M. Reeves <bmr>
Date: Mon Aug 8 19:29:12 2016 +0100
man: explain deletion of 1st group member in dmstats.8.in
Although the use of the first region_id in a group to store the
DMS_GROUP=... aux_data tag is an internal implementation detail,
it has a user visible consequence in that deleting this region will
cause the group to disappear: add an explanation of this to the
'group' command and 'Regions, areas, and groups' section.
commit 4bcbcdb1a2f276b8139ae0ee7f66ffb342c03f23
Author: Bryn M. Reeves <bmr>
Date: Mon Aug 8 19:06:40 2016 +0100
man: minor fixes to dmstats.8.in
Remove obsolete reference to '--target'.
They should show up in the latest build.
Marking as verified. 3.10.0-487.el7.x86_64 lvm2-2.02.163-1.el7 BUILT: Wed Aug 10 06:53:21 CDT 2016 lvm2-libs-2.02.163-1.el7 BUILT: Wed Aug 10 06:53:21 CDT 2016 lvm2-cluster-2.02.163-1.el7 BUILT: Wed Aug 10 06:53:21 CDT 2016 device-mapper-1.02.133-1.el7 BUILT: Wed Aug 10 06:53:21 CDT 2016 device-mapper-libs-1.02.133-1.el7 BUILT: Wed Aug 10 06:53:21 CDT 2016 device-mapper-event-1.02.133-1.el7 BUILT: Wed Aug 10 06:53:21 CDT 2016 device-mapper-event-libs-1.02.133-1.el7 BUILT: Wed Aug 10 06:53:21 CDT 2016 device-mapper-persistent-data-0.6.3-1.el7 BUILT: Fri Jul 22 05:29:13 CDT 2016 cmirror-2.02.163-1.el7 BUILT: Wed Aug 10 06:53:21 CDT 2016 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1445.html |