RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1378956 - dmstats: support monitoring and updating of --filemap groups
Summary: dmstats: support monitoring and updating of --filemap groups
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Bryn M. Reeves
QA Contact: cluster-qe@redhat.com
Steven J. Levine
URL:
Whiteboard:
: 1376563 (view as bug list)
Depends On:
Blocks: 1385242
TreeView+ depends on / blocked
 
Reported: 2016-09-23 15:52 UTC by Bryn M. Reeves
Modified: 2021-09-03 12:38 UTC (History)
13 users (show)

Fixed In Version: lvm2-2.02.171-5.el7
Doc Type: Release Note
Doc Text:
The dmstats facility can now track the statistics for files that change Previously, the `dmstats` facility was able to report statistics for files that did not change in size. It now has the ability to watch files for changes and update its mappings to track file I/O even as the file changes in size (or fills holes that may be in the file).
Clone Of:
Environment:
Last Closed: 2017-08-01 21:47:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2222 0 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2017-08-01 18:42:41 UTC

Description Bryn M. Reeves 2016-09-23 15:52:29 UTC 
Description of problem:
When dmstats is used to create a group of regions mapping the extents of a file, it takes a single snapshot of the extent table and configures corresponding dmstats regions: this is fine for the case of statically allocated files (for e.g. non-sparse VM images, and other large, fully-allocated files where IO happens within the existing file boundaries).

Sparse files that are being written to, or any file that is extended past its current length, will trigger expansion of current extents, or allocation of new extents, causing the original dmstats region table to become stale over time (hole-punching APIs also mean that regions may be de-allocated at runtime, possibly leading to misleading values if those areas of disk are subsequently allocated to other, unrelated files).

Address this by monitoring the file with the inotify(7) facility and update the region table on mapping changes. Since reacting to a change will take a certain amount of time, and the file may continue to grow or shrink, rate limiting may be required and we cannot guarantee perfect accuracy, but for objects that are not growing at extreme rates, it should be possible to achieve a reasonable balance between frequency of updates and accuracy of the maintained file mapping.

This change will require changes to libdevmapper to allow updating of regions created via dm_stats_create_regions_from_fd() (e.g. 'dm_stats_refresh_regions_from_fd()'), and changes to dmsetup to set up the necessary background monitoring process.

Version-Release number of selected component (if applicable):
2.02.166

How reproducible:
100%

Steps to Reproduce:
1. dmstats create --filemap foo
2. truncate foo
3. dmstats list

Actual results:
All original --filemap regions remain.

Expected results:
No regions should exist after truncating the file.

Additional info:
Comment 1 Bryn M. Reeves 2016-09-23 15:53:39 UTC 
*** Bug 1376563 has been marked as a duplicate of this bug. ***
Comment 2 Bryn M. Reeves 2016-11-04 15:28:51 UTC 
I started prototyping this with fanotify. We can trap modifications to the file, and then inspect the allocated blocks (via stat()) to determine whether there are any changes (fanotify is implemented at the VFS level so does not generate distinct events for block or extent allocation).

I'll try to make more time to work on this before the end of the year - it would be good to have this in place with plenty of time for testing before 7.4.
Comment 3 Bryn M. Reeves 2016-12-21 16:42:11 UTC 
This is close to being ready for upstream now. There are three branches being worked on git.fedorahosted:

https://git.fedorahosted.org/cgit/lvm2.git/log/?h=dev-bmr-dmstats-filemap-update

  * Adds the basic --filemap update infrastructure: dm_stats_update_regions_from_fd()
  * Adds a new "dmstats update_filemap" command

https://git.fedorahosted.org/cgit/lvm2.git/log/?h=dev-bmr-dmstats-dmfilemapd

  * Adds a daemon, dmfilemapd, that monitors files and automatically updates on changes
  * Automatically launches the daemon from "create --filemap" (unless --nomonitor).

https://git.fedorahosted.org/cgit/lvm2.git/log/?h=dev-bmr-dmstats-dmfilemapd-resize

  * Experimental branch to improve tracking accuracy with fast-growing files.
  * Relies on a (not yet upstream) kernel patch to allow growing stats regions
Comment 5 Bryn M. Reeves 2017-03-13 16:50:06 UTC 
Branches for --filemap and dmfilemapd are now merged:

  dev-bmr-dmstats-filemap-update
  dev-bmr-dmstats-dmfilemapd
Comment 9 Bryn M. Reeves 2017-05-24 14:19:14 UTC 
Right now it's designed to launch one daemon per monitored file: the daemon is tiny, and very lightweight - it's not clear that it would actually use less resources overall if we had a 'shared' daemon since it would greatly increase the daemon's complexity (threading, communication etc.). Whether this will change in future depends somewhat on the experience we gain from how people use the feature.

Also today the daemon is a simple UNIX-style daemon: there's little functional benefit for file monitoring in implementing a systemd service but it's something we are considering - if only so that people can use the familiar management and inspection tools that systemd provides. This would be work for a future update.

> report still contains regions after truncating the file

This looks like a bug - I'm not sure exactly what's happening right now, but it's possible for some reason we are not getting an inotify event for this - I can reproduce and I'll look into it this week.
Comment 11 Bryn M. Reeves 2017-06-07 18:30:58 UTC 
There were two separate problems for the truncate case:

* The library treating a truncated file as an error (it's not in the update case but a lot of code is shared with creation, where it is an error).

* The daemon failing to notice that the file is truncated when truncation is the first event seen for a file

The first is the reason that the regions are left behind: the update process is aborted early on, before removing no-longer-mapped extents. We should carry on in this case, and just return an empty region table at the end (instead of returning error).

The second is a result of the fact that the "old size" field was only being updated at the time of the first check: if this coincided with the truncation then the size is already 0 by the time we check.

I'm testing fixes for these, as well as a minor change to improve the debug messages & avoid a pause when the daemon shuts down in this case:

commit 36590dd627395493cbf45db3cdcb73bd123f2d4a
Author: Bryn M. Reeves <bmr>
Date:   Wed Jun 7 19:29:16 2017 +0100

    dmfilemapd: do not wait if file has been truncated

commit 654a0c0454e37bfd0c95eaef447bdf8b7f9892be
Author: Bryn M. Reeves <bmr>
Date:   Wed Jun 7 19:26:09 2017 +0100

    dmfilemapd: update file block count at daemon start
    
    The file block count stored in the filemap_monitor was lazily
    initialised at the time of the first check. This causes problems
    in the case that the file has been truncated between this time and
    the time the daemon started: the initial block count and current
    block count match and the daemon fails to detect a change.
    
    Separate the setting of the block count from the check and make a
    call to update the value at the start of _dmfilemapd().

commit 36de0474f37e34d050d7facbfad42f1f86fd9a76
Author: Bryn M. Reeves <bmr>
Date:   Wed Jun 7 19:21:10 2017 +0100

    libdm: allow truncated files dm_stats_update_regions_from_fd()
    
    It's not an error to attempt to update regions from an fd that has
    been truncated (or otherwise no longer has any allocated extents):
    in this case, the call should remove all regions corresponding to
    the group, and return an empty region table.
Comment 12 Bryn M. Reeves 2017-06-13 18:49:16 UTC 
The patches in comment #11 are now upstream (744f292..9a09435).
Comment 13 Roman Bednář 2017-06-20 07:49:22 UTC 
Marking verified with latest rpms, truncating a file now removes the report.

# dmstats create --filemap /mnt/mount1/file1 /mnt/mount2/file2 /mnt/mount3/file3
/mnt/mount1/file1: Created new group with 2 region(s) as group ID 0.
/mnt/mount2/file2: Created new group with 1 region(s) as group ID 0.
/mnt/mount3/file3: Created new group with 2 region(s) as group ID 0.

# dmstats list --group
Name             GrpID RgID ObjType RgStart RgSize  #Areas ArSize  ProgID 
file2                0    0 group   136.00m 100.00m      1 100.00m dmstats
file1                0  0-1 group   136.00m 100.00m      1 100.00m dmstats
file3                0  0-1 group   136.00m 100.00m      1 100.00m dmstats

# truncate -s 0 /mnt/mount1/file1

# dmstats list --group
Name             GrpID RgID ObjType RgStart RgSize  #Areas ArSize  ProgID 
file2                0    0 group   136.00m 100.00m      1 100.00m dmstats
file3                0  0-1 group   136.00m 100.00m      1 100.00m dmstats

# dmstats list
Name             GrpID RgID ObjType RgStart RgSize  #Areas ArSize  ProgID 
file2                0    0 region  136.00m 100.00m      1 100.00m dmstats
file2                0    0 group   136.00m 100.00m      1 100.00m dmstats
file3                0    0 region  136.00m  96.00m      1  96.00m dmstats
file3                0    1 region  132.00m   4.00m      1   4.00m dmstats
file3                0  0-1 group   136.00m 100.00m      1 100.00m dmstats


3.10.0-682.el7.x86_64

lvm2-2.02.171-5.el7    BUILT: Wed Jun 14 17:33:32 CEST 2017
lvm2-libs-2.02.171-5.el7    BUILT: Wed Jun 14 17:33:32 CEST 2017
lvm2-cluster-2.02.171-5.el7    BUILT: Wed Jun 14 17:33:32 CEST 2017
device-mapper-1.02.140-5.el7    BUILT: Wed Jun 14 17:33:32 CEST 2017
device-mapper-libs-1.02.140-5.el7    BUILT: Wed Jun 14 17:33:32 CEST 2017
device-mapper-event-1.02.140-5.el7    BUILT: Wed Jun 14 17:33:32 CEST 2017
device-mapper-event-libs-1.02.140-5.el7    BUILT: Wed Jun 14 17:33:32 CEST 2017
device-mapper-persistent-data-0.7.0-0.1.rc6.el7    BUILT: Mon Mar 27 17:15:46 CEST 2017
Comment 14 Steven J. Levine 2017-06-21 23:27:26 UTC 
Bryn:

Brassow wrote up a description in the "Doc text" field for the RHEL 7.4 release notes and suggested that I run it by you for review.  Any improvements or corrections here?

Steven
Comment 15 Bryn M. Reeves 2017-06-22 11:21:56 UTC 
Steven,

Looks good to me as a short summary - I know we are very limited in release notes space so I think this is fine. We've updated the manual pages with all the details for users who are interested.

Bryn.
Comment 16 errata-xmlrpc 2017-08-01 21:47:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2222
Note You need to log in before you can comment on or make changes to this bug.