Hide Forgot
Description of problem: When dmstats is used to create a group of regions mapping the extents of a file, it takes a single snapshot of the extent table and configures corresponding dmstats regions: this is fine for the case of statically allocated files (for e.g. non-sparse VM images, and other large, fully-allocated files where IO happens within the existing file boundaries). Sparse files that are being written to, or any file that is extended past its current length, will trigger expansion of current extents, or allocation of new extents, causing the original dmstats region table to become stale over time (hole-punching APIs also mean that regions may be de-allocated at runtime, possibly leading to misleading values if those areas of disk are subsequently allocated to other, unrelated files). Address this by monitoring the file with the inotify(7) facility and update the region table on mapping changes. Since reacting to a change will take a certain amount of time, and the file may continue to grow or shrink, rate limiting may be required and we cannot guarantee perfect accuracy, but for objects that are not growing at extreme rates, it should be possible to achieve a reasonable balance between frequency of updates and accuracy of the maintained file mapping. This change will require changes to libdevmapper to allow updating of regions created via dm_stats_create_regions_from_fd() (e.g. 'dm_stats_refresh_regions_from_fd()'), and changes to dmsetup to set up the necessary background monitoring process. Version-Release number of selected component (if applicable): 2.02.166 How reproducible: 100% Steps to Reproduce: 1. dmstats create --filemap foo 2. truncate foo 3. dmstats list Actual results: All original --filemap regions remain. Expected results: No regions should exist after truncating the file. Additional info:
*** Bug 1376563 has been marked as a duplicate of this bug. ***
I started prototyping this with fanotify. We can trap modifications to the file, and then inspect the allocated blocks (via stat()) to determine whether there are any changes (fanotify is implemented at the VFS level so does not generate distinct events for block or extent allocation). I'll try to make more time to work on this before the end of the year - it would be good to have this in place with plenty of time for testing before 7.4.
This is close to being ready for upstream now. There are three branches being worked on git.fedorahosted: https://git.fedorahosted.org/cgit/lvm2.git/log/?h=dev-bmr-dmstats-filemap-update * Adds the basic --filemap update infrastructure: dm_stats_update_regions_from_fd() * Adds a new "dmstats update_filemap" command https://git.fedorahosted.org/cgit/lvm2.git/log/?h=dev-bmr-dmstats-dmfilemapd * Adds a daemon, dmfilemapd, that monitors files and automatically updates on changes * Automatically launches the daemon from "create --filemap" (unless --nomonitor). https://git.fedorahosted.org/cgit/lvm2.git/log/?h=dev-bmr-dmstats-dmfilemapd-resize * Experimental branch to improve tracking accuracy with fast-growing files. * Relies on a (not yet upstream) kernel patch to allow growing stats regions
Branches for --filemap and dmfilemapd are now merged: dev-bmr-dmstats-filemap-update dev-bmr-dmstats-dmfilemapd
Right now it's designed to launch one daemon per monitored file: the daemon is tiny, and very lightweight - it's not clear that it would actually use less resources overall if we had a 'shared' daemon since it would greatly increase the daemon's complexity (threading, communication etc.). Whether this will change in future depends somewhat on the experience we gain from how people use the feature. Also today the daemon is a simple UNIX-style daemon: there's little functional benefit for file monitoring in implementing a systemd service but it's something we are considering - if only so that people can use the familiar management and inspection tools that systemd provides. This would be work for a future update. > report still contains regions after truncating the file This looks like a bug - I'm not sure exactly what's happening right now, but it's possible for some reason we are not getting an inotify event for this - I can reproduce and I'll look into it this week.
There were two separate problems for the truncate case: * The library treating a truncated file as an error (it's not in the update case but a lot of code is shared with creation, where it is an error). * The daemon failing to notice that the file is truncated when truncation is the first event seen for a file The first is the reason that the regions are left behind: the update process is aborted early on, before removing no-longer-mapped extents. We should carry on in this case, and just return an empty region table at the end (instead of returning error). The second is a result of the fact that the "old size" field was only being updated at the time of the first check: if this coincided with the truncation then the size is already 0 by the time we check. I'm testing fixes for these, as well as a minor change to improve the debug messages & avoid a pause when the daemon shuts down in this case: commit 36590dd627395493cbf45db3cdcb73bd123f2d4a Author: Bryn M. Reeves <bmr> Date: Wed Jun 7 19:29:16 2017 +0100 dmfilemapd: do not wait if file has been truncated commit 654a0c0454e37bfd0c95eaef447bdf8b7f9892be Author: Bryn M. Reeves <bmr> Date: Wed Jun 7 19:26:09 2017 +0100 dmfilemapd: update file block count at daemon start The file block count stored in the filemap_monitor was lazily initialised at the time of the first check. This causes problems in the case that the file has been truncated between this time and the time the daemon started: the initial block count and current block count match and the daemon fails to detect a change. Separate the setting of the block count from the check and make a call to update the value at the start of _dmfilemapd(). commit 36de0474f37e34d050d7facbfad42f1f86fd9a76 Author: Bryn M. Reeves <bmr> Date: Wed Jun 7 19:21:10 2017 +0100 libdm: allow truncated files dm_stats_update_regions_from_fd() It's not an error to attempt to update regions from an fd that has been truncated (or otherwise no longer has any allocated extents): in this case, the call should remove all regions corresponding to the group, and return an empty region table.
The patches in comment #11 are now upstream (744f292..9a09435).
Marking verified with latest rpms, truncating a file now removes the report. # dmstats create --filemap /mnt/mount1/file1 /mnt/mount2/file2 /mnt/mount3/file3 /mnt/mount1/file1: Created new group with 2 region(s) as group ID 0. /mnt/mount2/file2: Created new group with 1 region(s) as group ID 0. /mnt/mount3/file3: Created new group with 2 region(s) as group ID 0. # dmstats list --group Name GrpID RgID ObjType RgStart RgSize #Areas ArSize ProgID file2 0 0 group 136.00m 100.00m 1 100.00m dmstats file1 0 0-1 group 136.00m 100.00m 1 100.00m dmstats file3 0 0-1 group 136.00m 100.00m 1 100.00m dmstats # truncate -s 0 /mnt/mount1/file1 # dmstats list --group Name GrpID RgID ObjType RgStart RgSize #Areas ArSize ProgID file2 0 0 group 136.00m 100.00m 1 100.00m dmstats file3 0 0-1 group 136.00m 100.00m 1 100.00m dmstats # dmstats list Name GrpID RgID ObjType RgStart RgSize #Areas ArSize ProgID file2 0 0 region 136.00m 100.00m 1 100.00m dmstats file2 0 0 group 136.00m 100.00m 1 100.00m dmstats file3 0 0 region 136.00m 96.00m 1 96.00m dmstats file3 0 1 region 132.00m 4.00m 1 4.00m dmstats file3 0 0-1 group 136.00m 100.00m 1 100.00m dmstats 3.10.0-682.el7.x86_64 lvm2-2.02.171-5.el7 BUILT: Wed Jun 14 17:33:32 CEST 2017 lvm2-libs-2.02.171-5.el7 BUILT: Wed Jun 14 17:33:32 CEST 2017 lvm2-cluster-2.02.171-5.el7 BUILT: Wed Jun 14 17:33:32 CEST 2017 device-mapper-1.02.140-5.el7 BUILT: Wed Jun 14 17:33:32 CEST 2017 device-mapper-libs-1.02.140-5.el7 BUILT: Wed Jun 14 17:33:32 CEST 2017 device-mapper-event-1.02.140-5.el7 BUILT: Wed Jun 14 17:33:32 CEST 2017 device-mapper-event-libs-1.02.140-5.el7 BUILT: Wed Jun 14 17:33:32 CEST 2017 device-mapper-persistent-data-0.7.0-0.1.rc6.el7 BUILT: Mon Mar 27 17:15:46 CEST 2017
Bryn: Brassow wrote up a description in the "Doc text" field for the RHEL 7.4 release notes and suggested that I run it by you for review. Any improvements or corrections here? Steven
Steven, Looks good to me as a short summary - I know we are very limited in release notes space so I think this is fine. We've updated the manual pages with all the details for users who are interested. Bryn.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2222