Bug 1442992

Summary: When thin pool usage exceeds threshold allow the action taken to be configurable through an external script
Product: Red Hat Enterprise Linux 7 Reporter: Zdenek Kabelac <zkabelac>
Component: lvm2Assignee: Zdenek Kabelac <zkabelac>
lvm2 sub component: dmeventd QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact: Steven J. Levine <slevine>
Severity: unspecified    
Priority: unspecified CC: agk, cluster-qe, cmarthal, heinzm, jbrassow, msnitzer, pasik, prajnoha, prockai, rbednar, slevine, thornber, zkabelac
Version: 7.4Keywords: Documentation
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Release Note
Doc Text:
LVM extends the size of a thin pool logical volume when a thin pool fills over 50 percent When a thin pool logical volume fills by more than 50 percent, by default the `dmeventd thin` plugin now calls the `dmeventd` "thin_command" command with every 5 percent increase. This resizes the thin pool when it has been filled above the configured `thin_pool_autoextend_threshold` in the `activation` section of the configuration file. A user may override this default by configuring an external command and specifying this command as the value of `thin_command` in the `dmeventd` section of the `lvm.conf` file. For information on the `thin` plugin and on configuring external commands to maintain a thin pool, see the `dmeventd(8)` man page. In previous releases, when a thin pool resize failed, the `dmeventd` plugin would try to unmount unconditionally all thin volumes associated with the thin pool when a compile-time defined threshold of more than 95 percent was reached. The `dmeventd` plugin, by default, no longer unmounts any volumes. Reproducing the previous logic requires configuring an external script.
Story Points: ---
Clone Of: 1410768 Environment:
Last Closed: 2017-08-01 21:52:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1410768    
Bug Blocks: 1370490    

Description Zdenek Kabelac 2017-04-18 09:00:38 UTC
+++ This bug was initially created as a clone of Bug #1410768 +++

When 'dmeventd' plugin calls 'lvconvert'  it provides implicit and unconfigurable reaction when called command fails.
i.e. in case of thin-pool failing resize, it will try to umount
unconditionally all thin volumes associated with such thin pool
when compile-time defined threshold (>95%) is reached.

This reaction is not likely wanted for newer kernels and there are users wishing to operate full thin-pool without umount disruption.

As 'dmeventd' has no configuration, we need to provide some policy configurable knobs for lvm2 command (sacrificing some potential error handling paths).

We decided to allow the whole command to be configurable in lvm.conf as follows:

dmeventd {
        thin_command = "lvm lvextend --use-policies"
}

The plugin runs the command with each 5% increment when thin-pool data volume or metadata volume gets above 50% full.

A command that starts with the prefix 'lvm ' is a built-in lvm command.

A command that starts with '/' can be the full path to an external command or script of your own.

Comment 2 Zdenek Kabelac 2017-04-18 09:03:01 UTC
As this BZ present important change/improvement in thin-pool behavior worth a doc text -  cloned into 7.4

Comment 3 Jonathan Earl Brassow 2017-05-10 14:47:25 UTC
Need clear description for slevine (in doc_text box)

Comment 8 Alasdair Kergon 2017-05-30 21:56:54 UTC
This went into the code base back in January via upstream bug 1410768 to offer an alternative for those people for whom the automatic umount was a problem.

commit 85dab3963fbf8b3cc1be84f2d7c43257075ac2a3
Author: Zdenek Kabelac <zkabelac>
Date:   Fri Jan 20 22:54:20 2017 +0100

    dmeventd_thin: enable support for external command

https://www.redhat.com/archives/lvm-devel/2017-January/msg00036.html


commit 04a9cad499848cfc025d33ca28e9f657abc80ca3
Author: Zdenek Kabelac <zkabelac>
Date:   Wed Jan 18 09:54:32 2017 +0100

    config: new option dmeventd/thin_command
    
    This setting will allowing configuring which command gets executed
    when thin-pool fullness goes from 50%..100%

https://www.redhat.com/archives/lvm-devel/2017-January/msg00024.html

Comment 12 Zdenek Kabelac 2017-06-01 09:03:55 UTC
Here is the part of  lvm2  test suite script creating 'auto-umounting' behavior via execution of the external shell script:

---
#!/bin/sh

echo "Data: \$DMEVENTD_THIN_POOL_DATA"
echo "Metadata: \$DMEVENTD_THIN_POOL_METADATA"

$TESTDIR/lib/lvextend --use-policies \$1 || {
	umount "$mntdir"  || true
	umount "$mntusedir" || true
	return 0
}
test \$($TESTDIR/lib/lvs -o selected -S "data_percent>95||metadata_percent>95" --noheadings \$1) -eq 0 || {
	umount "$mntdir"  || true
	umount "$mntusedir" || true
	return 0
}
---

(part of  lvm2/test/shell/thin-autoumount-dmeventd.sh)
https://sourceware.org/git/?p=lvm2.git;a=blob;f=test/shell/thin-autoumount-dmeventd.sh;h=5e1c2be8abdb632ad8573b9094d1a0dc49fc6f51;hb=HEAD


So what the code does today (as documented in 'man dmeventd' thin plugin behavior) -  it no longer umounts anything  - as this appeared to be causing some other possibly very serious troubles like write to a completely different storage after umount.

To 'mimic' the old logic (if needed) - user has to use the similar script like the one pasted here.

You can see here how the  ENV vars are set and usable without the need to call 'lvs' inside the script for simple decisions (i.e. calling fstrim ??)


In general - if the script is NOT configured - lvm2 should just try to 'lvextend'.

If the script IS configured - it's all in hands of script - executions of commands within such script should not cause dmeventd deadlock - running in the special mode.

There is clearly a limitation of such script usage - if user would run the script from the same volume which is undergoing maintenance - possibly very bad things may happen.

Comment 14 Corey Marthaler 2017-06-26 21:38:25 UTC
Verified that if set, lvm will continue to call the external script defined in dmeventd { thin_command } as the thin_pool_autoextend_threshold is passed. 

3.10.0-685.el7.x86_64
lvm2-2.02.171-7.el7    BUILT: Thu Jun 22 08:35:15 CDT 2017
lvm2-libs-2.02.171-7.el7    BUILT: Thu Jun 22 08:35:15 CDT 2017
lvm2-cluster-2.02.171-7.el7    BUILT: Thu Jun 22 08:35:15 CDT 2017
device-mapper-1.02.140-7.el7    BUILT: Thu Jun 22 08:35:15 CDT 2017
device-mapper-libs-1.02.140-7.el7    BUILT: Thu Jun 22 08:35:15 CDT 2017
device-mapper-event-1.02.140-7.el7    BUILT: Thu Jun 22 08:35:15 CDT 2017
device-mapper-event-libs-1.02.140-7.el7    BUILT: Thu Jun 22 08:35:15 CDT 2017
device-mapper-persistent-data-0.7.0-0.1.rc6.el7    BUILT: Mon Mar 27 10:15:46 CDT 2017

Comment 15 Steven J. Levine 2017-07-17 21:57:07 UTC
Zdenek, Jonathan:

I tried to simplify/summarize this feature in the doc text for the release notes, but this seriously needs a technical review to be sure this conveys the defined behavior. The release notes don't really seem to be the place for documenting how to write an external script -- I don't know if that's a kBase article or an addition to the manual, but do you think we need to write that up?

Comment 16 Zdenek Kabelac 2017-07-17 22:28:46 UTC
There could be some more things added.


On default - lvm2 calls 'internal' 'lvresize' as plugin - which does resize according to configuration (threshold  values)


User may 'opt-out' and use his own plugin/script which is called at same points.
Advantage is - such plugin may do far more  - it may call i.e. 'fstrim' earlier,
it may try to drop unused thinLV (i.e. some old snapshots) to release some space in thin-pool and if that's still not enough it make call lvresize command.


The main change is then - that current 'default' internal command is not doing any umounting - if user still finds such thing as usable - he needs to write his own plugin emulating this logic  (we may possibly extend) dmeventd man page with some simple example if that's becomes wanted info.

Comment 18 errata-xmlrpc 2017-08-01 21:52:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2222