The purpose of reserved memory is to pre-allocate all the RAM process will ever need during
locked 'activation' section when device can be suspended (included i.e. swap device) 
and lvm2 trys to keep this size reasonably 'minimal'. Current default limit should
be mostly OK - unless you would be managing very massive metadata for concurrenly active LVs.

So while normally we ask for 8M - you try to go with 128M - assuming there is some good reason 
for this. So that all put here - this code has not changed for quite a while so it's
interesting, why it has now causing any sort of problems.

I'd estimate this is likely a symptom of memory trashing in some other part of code.

So please try to attach 'lvmdump -a' output of the failing system - so we may try to reproduce internally.

Also add 'ulimit -a'.

And if you can - you may try to run  'dmeventd' in debug mode - just kill the
existing one (without any monitored device) - and restart new dmeventd from cmdline:

'dmeventd -flddd  &>/tmp/log

this should give is some idea about what is going inside dmeventd (which internally run lvm command to resize your thin-pool)

eventually running this in valgrind may also enhance the picture.

Is there a way we can send you the lvmdump privately? We are a Red Hat partner if that helps facilitate transferring the information privately.

As for your other requests, see below for ulimits and see attached for dmeventd debug output (dmeventd-debug.log).

~]# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 257396
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 257396
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Created attachment 1716124 [details]
dmeventd debug output

Likely such private files can be submitted via opened support case at access.redhat.com.

The trace of dmeventd suggests this is crashing very early at start of 1st lvm command.

Assuming this is a common x86_64 HW and the crash does not happen with 8M ?

And it starts crashing with 128M (so 64M is still ok) ?

If you disable monitoring=0 in lvm.conf - lvm2 commands have no problem with memory locking on this machine 
(i.e. lvchange --refresh vg) with  128M reserved memory.

If you can post your GPG key or point me where I can find it, then I can send you an encrypted tar with the other information you are looking for. We will test the other memory and monitoring settings and get back to you.

Regarding your tests, here are my results:
* It does not matter what I set reserved_memory to, dmeventd crashes on LV creation (note that it does not crash on removal).
* Setting monitoring=0 stops dmeventd from crashing.
* Running `lvchange --refresh vg` results in many segfaults in the logs.

If you can provide your GPG key, I can provide the requested dump.

In there interest of getting this issue resolved sooner than later, I am uploading the lvmdump to this bug report, rather than waiting for a GPG key to encrypt it.

Created attachment 1718549 [details]
lvmdump -a

The problem is likely not GPG - but we still haven't found reasoning why dmeventd could be crashing there.

When you set  'use_mlockall = 1'  in lvm.conf -  does it change anything for your crashing problem ?

Also change of your 'reserved_stack = 64' (instead of current 512) ?

lvmdump has no info about boot & memory mapping - so this some kind of unusual hw ?

The code where it is crashing on your machine is quite 'time-proved' - so we are still unclear how
is it possible it can be crashing there.

Basically it tries to touch every allocated memory page the process has got to confirm the memory
belongs to the process and physically exists.

So if it is getting an exception - it looks like the kernel VM core has no mapping for the memory ??

But since this is bare metal - we are puzzle how this is possible.

Setting reserved_stack = 64 fixes the issue; we no longer get segfaults. Even with use_mlockall = 0 this is true. Hopefully that helps you determine the possible cause.

We are not using any unusual hardware configuration as far as I am aware. I am not onsite so I cannot get you exact hardware, but this is the processor we are using:
* Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz

I can get you more specific hardware specs later today if you think it will help.

The motherboard is nothing special, it is a Supermicro X9DBL-3F with 2 sockets and the CPUs noted above.

Ok this has helped - now it's more clear - as the problem is not 'reserved_memory'  but 'reserved_stack'

dmeventd sets internally 'per thread' stack size to  300KiB - and withing one such thread lvm2 command is executed.

So ATM keep the reserved size size ideally at default 64KiB.

We will try to think what we can do better here.

But dmeventd as such doesn't see  lvm.conf - it's only one of it's threads that is executed later which
is influenced by this config.

So if users sets there some high value - threads has been already 'limited' from dmeventd side. 

So we will think what is the best approach here.

ATM users shouldn't touch reserved_stack - as the code has been usually changed to not put large
data on stack - but into memory pools so the value 64K should be OK.

FWIW, this doesn't require thin snaps. Old type snapshots also hit this.

[root@hayes-03 ~]# lvcreate --yes -L 300M snapper -n origin
  Logical volume "origin" created.
[root@hayes-03 ~]# lvcreate --yes -s /dev/snapper/origin -n altered_memory -L 100M
  WARNING: Monitoring snapper/altered_memory failed.
  WARNING: Monitoring snapper/origin failed.
  Logical volume "altered_memory" created.

Resolved by this commit: https://www.redhat.com/archives/lvm-devel/2020-October/msg00168.html
so threaded dmeventd  ignores  reserved_stack setting  (we already mostly cleaned lvm code base
in past to avoid pushing large things on stack - so 64K should work).

Fix verified in the latest 7.9.z rpms.

3.10.0-1160.el7.x86_64

lvm2-2.02.187-6.el7_9.2    BUILT: Mon Oct 26 15:21:13 CDT 2020
lvm2-libs-2.02.187-6.el7_9.2    BUILT: Mon Oct 26 15:21:13 CDT 2020
lvm2-cluster-2.02.187-6.el7_9.2    BUILT: Mon Oct 26 15:21:13 CDT 2020


[root@mckinley-01 ~]# grep reserved_ /etc/lvm/lvm.conf
        # Configuration option activation/reserved_stack.
        reserved_stack = 512
        # Configuration option activation/reserved_memory.
        reserved_memory = 131072

[root@mckinley-01 ~]# lvcreate -ay --thinpool my_pool -L 10G vg
  Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data.
  Logical volume "my_pool" created.
[root@mckinley-01 ~]# lvcreate -ay --virtualsize 20G -T vg/my_pool -n my_origin
  WARNING: Sum of all thin volume sizes (20.00 GiB) exceeds the size of thin pool vg/my_pool (10.00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "my_origin" created.
[root@mckinley-01 ~]# lvcreate -ay -y -k n -s /dev/vg/my_origin -n my_snap
  WARNING: Sum of all thin volume sizes (40.00 GiB) exceeds the size of thin pool vg/my_pool (10.00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "my_snap" created.

Oct 29 13:44:59 mckinley-01 dmeventd[21105]: No longer monitoring thin pool vg-my_pool-tpool.
Oct 29 13:44:59 mckinley-01 lvm[21105]: Monitoring thin pool vg-my_pool-tpool.

[root@mckinley-01 ~]# ps -ef | grep dmeventd
root     21105     1  0 13:22 ?        00:00:00 /usr/sbin/dmeventd -f

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5461