Bug 1881235
Summary: | dm-event.service segfaults with creating a thin snapshot | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | bugzilla | ||||||||
Component: | lvm2 | Assignee: | Zdenek Kabelac <zkabelac> | ||||||||
lvm2 sub component: | dmeventd | QA Contact: | cluster-qe <cluster-qe> | ||||||||
Status: | CLOSED ERRATA | Docs Contact: | |||||||||
Severity: | unspecified | ||||||||||
Priority: | urgent | CC: | agk, cmarthal, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, thornber, zkabelac | ||||||||
Version: | 7.8 | ||||||||||
Target Milestone: | rc | ||||||||||
Target Release: | --- | ||||||||||
Hardware: | Unspecified | ||||||||||
OS: | Unspecified | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | lvm2-2.02.187-6.el7_9.2 | Doc Type: | If docs needed, set a value | ||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | |||||||||||
: | 1887598 (view as bug list) | Environment: | |||||||||
Last Closed: | 2020-12-15 11:23:10 UTC | Type: | Bug | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Bug Depends On: | |||||||||||
Bug Blocks: | 1887598 | ||||||||||
Attachments: |
|
Description
bugzilla
2020-09-21 21:50:13 UTC
The purpose of reserved memory is to pre-allocate all the RAM process will ever need during locked 'activation' section when device can be suspended (included i.e. swap device) and lvm2 trys to keep this size reasonably 'minimal'. Current default limit should be mostly OK - unless you would be managing very massive metadata for concurrenly active LVs. So while normally we ask for 8M - you try to go with 128M - assuming there is some good reason for this. So that all put here - this code has not changed for quite a while so it's interesting, why it has now causing any sort of problems. I'd estimate this is likely a symptom of memory trashing in some other part of code. So please try to attach 'lvmdump -a' output of the failing system - so we may try to reproduce internally. Also add 'ulimit -a'. And if you can - you may try to run 'dmeventd' in debug mode - just kill the existing one (without any monitored device) - and restart new dmeventd from cmdline: 'dmeventd -flddd &>/tmp/log this should give is some idea about what is going inside dmeventd (which internally run lvm command to resize your thin-pool) eventually running this in valgrind may also enhance the picture. Is there a way we can send you the lvmdump privately? We are a Red Hat partner if that helps facilitate transferring the information privately. As for your other requests, see below for ulimits and see attached for dmeventd debug output (dmeventd-debug.log). ~]# ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 257396 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 257396 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Created attachment 1716124 [details]
dmeventd debug output
Likely such private files can be submitted via opened support case at access.redhat.com. The trace of dmeventd suggests this is crashing very early at start of 1st lvm command. Assuming this is a common x86_64 HW and the crash does not happen with 8M ? And it starts crashing with 128M (so 64M is still ok) ? If you disable monitoring=0 in lvm.conf - lvm2 commands have no problem with memory locking on this machine (i.e. lvchange --refresh vg) with 128M reserved memory. If you can post your GPG key or point me where I can find it, then I can send you an encrypted tar with the other information you are looking for. We will test the other memory and monitoring settings and get back to you. Regarding your tests, here are my results: * It does not matter what I set reserved_memory to, dmeventd crashes on LV creation (note that it does not crash on removal). * Setting monitoring=0 stops dmeventd from crashing. * Running `lvchange --refresh vg` results in many segfaults in the logs. If you can provide your GPG key, I can provide the requested dump. In there interest of getting this issue resolved sooner than later, I am uploading the lvmdump to this bug report, rather than waiting for a GPG key to encrypt it. Created attachment 1718549 [details]
lvmdump -a
The problem is likely not GPG - but we still haven't found reasoning why dmeventd could be crashing there. When you set 'use_mlockall = 1' in lvm.conf - does it change anything for your crashing problem ? Also change of your 'reserved_stack = 64' (instead of current 512) ? lvmdump has no info about boot & memory mapping - so this some kind of unusual hw ? The code where it is crashing on your machine is quite 'time-proved' - so we are still unclear how is it possible it can be crashing there. Basically it tries to touch every allocated memory page the process has got to confirm the memory belongs to the process and physically exists. So if it is getting an exception - it looks like the kernel VM core has no mapping for the memory ?? But since this is bare metal - we are puzzle how this is possible. Setting reserved_stack = 64 fixes the issue; we no longer get segfaults. Even with use_mlockall = 0 this is true. Hopefully that helps you determine the possible cause. We are not using any unusual hardware configuration as far as I am aware. I am not onsite so I cannot get you exact hardware, but this is the processor we are using: * Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz I can get you more specific hardware specs later today if you think it will help. The motherboard is nothing special, it is a Supermicro X9DBL-3F with 2 sockets and the CPUs noted above. Ok this has helped - now it's more clear - as the problem is not 'reserved_memory' but 'reserved_stack' dmeventd sets internally 'per thread' stack size to 300KiB - and withing one such thread lvm2 command is executed. So ATM keep the reserved size size ideally at default 64KiB. We will try to think what we can do better here. But dmeventd as such doesn't see lvm.conf - it's only one of it's threads that is executed later which is influenced by this config. So if users sets there some high value - threads has been already 'limited' from dmeventd side. So we will think what is the best approach here. ATM users shouldn't touch reserved_stack - as the code has been usually changed to not put large data on stack - but into memory pools so the value 64K should be OK. FWIW, this doesn't require thin snaps. Old type snapshots also hit this. [root@hayes-03 ~]# lvcreate --yes -L 300M snapper -n origin Logical volume "origin" created. [root@hayes-03 ~]# lvcreate --yes -s /dev/snapper/origin -n altered_memory -L 100M WARNING: Monitoring snapper/altered_memory failed. WARNING: Monitoring snapper/origin failed. Logical volume "altered_memory" created. Resolved by this commit: https://www.redhat.com/archives/lvm-devel/2020-October/msg00168.html so threaded dmeventd ignores reserved_stack setting (we already mostly cleaned lvm code base in past to avoid pushing large things on stack - so 64K should work). Fix verified in the latest 7.9.z rpms. 3.10.0-1160.el7.x86_64 lvm2-2.02.187-6.el7_9.2 BUILT: Mon Oct 26 15:21:13 CDT 2020 lvm2-libs-2.02.187-6.el7_9.2 BUILT: Mon Oct 26 15:21:13 CDT 2020 lvm2-cluster-2.02.187-6.el7_9.2 BUILT: Mon Oct 26 15:21:13 CDT 2020 [root@mckinley-01 ~]# grep reserved_ /etc/lvm/lvm.conf # Configuration option activation/reserved_stack. reserved_stack = 512 # Configuration option activation/reserved_memory. reserved_memory = 131072 [root@mckinley-01 ~]# lvcreate -ay --thinpool my_pool -L 10G vg Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data. Logical volume "my_pool" created. [root@mckinley-01 ~]# lvcreate -ay --virtualsize 20G -T vg/my_pool -n my_origin WARNING: Sum of all thin volume sizes (20.00 GiB) exceeds the size of thin pool vg/my_pool (10.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "my_origin" created. [root@mckinley-01 ~]# lvcreate -ay -y -k n -s /dev/vg/my_origin -n my_snap WARNING: Sum of all thin volume sizes (40.00 GiB) exceeds the size of thin pool vg/my_pool (10.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "my_snap" created. Oct 29 13:44:59 mckinley-01 dmeventd[21105]: No longer monitoring thin pool vg-my_pool-tpool. Oct 29 13:44:59 mckinley-01 lvm[21105]: Monitoring thin pool vg-my_pool-tpool. [root@mckinley-01 ~]# ps -ef | grep dmeventd root 21105 1 0 13:22 ? 00:00:00 /usr/sbin/dmeventd -f Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5461 |