RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1916919 - lvcreate "buffer overflow detected" during 1000+ virt snap creations
Summary: lvcreate "buffer overflow detected" during 1000+ virt snap creations
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.4
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 8.0
Assignee: Zdenek Kabelac
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On: 1840384
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-15 20:15 UTC by Corey Marthaler
Modified: 2023-09-15 01:31 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1840384
Environment:
Last Closed: 2022-07-15 07:25:33 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Comment 1 Corey Marthaler 2021-01-15 20:18:48 UTC
Moving the issue described by me in comment #15 of bug 1730503 here.

kernel-4.18.0-271.el8    BUILT: Fri Jan  8 03:32:43 CST 2021
lvm2-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
lvm2-libs-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020


# With ulimit unlimited, I shouldn't run into a max, correct?

[root@hayes-03 ~]# ulimit
unlimited

[root@hayes-03 ~]# ulimit -n 2048 && lvcreate --yes  --config devices/scan_lvs=1 -k n -s /dev/snapper_thinp/origin -n many_1006
  WARNING: Sum of all thin volume sizes (<1.98 TiB) exceeds the size of thin pool snapper_thinp/POOL (2.00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
*** buffer overflow detected ***: lvcreate terminated
Aborted (core dumped)

[root@hayes-03 ~]# ulimit -n 2048 && lvcreate --yes  --config devices/scan_lvs=1 -k n -s /dev/snapper_thinp/origin -n many_1007
  WARNING: Sum of all thin volume sizes (<1.98 TiB) exceeds the size of thin pool snapper_thinp/POOL (2.00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
*** buffer overflow detected ***: lvcreate terminated
Aborted (core dumped)

[root@hayes-03 ~]# lsof | wc -l
10210

# Setting a much higher ulimit
[root@hayes-03 ~]# ulimit -n 10240 && lvcreate --yes  --config devices/scan_lvs=1 -k n -s /dev/snapper_thinp/origin -n many_1008
  WARNING: Sum of all thin volume sizes (1.98 TiB) exceeds the size of thin pool snapper_thinp/POOL (2.00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
*** buffer overflow detected ***: lvcreate terminated
Aborted (core dumped)

# Setting a much higher ulimit
[root@hayes-03 ~]# ulimit -n 11264 && lvcreate --yes  --config devices/scan_lvs=1 -k n -s /dev/snapper_thinp/origin -n many_1009
  WARNING: Sum of all thin volume sizes (1.98 TiB) exceeds the size of thin pool snapper_thinp/POOL (2.00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
*** buffer overflow detected ***: lvcreate terminated
Aborted (core dumped)


Core was generated by `lvcreate --yes --config devices/scan_lvs=1 -k n -s /dev/snapper_thinp/origin -n'.
Program terminated with signal SIGABRT, Aborted.

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fab87248db5 in __GI_abort () at abort.c:79
#2  0x00007fab872a14e7 in __libc_message (action=<optimized out>, fmt=fmt@entry=0x7fab873b07fc "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fab87334355 in __GI___fortify_fail_abort (need_backtrace=need_backtrace@entry=true, msg=msg@entry=0x7fab873b0779 "buffer overflow detected") at fortify_fail.c:28
#4  0x00007fab87334387 in __GI___fortify_fail (msg=msg@entry=0x7fab873b0779 "buffer overflow detected") at fortify_fail.c:44
#5  0x00007fab87332346 in __GI___chk_fail () at chk_fail.c:28
#6  0x00007fab8733427b in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
#7  0x00007fab88d26638 in _daemon_write (msg=msg@entry=0x7ffd6b903e50, fifos=<optimized out>, fifos=<optimized out>) at libdevmapper-event.c:329
#8  0x00007fab88d26ed8 in daemon_talk (fifos=fifos@entry=0x7ffd6b903d10, msg=msg@entry=0x7ffd6b903e50, cmd=cmd@entry=8, dso_name=dso_name@entry=0x0, dev_name=dev_name@entry=0x0, evmask=evmask@entry=0, timeout=0)
    at libdevmapper-event.c:381
#9  0x00007fab88d27823 in _do_event (cmd=cmd@entry=4, dmeventd_path=<optimized out>, msg=msg@entry=0x7ffd6b903e50, dso_name=0x564f926d81f0 "libdevmapper-event-lvm2thin.so", 
    dev_name=0x564f92754c80 "LVM-9ytrNmdwX3ERdBlpP2BOJ7wQ2Zb1YmM92WzHJLhyMJmE2bWrW8Uzo1ZXcNO1hmMV-tpool", evmask=DM_EVENT_ERROR_MASK, timeout=0) at libdevmapper-event.c:630
#10 0x00007fab88d27dd4 in dm_event_get_registered_device (dmevh=dmevh@entry=0x564f92162ea0, next=next@entry=0) at libdevmapper-event.c:768
#11 0x0000564f8f90a771 in target_registered_with_dmeventd (cmd=0x564f91516a10, dso=<optimized out>, lv=<optimized out>, pending=0x7ffd6b903f58, monitored=0x7ffd6b903f5c) at activate/activate.c:1662
#12 0x0000564f8f90af0d in monitor_dev_for_events (cmd=<optimized out>, lv=0x564f92a45450, laopts=<optimized out>, monitor=<optimized out>) at activate/activate.c:1882
#13 0x0000564f8f90ac88 in monitor_dev_for_events (cmd=cmd@entry=0x564f91516a10, lv=lv@entry=0x564f92a455b8, laopts=<optimized out>, laopts@entry=0x7ffd6b904190, monitor=monitor@entry=1) at activate/activate.c:1848
#14 0x0000564f8f90b7c7 in _lv_resume (cmd=cmd@entry=0x564f91516a10, laopts=laopts@entry=0x7ffd6b904190, error_if_not_active=error_if_not_active@entry=0, lv=lv@entry=0x564f92a455b8, lvid_s=<optimized out>)
    at activate/activate.c:2333
#15 0x0000564f8f90ca31 in _lv_resume (lvid_s=0x0, lv=0x564f92a455b8, error_if_not_active=0, laopts=0x7ffd6b904190, cmd=0x564f91516a10) at activate/activate.c:2361
#16 lv_resume_if_active (cmd=cmd@entry=0x564f91516a10, lvid_s=lvid_s@entry=0x0, origin_only=origin_only@entry=1, exclusive=exclusive@entry=0, revert=revert@entry=0, lv=0x564f92a455b8) at activate/activate.c:2361
#17 0x0000564f8f90d91c in resume_lv_origin (cmd=cmd@entry=0x564f91516a10, lv=lv@entry=0x564f9268c1e8) at activate/activate.c:2932
#18 0x0000564f8f965ec0 in _lv_create_an_lv (vg=vg@entry=0x564f9268a720, lp=lp@entry=0x7ffd6b904700, new_lv_name=<optimized out>) at metadata/lv_manip.c:8504
#19 0x0000564f8f96670c in lv_create_single (vg=vg@entry=0x564f9268a720, lp=lp@entry=0x7ffd6b904700) at metadata/lv_manip.c:8798
#20 0x0000564f8f8cec08 in _lvcreate_single (cmd=cmd@entry=0x564f91516a10, vg_name=vg_name@entry=0x564f9265c410 "snapper_thinp", vg=vg@entry=0x564f9268a720, handle=handle@entry=0x564f91528288) at lvcreate.c:1752
#21 0x0000564f8f8f3d4a in _process_vgnameid_list (process_single_vg=0x564f8f8cde40 <_lvcreate_single>, handle=0x564f91528288, arg_tags=0x7ffd6b904570, arg_vgnames=0x7ffd6b904580, vgnameids_to_process=0x7ffd6b9045a0, 
    read_flags=1048576, cmd=0x564f91516a10) at toollib.c:1933
#22 process_each_vg (cmd=cmd@entry=0x564f91516a10, argc=argc@entry=0, argv=argv@entry=0x0, one_vgname=<optimized out>, use_vgnames=use_vgnames@entry=0x0, read_flags=read_flags@entry=1048576, 
    include_internal=<optimized out>, handle=0x564f91528288, process_single_vg=<optimized out>) at toollib.c:2238
#23 0x0000564f8f8cfd7e in lvcreate (cmd=0x564f91516a10, argc=<optimized out>, argv=<optimized out>) at lvcreate.c:1797
#24 0x0000564f8f8d6e3d in lvm_run_command (cmd=cmd@entry=0x564f91516a10, argc=<optimized out>, argc@entry=10, argv=<optimized out>, argv@entry=0x7ffd6b904bf8) at lvmcmdline.c:3135
#25 0x0000564f8f8d8153 in lvm2_main (argc=10, argv=0x7ffd6b904bf8) at lvmcmdline.c:3667
#26 0x00007fab8724a493 in __libc_start_main (main=0x564f8f8b1ae0 <main>, argc=10, argv=0x7ffd6b904bf8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd6b904be8)
    at ../csu/libc-start.c:314
#27 0x0000564f8f8b1bae in _start () at lvm.c:22

Comment 2 Zdenek Kabelac 2021-01-19 16:00:49 UTC
Corey - can we please get the trace from a
'valgrind --leak-check=full --track-origins=yes --show-reachable=yes'
execution in your case attached to the BZ -

I'm trying to play with some reproducer - so far I've identified one memory leak fixed by this upstream commit:

https://www.redhat.com/archives/lvm-devel/2021-January/msg00015.html

Although there are several thing going wrong around cases like this:

1. lvm2 likely should handle better max amount of allowed descriptors - i.e. not using more then some X% of the given limit -
since we report to many errors - which are sometimes information messages - but sometimes are serious error causing errors with locking.

2. messaging needs cleaning - since 'log_error()' is supposed to be only printed for commands finishing with some error.

3. dmeventd with 1 monitored object consumes to much CPU - needs some thinking about reopening lvm plugin more efficently.

Comment 5 RHEL Program Management 2022-07-15 07:25:33 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 6 Red Hat Bugzilla 2023-09-15 01:31:54 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days


Note You need to log in before you can comment on or make changes to this bug.