Bug 2170554

Summary: lvextend should check for and report "Not enough free memory for VDO target" like lvcreate does
Product: Red Hat Enterprise Linux 9 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Zdenek Kabelac <zkabelac>
lvm2 sub component: VDO QA Contact: cluster-qe <cluster-qe>
Status: CLOSED MIGRATED Docs Contact:
Severity: medium    
Priority: unspecified CC: agk, awalsh, heinzm, jbrassow, prajnoha, zkabelac
Version: 9.2Keywords: MigratedToJIRA
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-23 18:51:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2023-02-16 17:23:08 UTC 
Description of problem:
This is an actual bug report for the extension issue mentioned here: https://bugzilla.redhat.com/show_bug.cgi?id=2079048#c1

It appears that lvcreate checks for free memory before attempting a create that is too large. lvextend should do that as well rather than running into "vmalloc errors"


# Doesn't cause kdump issues
[root@virt-506 ~]# lvcreate --yes --type vdo -n vdo_lv  -L 10G vdo_sanity -V 4096TiB  
  Not enough free memory for VDO target. 6.83 GiB RAM is required, but only <3.47 GiB RAM is available.
  VDO configuration needs 3.00 MiB RAM for physical volume size 10.00 GiB, <6.40 GiB RAM for virtual volume size 4.00 PiB, 150.00 MiB RAM for block map cache size 128.00 MiB and 256.00 MiB RAM for index memory.


# Doesn't cause kdump issues
[root@virt-498 ~]# lvextend --yes -l1000000%FREE vdo_sanity/vdo_lv 
  Size of logical volume vdo_sanity/vdo_lv changed from 1.00 PiB (268438081 extents) to <5.53 PiB (1484316801 extents).
  VDO logical size is larger than limit 4096 TiB by 1681715106816 KiB.
  Failed to suspend logical volume vdo_sanity/vdo_lv.


# Does cause kdump issues
[root@virt-498 ~]# lvextend --yes -L+2.75P vdo_sanity/vdo_lv 
  Size of logical volume vdo_sanity/vdo_lv changed from 1.00 PiB (268438081 extents) to 3.75 PiB (1006635585 extents).
  device-mapper: reload ioctl on  (253:3) failed: Cannot allocate memory
  Failed to suspend logical volume vdo_sanity/vdo_lv.


Feb 16 18:14:09 virt-498 kernel: kvdo12:lvextend: preparing to modify device '253:3'
Feb 16 18:14:09 virt-498 kernel: kvdo12:lvextend: Preparing to resize logical to 1030794839296
Feb 16 18:14:10 virt-498 kernel: lvextend: vmalloc error: size 4766163840, exceeds total pages, mode:0x4dc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
Feb 16 18:14:10 virt-498 kernel: CPU: 1 PID: 6658 Comm: lvextend Kdump: loaded Tainted: G           O     --------- ---  5.14.0-252.el9.x86_64 #1
Feb 16 18:14:10 virt-498 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Feb 16 18:14:10 virt-498 kernel: Call Trace:
Feb 16 18:14:10 virt-498 kernel: <TASK>
Feb 16 18:14:10 virt-498 kernel: dump_stack_lvl+0x34/0x48
Feb 16 18:14:10 virt-498 kernel: warn_alloc+0x138/0x160
Feb 16 18:14:10 virt-498 kernel: ? schedule+0x5a/0xc0
Feb 16 18:14:10 virt-498 kernel: __vmalloc_node_range+0x1f0/0x220
Feb 16 18:14:10 virt-498 kernel: __vmalloc_node+0x4a/0x70
Feb 16 18:14:10 virt-498 kernel: ? uds_allocate_memory+0x265/0x2e0 [kvdo]
Feb 16 18:14:10 virt-498 kernel: uds_allocate_memory+0x265/0x2e0 [kvdo]
Feb 16 18:14:10 virt-498 kernel: make_segment+0x107/0x360 [kvdo]
Feb 16 18:14:10 virt-498 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Feb 16 18:14:10 virt-498 kernel: ? uds_allocate_memory+0xf9/0x2e0 [kvdo]
Feb 16 18:14:10 virt-498 kernel: ? uds_log_embedded_message+0x3f/0x60 [kvdo]
Feb 16 18:14:10 virt-498 kernel: vdo_make_forest+0xd8/0x120 [kvdo]
Feb 16 18:14:10 virt-498 kernel: vdo_prepare_to_grow_logical+0x43/0xa0 [kvdo]
Feb 16 18:14:10 virt-498 kernel: vdo_prepare_to_modify+0x69/0x130 [kvdo]
Feb 16 18:14:10 virt-498 kernel: vdo_ctr+0xe4/0x240 [kvdo]
Feb 16 18:14:10 virt-498 kernel: dm_table_add_target+0x16f/0x3a0 [dm_mod]
Feb 16 18:14:10 virt-498 kernel: table_load+0x12b/0x370 [dm_mod]
Feb 16 18:14:10 virt-498 kernel: ctl_ioctl+0x1a2/0x290 [dm_mod]
Feb 16 18:14:10 virt-498 kernel: dm_ctl_ioctl+0xa/0x20 [dm_mod]
Feb 16 18:14:10 virt-498 kernel: __x64_sys_ioctl+0x8a/0xc0
Feb 16 18:14:10 virt-498 kernel: do_syscall_64+0x5c/0x90
Feb 16 18:14:10 virt-498 kernel: ? do_syscall_64+0x69/0x90
Feb 16 18:14:10 virt-498 kernel: ? syscall_exit_to_user_mode+0x12/0x30
Feb 16 18:14:10 virt-498 kernel: ? do_syscall_64+0x69/0x90
Feb 16 18:14:10 virt-498 kernel: ? syscall_exit_work+0x11a/0x150
Feb 16 18:14:10 virt-498 kernel: ? syscall_exit_to_user_mode+0x12/0x30
Feb 16 18:14:10 virt-498 kernel: ? do_syscall_64+0x69/0x90
Feb 16 18:14:10 virt-498 kernel: ? syscall_exit_to_user_mode+0x12/0x30
Feb 16 18:14:10 virt-498 kernel: ? do_syscall_64+0x69/0x90
Feb 16 18:14:10 virt-498 kernel: ? sysvec_apic_timer_interrupt+0x3c/0x90
Feb 16 18:14:10 virt-498 kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
Feb 16 18:14:10 virt-498 kernel: RIP: 0033:0x7f54dc63ec6b
Feb 16 18:14:10 virt-498 kernel: Code: 73 01 c3 48 8b 0d b5 b1 1b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 85 b1 1b 00 f7 d8 64 89 01 48
Feb 16 18:14:10 virt-498 kernel: RSP: 002b:00007fff3be91a88 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
Feb 16 18:14:10 virt-498 kernel: RAX: ffffffffffffffda RBX: 0000563698fe8e30 RCX: 00007f54dc63ec6b
Feb 16 18:14:10 virt-498 kernel: RDX: 000056369b5dc800 RSI: 00000000c138fd09 RDI: 0000000000000008
Feb 16 18:14:10 virt-498 kernel: RBP: 00005636990c70d6 R08: 0000563699140960 R09: 00007fff3be918e0
Feb 16 18:14:10 virt-498 kernel: R10: 0000000000000007 R11: 0000000000000206 R12: 000056369b5bfe00
Feb 16 18:14:10 virt-498 kernel: R13: 000056369b5dc8b0 R14: 000056369913ee9d R15: 000056369b5dc800
Feb 16 18:14:10 virt-498 kernel: </TASK>
Feb 16 18:14:10 virt-498 kernel: Mem-Info:
Feb 16 18:14:10 virt-498 kernel: active_anon:214 inactive_anon:18424 isolated_anon:0#012 active_file:23355 inactive_file:37859 isolated_file:0#012 unevictable:29716 dirty:3 writeback:0#012 slab_reclaimable:8047 slab_unreclaimable:18734#012 mapped:12288 shmem:2579 pagetables:630 bounce:0#012 kernel_misc_reclaimable:0#012 free:333023 free_pcp:10960 free_cma:0
Feb 16 18:14:10 virt-498 kernel: Node 0 active_anon:856kB inactive_anon:73696kB active_file:93420kB inactive_file:151436kB unevictable:118864kB isolated(anon):0kB isolated(file):0kB mapped:49152kB dirty:12kB writeback:0kB shmem:10316kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 104448kB writeback_tmp:0kB kernel_stack:3120kB pagetables:2520kB all_unreclaimable? no
Feb 16 18:14:10 virt-498 kernel: Node 0 DMA free:14848kB boost:0kB min:256kB low:320kB high:384kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Feb 16 18:14:10 virt-498 kernel: lowmem_reserve[]: 0 3213 3897 3897 3897
Feb 16 18:14:10 virt-498 kernel: Node 0 DMA32 free:1300468kB boost:0kB min:55500kB low:69372kB high:83244kB reserved_highatomic:0KB active_anon:20kB inactive_anon:5356kB active_file:2224kB inactive_file:66648kB unevictable:43480kB writepending:12kB present:3653612kB managed:3325932kB mlocked:43480kB bounce:0kB free_pcp:39604kB local_pcp:22128kB free_cma:0kB
Feb 16 18:14:10 virt-498 kernel: lowmem_reserve[]: 0 0 684 684 684
Feb 16 18:14:10 virt-498 kernel: Node 0 Normal free:16776kB boost:2048kB min:13872kB low:16828kB high:19784kB reserved_highatomic:0KB active_anon:836kB inactive_anon:68340kB active_file:91196kB inactive_file:84788kB unevictable:75384kB writepending:0kB present:835584kB managed:707192kB mlocked:73848kB bounce:0kB free_pcp:4236kB local_pcp:420kB free_cma:0kB
Feb 16 18:14:10 virt-498 kernel: lowmem_reserve[]: 0 0 0 0 0
Feb 16 18:14:10 virt-498 kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14848kB
Feb 16 18:14:10 virt-498 kernel: Node 0 DMA32: 119*4kB (UM) 53*8kB (U) 17*16kB (UE) 7*32kB (UE) 32*64kB (UME) 21*128kB (UME) 10*256kB (ME) 3*512kB (UE) 2*1024kB (UM) 1*2048kB (E) 314*4096kB (M) = 1300468kB
Feb 16 18:14:10 virt-498 kernel: Node 0 Normal: 112*4kB (UME) 223*8kB (UME) 171*16kB (UME) 67*32kB (UME) 29*64kB (UM) 27*128kB (UME) 5*256kB (UME) 4*512kB (UME) 1*1024kB (M) 0*2048kB 0*4096kB = 16776kB
Feb 16 18:14:10 virt-498 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 16 18:14:10 virt-498 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 16 18:14:10 virt-498 kernel: 68079 total pagecache pages
Feb 16 18:14:10 virt-498 kernel: 0 pages in swap cache
Feb 16 18:14:10 virt-498 kernel: Swap cache stats: add 0, delete 0, find 0/0
Feb 16 18:14:10 virt-498 kernel: Free swap  = 839676kB
Feb 16 18:14:10 virt-498 kernel: Total swap = 839676kB
Feb 16 18:14:10 virt-498 kernel: 1126297 pages RAM
Feb 16 18:14:10 virt-498 kernel: 0 pages HighMem/MovableOnly
Feb 16 18:14:10 virt-498 kernel: 114176 pages reserved
Feb 16 18:14:10 virt-498 kernel: 0 pages cma reserved
Feb 16 18:14:10 virt-498 kernel: 0 pages hwpoisoned
Feb 16 18:14:10 virt-498 kernel: kvdo12:lvextend: Could not allocate 4766163840 bytes for new forest pages in 1053 msecs
Feb 16 18:14:10 virt-498 kernel: device-mapper: table: 253:3: vdo: Device vdo_prepare_to_grow_logical failed (-ENOMEM)
Feb 16 18:14:10 virt-498 kernel: device-mapper: ioctl: error adding target to table




Version-Release number of selected component (if applicable):
kernel-5.14.0-252.el9    BUILT: Wed Feb  1 03:30:10 PM CET 2023
lvm2-2.03.17-7.el9    BUILT: Thu Feb 16 03:24:54 PM CET 2023
lvm2-libs-2.03.17-7.el9    BUILT: Thu Feb 16 03:24:54 PM CET 2023


How reproducible:
Everytime
Comment 1 Zdenek Kabelac 2023-02-16 17:35:18 UTC 
Ahh - this code path seem to bypass the memory validation path - good catch.
Comment 3 RHEL Program Management 2023-09-23 18:49:09 UTC 
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
Comment 4 RHEL Program Management 2023-09-23 18:51:04 UTC 
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.