RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1855038 - writecache large pool activation attempt causes vmalloc: allocation failure (device-mapper: reload ioctl on failed: Cannot allocate memory)
Summary: writecache large pool activation attempt causes vmalloc: allocation failure (...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.3
Hardware: x86_64
OS: Linux
low
high
Target Milestone: rc
: 8.0
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-08 17:58 UTC by Corey Marthaler
Modified: 2022-05-23 16:03 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-08 07:27:00 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CLUSTERQE-5752 0 None None None 2022-05-23 16:03:04 UTC

Description Corey Marthaler 2020-07-08 17:58:33 UTC 
Description of problem:
In this scenario, the pool is much larger than the writecache origin volume.

[root@hayes-01 ~]# lvcreate --wipesignatures y  -L 2G -n large_writecache_removal writecache_sanity /dev/sdb1
  Logical volume "large_writecache_removal" created.

[root@hayes-01 ~]# lvcreate  -l 100%PVS -n pool writecache_sanity /dev/sdc1
  Logical volume "pool" created.

[root@hayes-01 ~]# lvs -a -o +devices
  LV                       VG                Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices     
  large_writecache_removal writecache_sanity -wi-a-----  2.00g                                                     /dev/sdb1(0)
  pool                     writecache_sanity -wi-a----- <1.82t                                                     /dev/sdc1(0)

[root@hayes-01 ~]# lvchange -an writecache_sanity

[root@hayes-01 ~]# lvconvert --yes --type writecache --cachevol writecache_sanity/pool writecache_sanity/large_writecache_removal
  Using writecache block size 512 for unknown file system block size, logical block size 512, physical block size 512.
  Logical volume writecache_sanity/large_writecache_removal now has writecache.

[root@hayes-01 ~]# lvs -a -o +devices
  LV                                VG                Attr       LSize  Pool        Origin                            Data%  Meta%  Move Log Cpy%Sync Convert Devices                           
  large_writecache_removal          writecache_sanity Cwi---C---  2.00g [pool_cvol] [large_writecache_removal_wcorig]                                         large_writecache_removal_wcorig(0)
  [large_writecache_removal_wcorig] writecache_sanity owi---C---  2.00g                                                                                       /dev/sdb1(0)                      
  [pool_cvol]                       writecache_sanity Cwi---C--- <1.82t                                                                                       /dev/sdc1(0)
                      
[root@hayes-01 ~]# lvchange -ay writecache_sanity/large_writecache_removal
  device-mapper: reload ioctl on  (253:2) failed: Cannot allocate memory

[609161.467341] lvchange: vmalloc: allocation failure: 272705448096 bytes, mode:0x6000c0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0-1
[609161.481427] CPU: 11 PID: 504505 Comm: lvchange Kdump: loaded Tainted: G                 --------- -t - 4.18.0-215.el8.x86_64 #1
[609161.494334] Hardware name: Dell Inc. PowerEdge R830/0VVT0H, BIOS 1.7.1 01/29/2018
[609161.502780] Call Trace:
[609161.505613]  dump_stack+0x5c/0x80
[609161.509407]  warn_alloc.cold.118+0x7b/0x10d
[609161.514171]  __vmalloc_node_range+0x192/0x230
[609161.519128]  __vmalloc_node+0x36/0x60
[609161.523311]  ? writecache_alloc_entries.isra.39+0x30/0xa0 [dm_writecache]
[609161.530982]  writecache_alloc_entries.isra.39+0x30/0xa0 [dm_writecache]
[609161.538459]  init_memory+0x76/0x230 [dm_writecache]
[609161.543998]  writecache_ctr+0xc03/0x1040 [dm_writecache]
[609161.550027]  dm_table_add_target+0x17d/0x360 [dm_mod]
[609161.555764]  table_load+0x122/0x2e0 [dm_mod]
[609161.560616]  ? dev_status+0x40/0x40 [dm_mod]
[609161.565475]  ctl_ioctl+0x1af/0x3f0 [dm_mod]
[609161.570242]  ? selinux_file_ioctl+0x80/0x200
[609161.575094]  dm_ctl_ioctl+0xa/0x10 [dm_mod]
[609161.579858]  do_vfs_ioctl+0xa4/0x630
[609161.583943]  ksys_ioctl+0x60/0x90
[609161.587736]  __x64_sys_ioctl+0x16/0x20
[609161.592014]  do_syscall_64+0x5b/0x1a0
[609161.596195]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[609161.601926] RIP: 0033:0x7f54a3fb288b
[609161.606011] Code: 0f 1e fa 48 8b 05 fd 95 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c38
[609161.627062] RSP: 002b:00007ffe5bc253c8 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
[609161.635604] RAX: ffffffffffffffda RBX: 000055f989132f10 RCX: 00007f54a3fb288b
[609161.643661] RDX: 000055f98a96ba10 RSI: 00000000c138fd09 RDI: 000000000000000d
[609161.651718] RBP: 000055f9891e79f0 R08: 0000000000000000 R09: 00007ffe5bc25230
[609161.659775] R10: 000055f98925728a R11: 0000000000000202 R12: 0000000000000000
[609161.667833] R13: 000055f98a96ba40 R14: 000055f98a96ba10 R15: 000055f98a96fb20
[609161.675897] Mem-Info:
[609161.678535] active_anon:52872 inactive_anon:35324 isolated_anon:0
[609161.678535]  active_file:31563 inactive_file:89787 isolated_file:0
[609161.678535]  unevictable:5844 dirty:1 writeback:0 unstable:0
[609161.678535]  slab_reclaimable:86425 slab_unreclaimable:120435
[609161.678535]  mapped:56875 shmem:37478 pagetables:2588 bounce:0
[609161.678535]  free:50421400 free_pcp:2569 free_cma:0
[609161.717431] Node 0 active_anon:133740kB inactive_anon:89104kB active_file:76784kB inactive_file:296504kB unevictable:2876kB isolated(anon):0kB isolated(file):0kB mapped:132o
[609161.749250] Node 1 active_anon:77748kB inactive_anon:52192kB active_file:49468kB inactive_file:62644kB unevictable:20500kB isolated(anon):0kB isolated(file):0kB mapped:9512o
[609161.780876] Node 0 DMA free:15344kB min:4kB low:16kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:B
[609161.809596] lowmem_reserve[]: 0 1453 123860 123860 123860
[609161.815722] Node 0 DMA32 free:1511492kB min:528kB low:2016kB high:3504kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB
[609161.846091] lowmem_reserve[]: 0 0 122407 122407 122407
[609161.851924] Node 0 Normal free:128401128kB min:44468kB low:169812kB high:295156kB active_anon:133740kB inactive_anon:89104kB active_file:76784kB inactive_file:296504kB unevB
[609161.886462] lowmem_reserve[]: 0 0 0 0 0
[609161.890841] Node 1 Normal free:71757636kB min:45108kB low:172252kB high:299396kB active_anon:77748kB inactive_anon:52192kB active_file:49468kB inactive_file:62644kB unevictB
[609161.925376] lowmem_reserve[]: 0 0 0 0 0
[609161.929748] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 15344kB
[609161.944798] Node 0 DMA32: 7*4kB (UM) 5*8kB (UM) 4*16kB (M) 6*32kB (UM) 4*64kB (UM) 4*128kB (UM) 4*256kB (UM) 6*512kB (UM) 5*1024kB (UM) 3*2048kB (M) 365*4096kB (UM) = 15114B
[609161.962465] Node 0 Normal: 1101*4kB (UME) 2646*8kB (UME) 2289*16kB (UME) 1438*32kB (UME) 557*64kB (UME) 248*128kB (UME) 113*256kB (UME) 52*512kB (UM) 38*1024kB (UME) 40*204B
[609161.983721] Node 1 Normal: 85*4kB (UME) 10*8kB (M) 56*16kB (UM) 139*32kB (ME) 105*64kB (ME) 73*128kB (M) 18*256kB (UME) 48*512kB (M) 40*1024kB (UME) 5*2048kB (UME) 17494*40B
[609162.003038] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[609162.012846] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[609162.022364] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[609162.032173] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[609162.041682] 161452 total pagecache pages
[609162.046159] 0 pages in swap cache
[609162.049956] Swap cache stats: add 0, delete 0, find 0/0
[609162.055887] Free swap  = 4194300kB
[609162.059781] Total swap = 4194300kB
[609162.063668] 67050556 pages RAM
[609162.067174] 0 pages HighMem/MovableOnly
[609162.071551] 3542585 pages reserved
[609162.075446] 0 pages hwpoisoned
[609163.577103] device-mapper: table: 253:2: writecache: Unable to initialize device
[609163.585465] device-mapper: ioctl: error adding target to table



Version-Release number of selected component (if applicable):
kernel-4.18.0-215.el8    BUILT: Tue Jun 16 14:14:53 CDT 2020
lvm2-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
lvm2-libs-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
lvm2-dbusd-2.03.09-3.el8    BUILT: Mon Jun 29 13:53:38 CDT 2020
lvm2-lockd-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
boom-boot-1.2-1.el8    BUILT: Sun Jun  7 07:20:03 CDT 2020
device-mapper-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-libs-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-event-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-event-libs-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020


How reproducible:
everytime
Comment 1 Corey Marthaler 2020-07-08 19:15:54 UTC 
FWIW, this happens at lvconvert time if attempted on an active origin (but inactive pool).

# Deactivate fast pool volume only before conversion to write cache

# Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachevol writecache_sanity/pool writecache_sanity/large_writecache_removal
  device-mapper: reload ioctl on  (253:0) failed: Cannot allocate memory
  Failed to suspend logical volume writecache_sanity/large_writecache_removal.
Comment 2 David Teigland 2020-07-09 16:38:23 UTC 
The writecache size is constrained by the amount of RAM on the system (Mikulas explains that in SSD mode 16 bytes of RAM are required for each cache block).  Even though RAM size is not a realistic limit (no one would use all their RAM for writecache), lvm could provide a more user-friendly error path by imposing a <RAM limit.  That kind of check wouldn't always prevent the error path above, since the max is never available.  Also, you might have enough RAM to create the writecache, but at some future time you may not have enough RAM to activate it.  So, I could add a patch to catch some obvious cases, with the caveat that it won't eliminate the errors above.
Comment 6 David Teigland 2021-12-15 16:10:29 UTC 
I'd like to add some code that (1) estimates the memory requirement for the writecache, and (2) estimates available memory on the machine, and then adds a confirmation step to the command if the 2 > 1.
Comment 7 RHEL Program Management 2022-01-08 07:27:00 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Note You need to log in before you can comment on or make changes to this bug.