Description of problem: While attempting to remove a large writecache LV (1TB) from a VG, the system crashed due to Out-Of-Memory. My system actually has 32GiB RAM, and I was playing with a 1TB size writecache Version-Release number of selected component (if applicable): RHEL 8.6 How reproducible: Always Steps to Reproduce: - Add a large device as a writecache to a VG: #lvconvert --type writecache --cachevol fast vg/main - Attempt to remove it from the VG with lvconvert --uncache (or split cache) Actual results: System crashes and enters in an unrecoverable state even if the volume isn't mounted automatically. During boot, lvm spends a long time trying to scan the device with the large cache, and the system fails to boot because it starts to throw OOM errors before finishing to boot. Expected results: - LVM prevents a large device to be added as a writecache if the system has not enough memory to deal with it. - System shouldn't crash while removing a cache from the VG. If a system has been using a cache device, but for some reason the total RAM has to be reduced, this shouldn't cause a crash while removing the cache device from the volume group. Additional info: I managed to boot in recovery mode, but any attempt to remove the writecache device from the VG failed. The system was unable to remove it and crashed. I manage to manually remove the writecache device from the VG by hacking the current VG configuration file, removing the writecache from it, and restoring the configuration without the writecache device by using vgrestore.
I think this is a duplicate of bug 2059644 where the fix was to print a warning when writecache requires >50% of system memory, and requires a confirmation if it requires >90% of memory. It doesn't currently prevent creating a large writecache, but we could extend the solution to actually fail to create at some memory percentage.
(In reply to David Teigland from comment #1) > I think this is a duplicate of bug 2059644 where the fix was to print a > warning when writecache requires >50% of system memory, and requires a > confirmation if it requires >90% of memory. It doesn't currently prevent > creating a large writecache, but we could extend the solution to actually > fail to create at some memory percentage. Hi David. Yeah, that seems a reasonable solution to avoid people creating caches that will likely crash their systems, but I don't think this will do anything if somebody reduces their system's memory after creating the cache. But anyway, this is just an idea based on the fact my biggest problem when I hit this, was to actually recover the system, giving the fact I couldn't boot it because of the big cache, and was unable to remove it too after I realized my mistake. Cheers
> likely crash their systems, but I don't think this will do anything if > somebody reduces their system's memory after creating the cache. Yes, we could apply similar checking in the activation path and fail to activate an LV if we think the writecache would use too much memory. Or dm-writecache could do some similar checking in the kernel. Mikulas, do you think either of those options makes sense? > But anyway, this is just an idea based on the fact my biggest problem when I > hit this, was to actually recover the system, giving the fact I couldn't boot it > because of the big cache, and was unable to remove it too after I realized my > mistake. If the system is autoactivating the problematic LV then it's difficult to intervene and fix the problem, so I do think we should have a better way to handle this. Once you've started the system successfully and the problematic writecache is inactive, then lvconvert --splitcache --force LV will forcibly detach the writecache without attempting to activate and write back the data, with potential data loss.