Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2078452

Summary:	System crash attempting to remove a large writecache from a VG
Product:	Red Hat Enterprise Linux 8	Reporter:	Carlos Maiolino <cmaiolin>
Component:	lvm2	Assignee:	David Teigland <teigland>
lvm2 sub component:	Cache Logical Volumes	QA Contact:	cluster-qe <cluster-qe>
Status:	CLOSED MIGRATED	Docs Contact:
Severity:	medium
Priority:	medium	CC:	agk, heinzm, jbrassow, mpatocka, msnitzer, prajnoha, zkabelac
Version:	8.6	Keywords:	MigratedToJIRA, Triaged
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-09-23 15:55:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Carlos Maiolino 2022-04-25 11:26:30 UTC

Description of problem:

While attempting to remove a large writecache LV (1TB) from a VG, the system crashed
due to Out-Of-Memory.

My system actually has 32GiB RAM, and I was playing with a 1TB size writecache

Version-Release number of selected component (if applicable):

RHEL 8.6

How reproducible:

Always

Steps to Reproduce:
- Add a large device as a writecache to a VG:
#lvconvert --type writecache --cachevol fast vg/main

- Attempt to remove it from the VG with lvconvert --uncache (or split cache)

Actual results:
System crashes and enters in an unrecoverable state even if the volume isn't mounted automatically. During boot, lvm spends a long time trying to scan the device with the large cache, and the system fails to boot because it starts to throw OOM errors before finishing to boot.

Expected results:

- LVM prevents a large device to be added as a writecache if the system has not enough memory to deal with it.

- System shouldn't crash while removing a cache from the VG.
If a system has been using a cache device, but for some reason the total RAM has to be reduced, this shouldn't cause a crash while removing the cache device from the volume group.

Additional info:

I managed to boot in recovery mode, but any attempt to remove the writecache device from the VG failed. The system was unable to remove it and crashed.

I manage to manually remove the writecache device from the VG by hacking the current VG configuration file, removing the writecache from it, and restoring the configuration without the writecache device by using vgrestore.

Comment 1 David Teigland 2022-04-25 14:27:05 UTC

I think this is a duplicate of bug 2059644 where the fix was to print a warning when writecache requires >50% of system memory, and requires a confirmation if it requires >90% of memory.  It doesn't currently prevent creating a large writecache, but we could extend the solution to actually fail to create at some memory percentage.

Comment 2 Carlos Maiolino 2022-04-25 15:05:14 UTC

(In reply to David Teigland from comment #1)
> I think this is a duplicate of bug 2059644 where the fix was to print a
> warning when writecache requires >50% of system memory, and requires a
> confirmation if it requires >90% of memory.  It doesn't currently prevent
> creating a large writecache, but we could extend the solution to actually
> fail to create at some memory percentage.

Hi David.

Yeah, that seems a reasonable solution to avoid people creating caches that will
likely crash their systems, but I don't think this will do anything if somebody
reduces their system's memory after creating the cache.

But anyway, this is just an idea based on the fact my biggest problem when I hit
this, was to actually recover the system, giving the fact I couldn't boot it because
of the big cache, and was unable to remove it too after I realized my mistake.

Cheers

Comment 3 David Teigland 2022-04-25 17:12:24 UTC

> likely crash their systems, but I don't think this will do anything if
> somebody reduces their system's memory after creating the cache.

Yes, we could apply similar checking in the activation path and fail to activate an LV if we think the writecache would use too much memory.  Or dm-writecache could do some similar checking in the kernel.  Mikulas, do you think either of those options makes sense?

> But anyway, this is just an idea based on the fact my biggest problem when I
> hit this, was to actually recover the system, giving the fact I couldn't boot it
> because of the big cache, and was unable to remove it too after I realized my
> mistake.

If the system is autoactivating the problematic LV then it's difficult to intervene and fix the problem, so I do think we should have a better way to handle this.

Once you've started the system successfully and the problematic writecache is inactive, then lvconvert --splitcache --force LV will forcibly detach the writecache without attempting to activate and write back the data, with potential data loss.

Comment 5 RHEL Program Management 2023-09-23 15:53:06 UTC

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 6 RHEL Program Management 2023-09-23 15:55:07 UTC

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.