Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Force removing all LVs in a vg leaves a cache pool devfs entry behind. VG contained just a single cache pool and one origin on top of raid1 when force remove was attempted as shown in the reproducer. So far I was not able to reproduce manually, possible race.
Reproducer:
# lvcreate --activate ey --type raid1 -m 1 -L 100M -n corigin cache_sanity /dev/sda1 /dev/sdb1
# lvcreate --activate ey --type raid1 -m 1 -L 2G -n force_remove cache_sanity /dev/sdc1 /dev/sdd1
# lvcreate --activate ey --type raid1 -m 1 -L 2G -n force_remove_meta cache_sanity /dev/sdc1 /dev/sdd1
# lvconvert --yes --type cache-pool --cachepolicy mq --cachemode writeback -c 32 --poolmetadata cache_sanity/force_remove_meta cache_sanity/force_remove
# lvconvert --yes --type cache --cachemetadataformat 2 --cachepool cache_sanity/force_remove cache_sanity/corigin
# lvremove -ff cache_sanity
3.10.0-843.el7.x86_64
lvm2-2.02.177-2.el7 BUILT: Wed Feb 7 17:39:26 CET 2018
lvm2-libs-2.02.177-2.el7 BUILT: Wed Feb 7 17:39:26 CET 2018
lvm2-cluster-2.02.177-2.el7 BUILT: Wed Feb 7 17:39:26 CET 2018
package lvm2-lockd is not installed
date: invalid date ‘1970-01-01 UTC package lvm2-lockd is not installed seconds’
package lvm2-python-boom is not installed
date: invalid date ‘1970-01-01 UTC package lvm2-python-boom is not installed seconds’
cmirror-2.02.177-2.el7 BUILT: Wed Feb 7 17:39:26 CET 2018
device-mapper-1.02.146-2.el7 BUILT: Wed Feb 7 17:39:26 CET 2018
device-mapper-libs-1.02.146-2.el7 BUILT: Wed Feb 7 17:39:26 CET 2018
device-mapper-event-1.02.146-2.el7 BUILT: Wed Feb 7 17:39:26 CET 2018
device-mapper-event-libs-1.02.146-2.el7 BUILT: Wed Feb 7 17:39:26 CET 2018
device-mapper-persistent-data-0.7.3-3.el7 BUILT: Tue Nov 14 12:07:18 CET 2017
Test run:
SCENARIO - [force_vg_remove_cache_on_raid]
Create a cache volume, then force remove it while active ensuring no invalid error messages
*** Cache info for this scenario ***
* origin (slow): /dev/sdb1 /dev/sdd1
* pool (fast): /dev/sdf1 /dev/sda1
************************************
Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --activate ey --type raid1 -m 1 -L 4G -n corigin cache_sanity @slow
Waiting until all mirror|raid volumes become fully syncd...
0/1 mirror(s) are fully synced: ( 42.62% )
0/1 mirror(s) are fully synced: ( 92.90% )
1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
Create cache data and cache metadata (fast) volumes
lvcreate --activate ey --type raid1 -m 1 -L 2G -n force_remove cache_sanity @fast
lvcreate --activate ey --type raid1 -m 1 -L 12M -n force_remove_meta cache_sanity @fast
Waiting until all mirror|raid volumes become fully syncd...
2/2 mirror(s) are fully synced: ( 100.00% 100.00% )
Sleeping 15 sec
Sleeping 15 sec
Create cache pool volume by combining the cache data and cache metadata (fast) volumes with policy: mq mode: writeback
lvconvert --yes --type cache-pool --cachepolicy mq --cachemode writeback -c 32 --poolmetadata cache_sanity/force_remove_meta cache_sanity/force_remove
WARNING: Converting cache_sanity/force_remove and cache_sanity/force_remove_meta to cache pool's data and metadata volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Create cached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type cache --cachemetadataformat 2 --cachepool cache_sanity/force_remove cache_sanity/corigin
dmsetup status | grep cache_sanity-corigin | grep writeback | grep -w mq
Force remove all LVs associated with cache_sanity
lvremove -ff cache_sanity
There shouldn't be any left over devfs entries for this lv on cache_sanity/n at /usr/tests/sts-rhel7.5/lvm2/lib/cache_sanity/Cache_sanity.pm line 4868.
I'd expect this to be a bug of 'older' systemd-udevd leaking symlink.
To confirm this - you can enable "verify_udev_operations" in lvm.conf
in this case lvm2 should 'spot' missing symlink removal.
Also 'primary' way is to check present in dm table
(dmsetup table)
If device is NOT there, while symlink is in /dev dir - it's udev bug.
I also believe it's the case of older version of udev in RHEL, since recent upsteam seems to be working fine here.
dm table did not contain any entry related to removed lv. Running ~30 iterations of the same scenario with verify_udev_operations enabled did not reproduce the bug. Reassigning to udev.
Isn't this another version of raid scrub bug 1549272?
Check the log for "Failed to lock logical volume" messages during the scrubbing actions. I don't believe this requires a cluster to reproduce.
Quite frankly I have no idea how to move this forward. I can't reproduce locally and without reproducer it is close to impossible to say what exactly happened and why the symlink wasn't removed. At the very least I need a dump of udev database (before and after lvremove), part of the udev debug log from the time of removal of VG and corresponding udevadm monitor output. According to last few comments the bug was not observed with latest LVM builds so I am closing this as INSUFFICIENT_DATA.
In case someone is able to reproduce please reopen and attach relevant debug information.
Comment 16Red Hat Bugzilla
2023-09-14 04:16:31 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days