Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1982889

Summary: IDTYPE=lvmlv_uuid stacked devices shouldn't cause warnings once pvremove'd and the underlining lv is also removed
Product: Red Hat Enterprise Linux 8 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: David Teigland <teigland>
lvm2 sub component: Devices, Filtering and Stacking QA Contact: cluster-qe <cluster-qe>
Status: CLOSED NOTABUG Docs Contact:
Severity: low    
Priority: low CC: agk, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, zkabelac
Version: 8.5Keywords: Triaged
Target Milestone: betaFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-04-11 23:45:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2021-07-15 22:37:55 UTC
Description of problem:
After a normal PV is remove and the underling storage is removed, it properly doesn't cause a warning. PVs on stacked LVs do. 

[root@hayes-01 ~]# pvcreate  /dev/sdg1
  Physical volume "/dev/sdg1" successfully created.
[root@hayes-01 ~]# vgcreate vg /dev/sdg1
  Volume group "vg" successfully created with system ID hayes-01.lab.msp.redhat.com
[root@hayes-01 ~]# lvcreate --type linear -n lv_stack -L 500M vg
  Logical volume "lv_stack" created.

[root@hayes-01 ~]# pvscan
  PV /dev/sdg1   VG vg              lvm2 [<1.82 TiB / <1.82 TiB free]
  Total: 1 [<1.82 TiB] / in use: 1 [<1.82 TiB] / in no VG: 0 [0   ]

[root@hayes-01 ~]# pvcreate --config devices/scan_lvs=1 /dev/vg/lv_stack
  Physical volume "/dev/vg/lv_stack" successfully created.
[root@hayes-01 ~]# grep stack /etc/lvm/devices/system.devices 
IDTYPE=lvmlv_uuid IDNAME=LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 DEVNAME=/dev/vg/lv_stack PVID=VAxdEeDwB80Ul3Ge5j6KpYKzLI31uBZN

[root@hayes-01 ~]# pvremove --config devices/scan_lvs=1 /dev/vg/lv_stack
  Labels on physical volume "/dev/vg/lv_stack" successfully wiped.

# PVID is removed
[root@hayes-01 ~]# grep stack /etc/lvm/devices/system.devices 
IDTYPE=lvmlv_uuid IDNAME=LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 DEVNAME=/dev/vg/lv_stack PVID=.

[root@hayes-01 ~]# pvscan
  PV /dev/sdg1   VG vg              lvm2 [<1.82 TiB / <1.82 TiB free]
  Total: 1 [<1.82 TiB] / in use: 1 [<1.82 TiB] / in no VG: 0 [0   ]
[root@hayes-01 ~]# lvs
  LV       VG Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_stack vg -wi-a----- 500.00m                                                    

[root@hayes-01 ~]# vgremove -f vg
  Logical volume "lv_stack" successfully removed.
  Volume group "vg" successfully removed

# lvmlv_uuid device missing warning now appears
[root@hayes-01 ~]# pvscan
  Devices file lvmlv_uuid LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 PVID none last seen on /dev/vg/lv_stack not found.
  PV /dev/sdg1                      lvm2 [<1.82 TiB]
  Total: 1 [<1.82 TiB] / in use: 0 [0   ] / in no VG: 1 [<1.82 TiB]

[root@hayes-01 ~]# grep stack /etc/lvm/devices/system.devices 
IDTYPE=lvmlv_uuid IDNAME=LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 DEVNAME=/dev/vg/lv_stack PVID=.

[root@hayes-01 ~]# pvremove -f /dev/sdg1
  Devices file lvmlv_uuid LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 PVID none last seen on /dev/vg/lv_stack not found.
  Labels on physical volume "/dev/sdg1" successfully wiped.

# I removed the backing device for the last PV /dev/sdg1 as well, but there is no warning about that device not found
[root@hayes-01 ~]# echo offline > /sys/block/sdg/device/state

[root@hayes-01 ~]# pvscan
  Devices file lvmlv_uuid LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 PVID none last seen on /dev/vg/lv_stack not found.
  No matching physical volumes found

[root@hayes-01 ~]# cat /etc/lvm/devices/system.devices 
# LVM uses devices listed in this file.
# Created by LVM command pvremove pid 50750 at Thu Jul 15 17:05:20 2021
SYSTEMID=hayes-01.lab.msp.redhat.com
VERSION=1.1.391
IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bade2acdbd28 DEVNAME=/dev/sde1 PVID=. PART=1
IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bb4a313e4759 DEVNAME=/dev/sdf1 PVID=. PART=1
IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291be7e6227ca72 DEVNAME=/dev/sdi1 PVID=. PART=1
IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bf456dfeb02b DEVNAME=/dev/sdj1 PVID=. PART=1
IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291ba5c230d16c6 DEVNAME=/dev/sdd1 PVID=. PART=1
IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291b9e01bad8691 DEVNAME=/dev/sdc1 PVID=. PART=1
IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291b958138a7c80 DEVNAME=/dev/sdb1 PVID=. PART=1
IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bdea594f2502 DEVNAME=/dev/sdh1 PVID=. PART=1
IDTYPE=sys_wwid IDNAME=naa.6d094660575ece002291bd67517f677a DEVNAME=/dev/sdg1 PVID=. PART=1
IDTYPE=lvmlv_uuid IDNAME=LVM-DsL7OSnInKDkkVvkKkFGabtdfK9SevEPLH0eWngN1onjDaONci8g7hfQW7j62Fq1 DEVNAME=/dev/vg/lv_stack PVID=.


Version-Release number of selected component (if applicable):
kernel-4.18.0-319.el8    BUILT: Mon Jun 28 20:52:51 CDT 2021
lvm2-2.03.12-5.el8    BUILT: Tue Jul 13 11:50:03 CDT 2021
lvm2-libs-2.03.12-5.el8    BUILT: Tue Jul 13 11:50:03 CDT 2021

Comment 1 David Teigland 2021-07-16 17:29:44 UTC
The general principle is that the user needs to explicitly add and remove devices from the devices file using lvmdevices --adddev|--deldev.
So, pvremove doesn't remove the PV from the file, it just clears the PVID field as shown.

As a way to ease the user transition, we made a special case for pvcreate/vgcreate so that those commands *do* add the target device to the devices file.  This exception is technically a violation of the devices file concept, and we may at some point add a config setting to turn off this exception for pvcreate/vgcreate.  In that case, a user would need to do lvmdevices --adddev prior to running pvcreate on the device.

We've not come up with any exceptions where a command should automatically *remove* a device from the devices file, as we have for adding devices.  It's possible that a case might come up which we've not thought of (possibly the stacking case if we find that's a recurring issue?)

The stacking scenario where an LV is the device is being handled in the same way as a scsi device, where the user needs to explicitly remove it from the devices file if it's permanently detached from the system.  (In the test above, setting the state to "offline" isn't actually detaching the device, it's still visible on the system even while offline.)

[root@null-03 ~]# pvs
  PV         VG           Fmt  Attr PSize   PFree  
  /dev/sda2  rhel_null-03 lvm2 a--  465.27g      0 
  /dev/sdf   ff           lvm2 a--  931.01g 931.01g
[root@null-03 ~]# lvmdevices
  Device /dev/sdf IDTYPE=sys_wwid IDNAME=naa.690b11c0000438ad0000057150910423 DEVNAME=/dev/sdf PVID=8hkmvJmTBHhnzn63RFXEFAdEy64I24HM
  Device /dev/sdb IDTYPE=sys_wwid IDNAME=naa.690b11c0000438ad0000056550910404 DEVNAME=/dev/sdb PVID=b1BaIjgRKsoXI9CGdWhJ50kKPaOZDZzo
  Device /dev/sda2 IDTYPE=sys_wwid IDNAME=naa.50014ee0ae1598c2 DEVNAME=/dev/sda2 PVID=0GfvU7fNrko5iAn1UIVOoKN2JyZr1s8z PART=2
[root@null-03 ~]# pvs /dev/sdb
  Cannot use PV /dev/sdb in shared VG bbsan.
[root@null-03 ~]# pvs /dev/sdf
  PV         VG Fmt  Attr PSize   PFree  
  /dev/sdf   ff lvm2 a--  931.01g 931.01g
[root@null-03 ~]# echo "offline" > /sys/block/sdf/device/state
[root@null-03 ~]# pvs
  PV         VG           Fmt  Attr PSize   PFree
  /dev/sda2  rhel_null-03 lvm2 a--  465.27g    0 
[root@null-03 ~]# pvs /dev/sdf
  Cannot use /dev/sdf: device is too small (pv_min_size)
[root@null-03 ~]# cat /sys/block/sdf/device/state 
offline
[root@null-03 ~]# cat /sys/block/sdf/device/wwid
naa.690b11c0000438ad0000057150910423

Comment 2 David Teigland 2021-07-19 15:46:20 UTC
AFAICT we can probably close this, but parking in need info for a bit in case there's some usability aspect to this that we should look at.

Comment 4 Corey Marthaler 2022-04-11 23:44:27 UTC
You're correct, this can be closed. 

This LV stack removal condition is no different than any other device being removed with an existing devicesfile entry. Plus, we also already have bug 1956966 to deal with the potential annoyance of stale device/partition entries. 


# A similar md example.
[root@hayes-01 ~]# pvcreate /dev/md127
  Physical volume "/dev/md127" successfully created.
[root@hayes-01 ~]# grep md /etc/lvm/devices/system.devices 
IDTYPE=md_uuid IDNAME=24904443-4c7a-7e4f-2f97-8ecfec58aeea DEVNAME=/dev/md127 PVID=f1lfjzc78b3oqO9xfbvjCqInli84OHwT

[root@hayes-01 ~]# pvremove /dev/md127
  Labels on physical volume "/dev/md127" successfully wiped.
[root@hayes-01 ~]# grep md /etc/lvm/devices/system.devices 
IDTYPE=md_uuid IDNAME=24904443-4c7a-7e4f-2f97-8ecfec58aeea DEVNAME=/dev/md127 PVID=.

[root@hayes-01 ~]# mdadm --stop /dev/md127
mdadm: stopped /dev/md127
[root@hayes-01 ~]# mdadm --zero-superblock /dev/sdc1 /dev/sde1
[root@hayes-01 ~]# pvscan
  Devices file md_uuid 24904443-4c7a-7e4f-2f97-8ecfec58aeea PVID none last seen on /dev/md127 not found.
  No matching physical volumes found