RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2026640 - Host fail to boot if lvm filter uses /dev/disk/by-id/lvm-pv-uuid-*
Summary: Host fail to boot if lvm filter uses /dev/disk/by-id/lvm-pv-uuid-*
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 2026370
TreeView+ depends on / blocked
 
Reported: 2021-11-25 11:31 UTC by Nir Soffer
Modified: 2022-05-10 16:37 UTC (History)
12 users (show)

Fixed In Version: lvm2-2.03.14-3.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-10 15:22:14 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CLUSTERQE-5269 0 None None None 2022-01-14 11:01:43 UTC
Red Hat Issue Tracker CLUSTERQE-5512 0 None None None 2022-03-16 09:01:00 UTC
Red Hat Issue Tracker RHELPLAN-103940 0 None None None 2021-11-25 11:32:45 UTC
Red Hat Product Errata RHBA-2022:2038 0 None None None 2022-05-10 15:22:30 UTC

Description Nir Soffer 2021-11-25 11:31:21 UTC 
Description of problem:

When using udev "stable" link in lvm filter:

# grep ^filter /etc/lvm/lvm.conf
filter = ["a|^/dev/disk/by-id/lvm-pv-uuid-5M4lBs-4jIZ-2NBF-DRSl-dHwR-6Vfs-PQIOEB$|", "r|.*|"]

This link points to /dev/vda3:

# ls -l /dev/disk/by-id/lvm-pv-uuid-5M4lBs-4jIZ-2NBF-DRSl-dHwR-6Vfs-PQIOEB
lrwxrwxrwx. 1 root root 10 Nov 21 13:34
/dev/disk/by-id/lvm-pv-uuid-5M4lBs-4jIZ-2NBF-DRSl-dHwR-6Vfs-PQIOEB -> ../../vda3

During boot, host fail to mount /dev/vda3, and go into emergency shell.

In the logs we see:

$ journalctl -o json-pretty has:

    "MESSAGE" : "/dev/vda3 excluded by filters: device is rejected by filter config.", ...
    "_CMDLINE" : "/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline 
                  --udevoutput --journal=output /dev/vda3",

Related change:

https://github.com/lvmteam/lvm2/commit/67722b312390cdab29c076c912e14bd739c5c0f6#diff-6a1e9a3e15f9d614cbda0b5b26084c30ffa609a9c951ed1f89fd8a25a12edbb3R82

Why RHV uses /dev/disk/by-id/lvm-pv-uuid-* links in the lvm filter?
See bug 1635614.

Version-Release number of selected component (if applicable):
lvm2-2.03.14-1.el8.x86_64

How reproducible:
Always (oVirt system tests failed for few days because of this bug)

Steps to Reproduce:
1. Configure lvm filter using udev /dev/disk/by-id/lvm-pv-uuid-* link
2. Reboot

Actual results:
LVM reject the device, mount fail, host go into emergency shell.

Expected results:
LVM uses the device pointed by the symlink


Additional info:

According to David Teigland:

>         "_CMDLINE" : "/usr/sbin/lvm pvscan --cache --listvg
> --checkcomplete --vgonline --udevoutput --journal=output /dev/vda3",

is the udev rule for the new RHEL9 autoactivation method, but we see
this rule in a Centos Stream 8.

How a change from the RHEL 9 branch went into RHEL 8 release?!

This change breaks now oVirt users using Centos Stream 8. If it will be
released in RHEL 8.6, it will break RHV installations.
Comment 1 Nir Soffer 2021-11-25 11:33:52 UTC 
Didi, can you add more details on how to reproduce this?

I guess we need specific partition layout since we reproduced this
only on oVirt node (RHV-H).
Comment 2 Yedidyah Bar David 2021-11-25 12:48:04 UTC 
(In reply to Nir Soffer from comment #1)
> Didi, can you add more details on how to reproduce this?

Not sure how to create a minimal reproducer, sorry.

> 
> I guess we need specific partition layout since we reproduced this
> only on oVirt node (RHV-H).

This is what you have on a live machine installed from ovirt-node:

# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sun Nov 14 02:26:16 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/onn_ibm-p8-kvm-03-guest-02/ovirt-node-ng-4.5.0-0.20211111.0+1 / xfs defaults,discard 0 0
UUID=2fd3bb26-3437-4b89-b24f-3a7b067c3d57 /boot                   xfs     defaults        0 0
/dev/mapper/onn_ibm--p8--kvm--03--guest--02-home /home xfs defaults,discard 0 0
/dev/mapper/onn_ibm--p8--kvm--03--guest--02-tmp /tmp xfs defaults,discard 0 0
/dev/mapper/onn_ibm--p8--kvm--03--guest--02-var /var xfs defaults,discard 0 0
/dev/mapper/onn_ibm--p8--kvm--03--guest--02-var_crash /var/crash xfs defaults,discard 0 0
/dev/mapper/onn_ibm--p8--kvm--03--guest--02-var_log /var/log xfs defaults,discard 0 0
/dev/mapper/onn_ibm--p8--kvm--03--guest--02-var_log_audit /var/log/audit xfs defaults,discard 0 0
/dev/mapper/onn_ibm--p8--kvm--03--guest--02-swap none                    swap    defaults        0 0

# lvs
  LV                                 VG                         Attr       LSize   Pool   Origin                           Data%  Meta%  Move Log Cpy%Sync Convert
  home                               onn_ibm-p8-kvm-03-guest-02 Vwi-aotz--   1.00g pool00                                  1.04                                   
  ovirt-node-ng-4.5.0-0.20211111.0   onn_ibm-p8-kvm-03-guest-02 Vwi---tz-k <20.09g pool00 root                                                                    
  ovirt-node-ng-4.5.0-0.20211111.0+1 onn_ibm-p8-kvm-03-guest-02 Vwi-aotz-- <20.09g pool00 ovirt-node-ng-4.5.0-0.20211111.0 31.14                                  
  pool00                             onn_ibm-p8-kvm-03-guest-02 twi-aotz-- <57.09g                                         12.85  2.05                            
  root                               onn_ibm-p8-kvm-03-guest-02 Vri---tz-k <20.09g pool00                                                                         
  swap                               onn_ibm-p8-kvm-03-guest-02 -wi-ao----   5.93g                                                                                
  tmp                                onn_ibm-p8-kvm-03-guest-02 Vwi-aotz--   1.00g pool00                                  1.29                                   
  var                                onn_ibm-p8-kvm-03-guest-02 Vwi-aotz--  15.00g pool00                                  4.62                                   
  var_crash                          onn_ibm-p8-kvm-03-guest-02 Vwi-aotz--  10.00g pool00                                  0.11                                   
  var_log                            onn_ibm-p8-kvm-03-guest-02 Vwi-aotz--   8.00g pool00                                  2.83                                   
  var_log_audit                      onn_ibm-p8-kvm-03-guest-02 Vwi-aotz--   2.00g pool00                                  1.42                                   

This is on an OST env with an ovirt-node image from a few days before we started building with the new lvm2.
Comment 3 David Teigland 2021-11-29 15:45:27 UTC 
We can probably improve how lvm handles PV symlinks in the filter.  However, that is not relevant here because this is all an effect of using the RHEL9 autoactivation code in RHEL8.
Comment 4 David Teigland 2021-11-30 17:02:22 UTC 
I have pushed a patch upstream to handle the case of a filter accepting a PV via symlinks, where that PV is needed for autoactivation.
https://sourceware.org/git/?p=lvm2.git;a=commit;h=d12baba1a9bfe2d82537b20bc768758d84b263b6
Comment 9 Corey Marthaler 2022-01-07 16:43:31 UTC 
Marking this Verified:Tested (SanityOnly) in the latest rpms based on our boot stack regression test. 
To be clear QA was never able to reproduce this issue.


kernel-4.18.0-355.el8.kpq0    BUILT: Wed Dec 15 13:27:55 CST 2021
lvm2-2.03.14-3.el8    BUILT: Tue Jan  4 14:54:16 CST 2022
lvm2-libs-2.03.14-3.el8    BUILT: Tue Jan  4 14:54:16 CST 2022
Comment 13 Corey Marthaler 2022-02-01 18:41:13 UTC 
Marking VERIFIED (SanityOnly) with the latest kernel/userspace. Boot and autoactivate sanity checks passed.

kernel-4.18.0-360.el8    BUILT: Sun Jan 16 20:27:55 CST 2022
lvm2-2.03.14-3.el8    BUILT: Tue Jan  4 14:54:16 CST 2022
lvm2-libs-2.03.14-3.el8    BUILT: Tue Jan  4 14:54:16 CST 2022
Comment 15 errata-xmlrpc 2022-05-10 15:22:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2038
Note You need to log in before you can comment on or make changes to this bug.