RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2049091 - ReaR can exclude still wanted multipath devices
Summary: ReaR can exclude still wanted multipath devices
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: rear
Version: 8.5
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Pavel Cahyna
QA Contact: David Jež
Šárka Jana
URL:
Whiteboard:
Depends On:
Blocks: 2054294
TreeView+ depends on / blocked
 
Reported: 2022-02-01 14:30 UTC by Renaud Métrich
Modified: 2022-05-10 14:55 UTC (History)
4 users (show)

Fixed In Version: rear-2.6-4.el8
Doc Type: Bug Fix
Doc Text:
.`ReaR` does not incorrectly exclude multipath devices from the backup Previously, `ReaR` was incorrectly excluding certain multipath devices whose names contained the names of multipath devices that should have been excluded from the backup. For example, if a device named `/dev/mapper/mpatha` was excluded from the backup, then a second device named `/dev/mapper/mpathaa` would be incorrectly excluded as well. This would occur with more than 26 multipath devices. The bug has been fixed and `ReaR` now does not exclude multipath devices from the backup unless they should be excluded. Note that you have to specify `AUTOEXCLUDE_MULTIPATH=n` in the `ReaR` configuration file if there are multipath devices that should be included in the backup, otherwise `ReaR` excludes all multipath devices automatically. This behavior has not changed.
Clone Of:
: 2054294 (view as bug list)
Environment:
Last Closed: 2022-05-10 14:34:16 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github rear rear pull 2750 0 None open multipath: fix exclusion of still wanted devices 2022-02-01 14:30:40 UTC
Red Hat Issue Tracker RHELPLAN-110537 0 None None None 2022-02-01 14:32:51 UTC
Red Hat Product Errata RHBA-2022:1966 0 None None None 2022-05-10 14:34:25 UTC

Description Renaud Métrich 2022-02-01 14:30:40 UTC 
Description of problem:

The current code excluding multipath devices is broken when a device being excluded matches other devices.
This leads to excluding wanted devices.
This happens when having custom alias for multipath devices or there are more than 26 multipath devices and 'mpatha' is getting excluded, which leads to excluding all 'mpathaX' devices are well.

See example below:

- have /boot on a dedicated multipath device 'mpathba'
- have / on a dedicated multipath device 'mpatha' and LVM vg 'rhel'
- have a dedicated multipath device 'mpathb' and LVM vg 'data'
- tell ReaR to only include 'rhel' VG (and not 'data' VG)
- tell ReaR to not auto-exclude multipath devices (otherwise everything gets commented out)

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
$ lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda               8:0    0    5G  0 disk
└─mpathba       253:1    0    5G  0 mpath
  └─mpathba1    253:2    0    1G  0 part  /boot
sdb               8:16   0    5G  0 disk
└─mpathba       253:1    0    5G  0 mpath
  └─mpathba1    253:2    0    1G  0 part  /boot
sdc               8:32   0   20G  0 disk
└─mpatha        253:0    0   20G  0 mpath
  ├─mpatha1     253:3    0    1G  0 part
  └─mpatha2     253:4    0   19G  0 part
    ├─rhel-root 253:5    0   10G  0 lvm   /
    └─rhel-swap 253:6    0    2G  0 lvm   [SWAP]
sdd               8:48   0   20G  0 disk
└─mpatha        253:0    0   20G  0 mpath
  ├─mpatha1     253:3    0    1G  0 part
  └─mpatha2     253:4    0   19G  0 part
    ├─rhel-root 253:5    0   10G  0 lvm   /
    └─rhel-swap 253:6    0    2G  0 lvm   [SWAP]
sde               8:64   0    2G  0 disk
└─mpathb        253:7    0    2G  0 mpath
sdf               8:80   0    2G  0 disk
└─mpathb        253:7    0    2G  0 mpath
sr0              11:0    1 1024M  0 rom

$ vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  data   1   0   0 wz--n-  <2.00g <2.00g
  rhel   1   2   0 wz--n- <19.00g <7.00g

$ cat /etc/rear/local.conf
ONLY_INCLUDE_VG=("rhel")
AUTOEXCLUDE_MULTIPATH=n
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

With original code:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
$ rear mkrescue
$ grep -v ^# /var/lib/rear/layout/disklayout.conf | grep "mpathba"
--> device excluded even though it hosts /boot
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

With the fix:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
$ rear mkrescue
$ grep -v ^# /var/lib/rear/layout/disklayout.conf | grep "mpathba"
fs /dev/mapper/mpathba1 /boot xfs ...
multipath /dev/mapper/mpathba 5368709120 msdos /dev/sda,/dev/sdb
part /dev/mapper/mpathba 1073741824 1048576 primary none /dev/mapper/mpathba1
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

The root cause behind this is ReaR excludes 'mpathb' (hosting the 'data' VG) through using a non-word matching grep command, which causes 'mpathba' (hosting /boot) to be excluded as well.


Version-Release number of selected component (if applicable):

rear-2.4 and 2.6

How reproducible:

Always, see above.

Steps to Reproduce:
1. Install a VM with 3 multipath disks (only first used during installation)

  "rhel" VG being the system disk (mpatha)

2. Move /boot to disk 3 (initially mpathc)

  # fdisk /dev/mapper/mpathc
  # mkfs.xfs /dev/mapper/mpathc1
  # mount /dev/mapper/mpathc1 /mnt
  # tar cf -C /boot . | tar xf -C /mnt
  # umount /boot
  # vi /etc/fstab (to change uuid for /boot to new one)

3. Reconfigure "mpathc" in "mpathba"

  # multipath -f /dev/mapper/mpathc
  # vi /etc/multipath/bindings
  # systemctl reload multipathd

4. Create a rescue
Comment 1 Pavel Cahyna 2022-02-01 14:59:26 UTC 
thanks for the PR, looking...
Comment 2 Pavel Cahyna 2022-02-02 11:17:13 UTC 
I am trying it without moving /boot to a separate device (to siplify the reproducer). The generated disklayout is also incorrect, but at least it is detected and rear aborts.

LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
ERROR: 
====================
BUG in /usr/share/rear/layout/save/default/950_verify_disklayout_file.sh line 254:
'Entries in /var/lib/rear/layout/disklayout.conf are broken ('rear recover' would fail)'
--------------------
Please report this issue at https://github.com/rear/rear/issues
and include the relevant parts from /var/log/rear/rear-ibm-p9z-27-lp1.log
preferably with full debug information via 'rear -D savelayout'
====================
Some latest log messages since the last called script 950_verify_disklayout_file.sh:
  2022-02-02 06:10:42.927818695 Including layout/save/default/950_verify_disklayout_file.sh
  2022-02-02 06:10:42.929142978 Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
  2022-02-02 06:10:42.935657492 Verifying that the 'disk' entries in /var/lib/rear/layout/disklayout.conf are correct
  2022-02-02 06:10:42.944848092 Verifying that the 'lvm...' entries in /var/lib/rear/layout/disklayout.conf are correct
  2022-02-02 06:10:42.964778550 LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
  2022-02-02 06:10:42.967817545 LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
Aborting due to an error, check /var/log/rear/rear-ibm-p9z-27-lp1.log for details
Terminated

The generated disklayout has:
# grep -v '^#' /var/lib/rear/layout/disklayout.conf
lvmgrp /dev/rhel_ibm-p9z-27-lp1 4096 9982 40886272
lvmvol /dev/rhel_ibm-p9z-27-lp1 root 37572575232b linear 
lvmvol /dev/rhel_ibm-p9z-27-lp1 swap 4294967296b linear 
fs /dev/mapper/rhel_ibm--p9z--27--lp1-root / xfs uuid=bed18157-ec8c-4e4e-82a0-7646a529cd90 label=  options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
swap /dev/mapper/rhel_ibm--p9z--27--lp1-swap uuid=e8b852f7-e668-47de-8d80-74ca005d45bd label=
Comment 3 Pavel Cahyna 2022-02-02 13:21:10 UTC 
The detection shown above unfortunately works only for LVM. Filesystems sitting directly on disks and disk partition are omitted silently (that was probably your point with the moving /boot trick). One can install with plain autopartitioning (autopart --type=plain in kickstart) and one will reproduce the issue on the root disk (to reduce manual interventions).

There are other steps that one can remove to create a minimal reproducer. Names like mpathaa are not needed either, you can name your unused mpath device just "mpath" without suffix and it will match "mpatha" of the root disk. And "rear savelayout" is enough to create the layout (no need for mkrescue). On the other hand, having a second VG and exclude is essential (ReaR includes empty disks by default, but for some reason that does not trigger the problem).

Here is the complete reproducer:

# lsblk
NAME        MAJ:MIN RM SIZE RO TYPE  MOUNTPOINT
sda           8:0    0  40G  0 disk  
└─mpatha    253:0    0  40G  0 mpath 
  ├─mpatha1 253:2    0   4M  0 part  
  ├─mpatha2 253:3    0   1G  0 part  /boot
  └─mpatha3 253:4    0  39G  0 part  /
sdb           8:16   0   5G  0 disk  
└─mpathb    253:1    0   5G  0 mpath 
  └─mpathb1 253:6    0   4G  0 part  [SWAP]
sdc           8:32   0   5G  0 disk  
└─mpathc    253:5    0   5G  0 mpath 
# multipath -f mpathc
# sed  -i.orig 's/mpathc /mpath /' /etc/multipath/bindings
# systemctl restart multipathd
# lsblk
NAME        MAJ:MIN RM SIZE RO TYPE  MOUNTPOINT
sda           8:0    0  40G  0 disk  
└─mpatha    253:0    0  40G  0 mpath 
  ├─mpatha1 253:2    0   4M  0 part  
  ├─mpatha2 253:3    0   1G  0 part  /boot
  └─mpatha3 253:4    0  39G  0 part  /
sdb           8:16   0   5G  0 disk  
└─mpathb    253:1    0   5G  0 mpath 
  └─mpathb1 253:6    0   4G  0 part  [SWAP]
sdc           8:32   0   5G  0 disk  
└─mpath     253:5    0   5G  0 mpath 
# yum -y install rear lvm2
# echo AUTOEXCLUDE_MULTIPATH=n >> /etc/rear/local.conf
# pvcreate /dev/mapper/mpath
# vgcreate unused /dev/mapper/mpath
# echo 'EXCLUDE_VG=("unused")' >> /etc/rear/local.conf
# rear savelayout
# grep -v '^#' /var/lib/rear/layout/disklayout.conf
<nothing>
# cat /var/lib/rear/layout/disklayout.conf
# Disk layout dated 20220202080156 (YYYYmmddHHMMSS)
# NAME                    KNAME     PKNAME    TRAN TYPE  FSTYPE       SIZE MOUNTPOINT
# /dev/sda                /dev/sda                 disk  mpath_member  40G 
# `-/dev/mapper/mpatha    /dev/dm-0 /dev/sda       mpath               40G 
#   |-/dev/mapper/mpatha1 /dev/dm-2 /dev/dm-0      part                 4M 
#   |-/dev/mapper/mpatha2 /dev/dm-3 /dev/dm-0      part  xfs            1G /boot
#   `-/dev/mapper/mpatha3 /dev/dm-4 /dev/dm-0      part  xfs           39G /
# /dev/sdb                /dev/sdb                 disk  mpath_member   5G 
# `-/dev/mapper/mpathb    /dev/dm-1 /dev/sdb       mpath                5G 
#   `-/dev/mapper/mpathb1 /dev/dm-6 /dev/dm-1      part  swap           4G [SWAP]
# /dev/sdc                /dev/sdc                 disk  mpath_member   5G 
# `-/dev/mapper/mpath     /dev/dm-5 /dev/sdc       mpath LVM2_member    5G 
# /dev/sdd                /dev/sdd                 disk  mpath_member   5G 
# `-/dev/mapper/mpathd    /dev/dm-7 /dev/sdd       mpath                5G 
# /dev/sde                /dev/sde                 disk  mpath_member   5G 
# `-/dev/mapper/mpathe    /dev/dm-8 /dev/sde       mpath                5G 
# Format for LVM PVs
# lvmdev <volume_group> <device> [<uuid>] [<size(bytes)>]
#lvmdev /dev/unused /dev/mapper/mpath cx2cyW-ZP1J-fNGK-wmrU-Up6t-5w4P-cGw9r3 10485760
# Format for LVM VGs
# lvmgrp <volume_group> <extentsize> [<size(extents)>] [<size(bytes)>]
#lvmgrp /dev/unused 4096 1279 5238784
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
#fs /dev/mapper/mpatha2 /boot xfs uuid=83a7820e-304a-4b42-be85-ce3889d76300 label=  options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
#fs /dev/mapper/mpatha3 / xfs uuid=bb651a8d-a360-455d-b2f4-52fb28deae70 label=  options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
#swap /dev/mapper/mpathb1 uuid=287bb9ed-b5df-46a5-9f5c-5173f2988ca3 label=
#multipath /dev/mapper/mpathe 5368709120 msdos /dev/sde
#multipath /dev/mapper/mpathd 5368709120 msdos /dev/sdd
#multipath /dev/mapper/mpath 5368709120 unknown /dev/sdc
#multipath /dev/mapper/mpathb 5368709120 msdos /dev/sdb
#part /dev/mapper/mpathb 4322230272 1048576 primary none /dev/mapper/mpathb1
#multipath /dev/mapper/mpatha 42949672960 msdos /dev/sda
#part /dev/mapper/mpatha 4194304 1048576 primary boot,prep /dev/mapper/mpatha1
#part /dev/mapper/mpatha 1073741824 5242880 primary none /dev/mapper/mpatha2
#part /dev/mapper/mpatha 41870688256 1078984704 primary none /dev/mapper/mpatha3

This is with plain partitioning, with the default LVM partitioning it produces an error like shown in the previous comment.
Comment 17 errata-xmlrpc 2022-05-10 14:34:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rear bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1966
Note You need to log in before you can comment on or make changes to this bug.