2049091 – ReaR can exclude still wanted multipath devices

Bug 2049091 - ReaR can exclude still wanted multipath devices

Summary: ReaR can exclude still wanted multipath devices

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	rear
Sub Component:
Version:	8.5
Hardware:	All
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Pavel Cahyna
QA Contact:	David Jež
Docs Contact:	Šárka Jana
URL:
Whiteboard:
Depends On:
Blocks:	2054294
TreeView+	depends on / blocked

Reported:	2022-02-01 14:30 UTC by Renaud Métrich
Modified:	2022-05-10 14:55 UTC (History)
CC List:	4 users (show)
Fixed In Version:	rear-2.6-4.el8
Doc Type:	Bug Fix
Doc Text:	.`ReaR` does not incorrectly exclude multipath devices from the backup Previously, `ReaR` was incorrectly excluding certain multipath devices whose names contained the names of multipath devices that should have been excluded from the backup. For example, if a device named `/dev/mapper/mpatha` was excluded from the backup, then a second device named `/dev/mapper/mpathaa` would be incorrectly excluded as well. This would occur with more than 26 multipath devices. The bug has been fixed and `ReaR` now does not exclude multipath devices from the backup unless they should be excluded. Note that you have to specify `AUTOEXCLUDE_MULTIPATH=n` in the `ReaR` configuration file if there are multipath devices that should be included in the backup, otherwise `ReaR` excludes all multipath devices automatically. This behavior has not changed.
Clone Of:
Clones:	2054294 (view as bug list)
Environment:
Last Closed:	2022-05-10 14:34:16 UTC
Type:	Bug
Target Upstream Version:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	rear rear pull 2750	None	open	multipath: fix exclusion of still wanted devices	2022-02-01 14:30:40 UTC
Red Hat Issue Tracker	RHELPLAN-110537	None	None	None	2022-02-01 14:32:51 UTC
Red Hat Product Errata	RHBA-2022:1966	None	None	None	2022-05-10 14:34:25 UTC

Description Renaud Métrich 2022-02-01 14:30:40 UTC

Description of problem:

The current code excluding multipath devices is broken when a device being excluded matches other devices.
This leads to excluding wanted devices.
This happens when having custom alias for multipath devices or there are more than 26 multipath devices and 'mpatha' is getting excluded, which leads to excluding all 'mpathaX' devices are well.

See example below:

- have /boot on a dedicated multipath device 'mpathba'
- have / on a dedicated multipath device 'mpatha' and LVM vg 'rhel'
- have a dedicated multipath device 'mpathb' and LVM vg 'data'
- tell ReaR to only include 'rhel' VG (and not 'data' VG)
- tell ReaR to not auto-exclude multipath devices (otherwise everything gets commented out)

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
$ lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda               8:0    0    5G  0 disk
└─mpathba       253:1    0    5G  0 mpath
  └─mpathba1    253:2    0    1G  0 part  /boot
sdb               8:16   0    5G  0 disk
└─mpathba       253:1    0    5G  0 mpath
  └─mpathba1    253:2    0    1G  0 part  /boot
sdc               8:32   0   20G  0 disk
└─mpatha        253:0    0   20G  0 mpath
  ├─mpatha1     253:3    0    1G  0 part
  └─mpatha2     253:4    0   19G  0 part
    ├─rhel-root 253:5    0   10G  0 lvm   /
    └─rhel-swap 253:6    0    2G  0 lvm   [SWAP]
sdd               8:48   0   20G  0 disk
└─mpatha        253:0    0   20G  0 mpath
  ├─mpatha1     253:3    0    1G  0 part
  └─mpatha2     253:4    0   19G  0 part
    ├─rhel-root 253:5    0   10G  0 lvm   /
    └─rhel-swap 253:6    0    2G  0 lvm   [SWAP]
sde               8:64   0    2G  0 disk
└─mpathb        253:7    0    2G  0 mpath
sdf               8:80   0    2G  0 disk
└─mpathb        253:7    0    2G  0 mpath
sr0              11:0    1 1024M  0 rom

$ vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  data   1   0   0 wz--n-  <2.00g <2.00g
  rhel   1   2   0 wz--n- <19.00g <7.00g

$ cat /etc/rear/local.conf
ONLY_INCLUDE_VG=("rhel")
AUTOEXCLUDE_MULTIPATH=n
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

With original code:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
$ rear mkrescue
$ grep -v ^# /var/lib/rear/layout/disklayout.conf | grep "mpathba"
--> device excluded even though it hosts /boot
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

With the fix:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
$ rear mkrescue
$ grep -v ^# /var/lib/rear/layout/disklayout.conf | grep "mpathba"
fs /dev/mapper/mpathba1 /boot xfs ...
multipath /dev/mapper/mpathba 5368709120 msdos /dev/sda,/dev/sdb
part /dev/mapper/mpathba 1073741824 1048576 primary none /dev/mapper/mpathba1
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

The root cause behind this is ReaR excludes 'mpathb' (hosting the 'data' VG) through using a non-word matching grep command, which causes 'mpathba' (hosting /boot) to be excluded as well.


Version-Release number of selected component (if applicable):

rear-2.4 and 2.6

How reproducible:

Always, see above.

Steps to Reproduce:
1. Install a VM with 3 multipath disks (only first used during installation)

  "rhel" VG being the system disk (mpatha)

2. Move /boot to disk 3 (initially mpathc)

  # fdisk /dev/mapper/mpathc
  # mkfs.xfs /dev/mapper/mpathc1
  # mount /dev/mapper/mpathc1 /mnt
  # tar cf -C /boot . | tar xf -C /mnt
  # umount /boot
  # vi /etc/fstab (to change uuid for /boot to new one)

3. Reconfigure "mpathc" in "mpathba"

  # multipath -f /dev/mapper/mpathc
  # vi /etc/multipath/bindings
  # systemctl reload multipathd

4. Create a rescue

Comment 1 Pavel Cahyna 2022-02-01 14:59:26 UTC

thanks for the PR, looking...

Comment 2 Pavel Cahyna 2022-02-02 11:17:13 UTC

I am trying it without moving /boot to a separate device (to siplify the reproducer). The generated disklayout is also incorrect, but at least it is detected and rear aborts.

LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
ERROR: 
====================
BUG in /usr/share/rear/layout/save/default/950_verify_disklayout_file.sh line 254:
'Entries in /var/lib/rear/layout/disklayout.conf are broken ('rear recover' would fail)'
--------------------
Please report this issue at https://github.com/rear/rear/issues
and include the relevant parts from /var/log/rear/rear-ibm-p9z-27-lp1.log
preferably with full debug information via 'rear -D savelayout'
====================
Some latest log messages since the last called script 950_verify_disklayout_file.sh:
  2022-02-02 06:10:42.927818695 Including layout/save/default/950_verify_disklayout_file.sh
  2022-02-02 06:10:42.929142978 Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
  2022-02-02 06:10:42.935657492 Verifying that the 'disk' entries in /var/lib/rear/layout/disklayout.conf are correct
  2022-02-02 06:10:42.944848092 Verifying that the 'lvm...' entries in /var/lib/rear/layout/disklayout.conf are correct
  2022-02-02 06:10:42.964778550 LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
  2022-02-02 06:10:42.967817545 LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
Aborting due to an error, check /var/log/rear/rear-ibm-p9z-27-lp1.log for details
Terminated

The generated disklayout has:
# grep -v '^#' /var/lib/rear/layout/disklayout.conf
lvmgrp /dev/rhel_ibm-p9z-27-lp1 4096 9982 40886272
lvmvol /dev/rhel_ibm-p9z-27-lp1 root 37572575232b linear 
lvmvol /dev/rhel_ibm-p9z-27-lp1 swap 4294967296b linear 
fs /dev/mapper/rhel_ibm--p9z--27--lp1-root / xfs uuid=bed18157-ec8c-4e4e-82a0-7646a529cd90 label=  options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
swap /dev/mapper/rhel_ibm--p9z--27--lp1-swap uuid=e8b852f7-e668-47de-8d80-74ca005d45bd label=

Comment 3 Pavel Cahyna 2022-02-02 13:21:10 UTC

The detection shown above unfortunately works only for LVM. Filesystems sitting directly on disks and disk partition are omitted silently (that was probably your point with the moving /boot trick). One can install with plain autopartitioning (autopart --type=plain in kickstart) and one will reproduce the issue on the root disk (to reduce manual interventions).

There are other steps that one can remove to create a minimal reproducer. Names like mpathaa are not needed either, you can name your unused mpath device just "mpath" without suffix and it will match "mpatha" of the root disk. And "rear savelayout" is enough to create the layout (no need for mkrescue). On the other hand, having a second VG and exclude is essential (ReaR includes empty disks by default, but for some reason that does not trigger the problem).

Here is the complete reproducer:

# lsblk
NAME        MAJ:MIN RM SIZE RO TYPE  MOUNTPOINT
sda           8:0    0  40G  0 disk  
└─mpatha    253:0    0  40G  0 mpath 
  ├─mpatha1 253:2    0   4M  0 part  
  ├─mpatha2 253:3    0   1G  0 part  /boot
  └─mpatha3 253:4    0  39G  0 part  /
sdb           8:16   0   5G  0 disk  
└─mpathb    253:1    0   5G  0 mpath 
  └─mpathb1 253:6    0   4G  0 part  [SWAP]
sdc           8:32   0   5G  0 disk  
└─mpathc    253:5    0   5G  0 mpath 
# multipath -f mpathc
# sed  -i.orig 's/mpathc /mpath /' /etc/multipath/bindings
# systemctl restart multipathd
# lsblk
NAME        MAJ:MIN RM SIZE RO TYPE  MOUNTPOINT
sda           8:0    0  40G  0 disk  
└─mpatha    253:0    0  40G  0 mpath 
  ├─mpatha1 253:2    0   4M  0 part  
  ├─mpatha2 253:3    0   1G  0 part  /boot
  └─mpatha3 253:4    0  39G  0 part  /
sdb           8:16   0   5G  0 disk  
└─mpathb    253:1    0   5G  0 mpath 
  └─mpathb1 253:6    0   4G  0 part  [SWAP]
sdc           8:32   0   5G  0 disk  
└─mpath     253:5    0   5G  0 mpath 
# yum -y install rear lvm2
# echo AUTOEXCLUDE_MULTIPATH=n >> /etc/rear/local.conf
# pvcreate /dev/mapper/mpath
# vgcreate unused /dev/mapper/mpath
# echo 'EXCLUDE_VG=("unused")' >> /etc/rear/local.conf
# rear savelayout
# grep -v '^#' /var/lib/rear/layout/disklayout.conf
<nothing>
# cat /var/lib/rear/layout/disklayout.conf
# Disk layout dated 20220202080156 (YYYYmmddHHMMSS)
# NAME                    KNAME     PKNAME    TRAN TYPE  FSTYPE       SIZE MOUNTPOINT
# /dev/sda                /dev/sda                 disk  mpath_member  40G 
# `-/dev/mapper/mpatha    /dev/dm-0 /dev/sda       mpath               40G 
#   |-/dev/mapper/mpatha1 /dev/dm-2 /dev/dm-0      part                 4M 
#   |-/dev/mapper/mpatha2 /dev/dm-3 /dev/dm-0      part  xfs            1G /boot
#   `-/dev/mapper/mpatha3 /dev/dm-4 /dev/dm-0      part  xfs           39G /
# /dev/sdb                /dev/sdb                 disk  mpath_member   5G 
# `-/dev/mapper/mpathb    /dev/dm-1 /dev/sdb       mpath                5G 
#   `-/dev/mapper/mpathb1 /dev/dm-6 /dev/dm-1      part  swap           4G [SWAP]
# /dev/sdc                /dev/sdc                 disk  mpath_member   5G 
# `-/dev/mapper/mpath     /dev/dm-5 /dev/sdc       mpath LVM2_member    5G 
# /dev/sdd                /dev/sdd                 disk  mpath_member   5G 
# `-/dev/mapper/mpathd    /dev/dm-7 /dev/sdd       mpath                5G 
# /dev/sde                /dev/sde                 disk  mpath_member   5G 
# `-/dev/mapper/mpathe    /dev/dm-8 /dev/sde       mpath                5G 
# Format for LVM PVs
# lvmdev <volume_group> <device> [<uuid>] [<size(bytes)>]
#lvmdev /dev/unused /dev/mapper/mpath cx2cyW-ZP1J-fNGK-wmrU-Up6t-5w4P-cGw9r3 10485760
# Format for LVM VGs
# lvmgrp <volume_group> <extentsize> [<size(extents)>] [<size(bytes)>]
#lvmgrp /dev/unused 4096 1279 5238784
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
#fs /dev/mapper/mpatha2 /boot xfs uuid=83a7820e-304a-4b42-be85-ce3889d76300 label=  options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
#fs /dev/mapper/mpatha3 / xfs uuid=bb651a8d-a360-455d-b2f4-52fb28deae70 label=  options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
#swap /dev/mapper/mpathb1 uuid=287bb9ed-b5df-46a5-9f5c-5173f2988ca3 label=
#multipath /dev/mapper/mpathe 5368709120 msdos /dev/sde
#multipath /dev/mapper/mpathd 5368709120 msdos /dev/sdd
#multipath /dev/mapper/mpath 5368709120 unknown /dev/sdc
#multipath /dev/mapper/mpathb 5368709120 msdos /dev/sdb
#part /dev/mapper/mpathb 4322230272 1048576 primary none /dev/mapper/mpathb1
#multipath /dev/mapper/mpatha 42949672960 msdos /dev/sda
#part /dev/mapper/mpatha 4194304 1048576 primary boot,prep /dev/mapper/mpatha1
#part /dev/mapper/mpatha 1073741824 5242880 primary none /dev/mapper/mpatha2
#part /dev/mapper/mpatha 41870688256 1078984704 primary none /dev/mapper/mpatha3

This is with plain partitioning, with the default LVM partitioning it produces an error like shown in the previous comment.

Comment 17 errata-xmlrpc 2022-05-10 14:34:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rear bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1966

Note You need to log in before you can comment on or make changes to this bug.