| Summary: | ReaR can exclude still wanted multipath devices | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Renaud Métrich <rmetrich> | |
| Component: | rear | Assignee: | Pavel Cahyna <pcahyna> | |
| Status: | CLOSED ERRATA | QA Contact: | David Jež <djez> | |
| Severity: | high | Docs Contact: | Šárka Jana <sjanderk> | |
| Priority: | high | |||
| Version: | 8.5 | CC: | djez, ovasik, pcahyna, sjanderk | |
| Target Milestone: | rc | |||
| Target Release: | --- | |||
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | rear-2.6-4.el8 | Doc Type: | Bug Fix | |
| Doc Text: |
.`ReaR` does not incorrectly exclude multipath devices from the backup
Previously, `ReaR` was incorrectly excluding certain multipath devices whose names contained the names of multipath devices that should have been excluded from the backup.
For example, if a device named `/dev/mapper/mpatha` was excluded from the backup, then a second device named `/dev/mapper/mpathaa` would be incorrectly excluded as well.
This would occur with more than 26 multipath devices.
The bug has been fixed and `ReaR` now does not exclude multipath devices from the backup unless they should be excluded.
Note that you have to specify `AUTOEXCLUDE_MULTIPATH=n` in the `ReaR` configuration file if there are multipath devices that should be included in the backup, otherwise `ReaR` excludes all multipath devices automatically. This behavior has not changed.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 2054294 (view as bug list) | Environment: | ||
| Last Closed: | 2022-05-10 14:34:16 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Bug Depends On: | ||||
| Bug Blocks: | 2054294 | |||
thanks for the PR, looking... I am trying it without moving /boot to a separate device (to siplify the reproducer). The generated disklayout is also incorrect, but at least it is detected and rear aborts.
LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
ERROR:
====================
BUG in /usr/share/rear/layout/save/default/950_verify_disklayout_file.sh line 254:
'Entries in /var/lib/rear/layout/disklayout.conf are broken ('rear recover' would fail)'
--------------------
Please report this issue at https://github.com/rear/rear/issues
and include the relevant parts from /var/log/rear/rear-ibm-p9z-27-lp1.log
preferably with full debug information via 'rear -D savelayout'
====================
Some latest log messages since the last called script 950_verify_disklayout_file.sh:
2022-02-02 06:10:42.927818695 Including layout/save/default/950_verify_disklayout_file.sh
2022-02-02 06:10:42.929142978 Verifying that the entries in /var/lib/rear/layout/disklayout.conf are correct ...
2022-02-02 06:10:42.935657492 Verifying that the 'disk' entries in /var/lib/rear/layout/disklayout.conf are correct
2022-02-02 06:10:42.944848092 Verifying that the 'lvm...' entries in /var/lib/rear/layout/disklayout.conf are correct
2022-02-02 06:10:42.964778550 LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
2022-02-02 06:10:42.967817545 LVM no 'lvmdev /dev/rhel_ibm-p9z-27-lp1' for 'lvmvol /dev/rhel_ibm-p9z-27-lp1'
Aborting due to an error, check /var/log/rear/rear-ibm-p9z-27-lp1.log for details
Terminated
The generated disklayout has:
# grep -v '^#' /var/lib/rear/layout/disklayout.conf
lvmgrp /dev/rhel_ibm-p9z-27-lp1 4096 9982 40886272
lvmvol /dev/rhel_ibm-p9z-27-lp1 root 37572575232b linear
lvmvol /dev/rhel_ibm-p9z-27-lp1 swap 4294967296b linear
fs /dev/mapper/rhel_ibm--p9z--27--lp1-root / xfs uuid=bed18157-ec8c-4e4e-82a0-7646a529cd90 label= options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
swap /dev/mapper/rhel_ibm--p9z--27--lp1-swap uuid=e8b852f7-e668-47de-8d80-74ca005d45bd label=
The detection shown above unfortunately works only for LVM. Filesystems sitting directly on disks and disk partition are omitted silently (that was probably your point with the moving /boot trick). One can install with plain autopartitioning (autopart --type=plain in kickstart) and one will reproduce the issue on the root disk (to reduce manual interventions).
There are other steps that one can remove to create a minimal reproducer. Names like mpathaa are not needed either, you can name your unused mpath device just "mpath" without suffix and it will match "mpatha" of the root disk. And "rear savelayout" is enough to create the layout (no need for mkrescue). On the other hand, having a second VG and exclude is essential (ReaR includes empty disks by default, but for some reason that does not trigger the problem).
Here is the complete reproducer:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk
└─mpatha 253:0 0 40G 0 mpath
├─mpatha1 253:2 0 4M 0 part
├─mpatha2 253:3 0 1G 0 part /boot
└─mpatha3 253:4 0 39G 0 part /
sdb 8:16 0 5G 0 disk
└─mpathb 253:1 0 5G 0 mpath
└─mpathb1 253:6 0 4G 0 part [SWAP]
sdc 8:32 0 5G 0 disk
└─mpathc 253:5 0 5G 0 mpath
# multipath -f mpathc
# sed -i.orig 's/mpathc /mpath /' /etc/multipath/bindings
# systemctl restart multipathd
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk
└─mpatha 253:0 0 40G 0 mpath
├─mpatha1 253:2 0 4M 0 part
├─mpatha2 253:3 0 1G 0 part /boot
└─mpatha3 253:4 0 39G 0 part /
sdb 8:16 0 5G 0 disk
└─mpathb 253:1 0 5G 0 mpath
└─mpathb1 253:6 0 4G 0 part [SWAP]
sdc 8:32 0 5G 0 disk
└─mpath 253:5 0 5G 0 mpath
# yum -y install rear lvm2
# echo AUTOEXCLUDE_MULTIPATH=n >> /etc/rear/local.conf
# pvcreate /dev/mapper/mpath
# vgcreate unused /dev/mapper/mpath
# echo 'EXCLUDE_VG=("unused")' >> /etc/rear/local.conf
# rear savelayout
# grep -v '^#' /var/lib/rear/layout/disklayout.conf
<nothing>
# cat /var/lib/rear/layout/disklayout.conf
# Disk layout dated 20220202080156 (YYYYmmddHHMMSS)
# NAME KNAME PKNAME TRAN TYPE FSTYPE SIZE MOUNTPOINT
# /dev/sda /dev/sda disk mpath_member 40G
# `-/dev/mapper/mpatha /dev/dm-0 /dev/sda mpath 40G
# |-/dev/mapper/mpatha1 /dev/dm-2 /dev/dm-0 part 4M
# |-/dev/mapper/mpatha2 /dev/dm-3 /dev/dm-0 part xfs 1G /boot
# `-/dev/mapper/mpatha3 /dev/dm-4 /dev/dm-0 part xfs 39G /
# /dev/sdb /dev/sdb disk mpath_member 5G
# `-/dev/mapper/mpathb /dev/dm-1 /dev/sdb mpath 5G
# `-/dev/mapper/mpathb1 /dev/dm-6 /dev/dm-1 part swap 4G [SWAP]
# /dev/sdc /dev/sdc disk mpath_member 5G
# `-/dev/mapper/mpath /dev/dm-5 /dev/sdc mpath LVM2_member 5G
# /dev/sdd /dev/sdd disk mpath_member 5G
# `-/dev/mapper/mpathd /dev/dm-7 /dev/sdd mpath 5G
# /dev/sde /dev/sde disk mpath_member 5G
# `-/dev/mapper/mpathe /dev/dm-8 /dev/sde mpath 5G
# Format for LVM PVs
# lvmdev <volume_group> <device> [<uuid>] [<size(bytes)>]
#lvmdev /dev/unused /dev/mapper/mpath cx2cyW-ZP1J-fNGK-wmrU-Up6t-5w4P-cGw9r3 10485760
# Format for LVM VGs
# lvmgrp <volume_group> <extentsize> [<size(extents)>] [<size(bytes)>]
#lvmgrp /dev/unused 4096 1279 5238784
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
#fs /dev/mapper/mpatha2 /boot xfs uuid=83a7820e-304a-4b42-be85-ce3889d76300 label= options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
#fs /dev/mapper/mpatha3 / xfs uuid=bb651a8d-a360-455d-b2f4-52fb28deae70 label= options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
#swap /dev/mapper/mpathb1 uuid=287bb9ed-b5df-46a5-9f5c-5173f2988ca3 label=
#multipath /dev/mapper/mpathe 5368709120 msdos /dev/sde
#multipath /dev/mapper/mpathd 5368709120 msdos /dev/sdd
#multipath /dev/mapper/mpath 5368709120 unknown /dev/sdc
#multipath /dev/mapper/mpathb 5368709120 msdos /dev/sdb
#part /dev/mapper/mpathb 4322230272 1048576 primary none /dev/mapper/mpathb1
#multipath /dev/mapper/mpatha 42949672960 msdos /dev/sda
#part /dev/mapper/mpatha 4194304 1048576 primary boot,prep /dev/mapper/mpatha1
#part /dev/mapper/mpatha 1073741824 5242880 primary none /dev/mapper/mpatha2
#part /dev/mapper/mpatha 41870688256 1078984704 primary none /dev/mapper/mpatha3
This is with plain partitioning, with the default LVM partitioning it produces an error like shown in the previous comment.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (rear bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1966 |
Description of problem: The current code excluding multipath devices is broken when a device being excluded matches other devices. This leads to excluding wanted devices. This happens when having custom alias for multipath devices or there are more than 26 multipath devices and 'mpatha' is getting excluded, which leads to excluding all 'mpathaX' devices are well. See example below: - have /boot on a dedicated multipath device 'mpathba' - have / on a dedicated multipath device 'mpatha' and LVM vg 'rhel' - have a dedicated multipath device 'mpathb' and LVM vg 'data' - tell ReaR to only include 'rhel' VG (and not 'data' VG) - tell ReaR to not auto-exclude multipath devices (otherwise everything gets commented out) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 5G 0 disk └─mpathba 253:1 0 5G 0 mpath └─mpathba1 253:2 0 1G 0 part /boot sdb 8:16 0 5G 0 disk └─mpathba 253:1 0 5G 0 mpath └─mpathba1 253:2 0 1G 0 part /boot sdc 8:32 0 20G 0 disk └─mpatha 253:0 0 20G 0 mpath ├─mpatha1 253:3 0 1G 0 part └─mpatha2 253:4 0 19G 0 part ├─rhel-root 253:5 0 10G 0 lvm / └─rhel-swap 253:6 0 2G 0 lvm [SWAP] sdd 8:48 0 20G 0 disk └─mpatha 253:0 0 20G 0 mpath ├─mpatha1 253:3 0 1G 0 part └─mpatha2 253:4 0 19G 0 part ├─rhel-root 253:5 0 10G 0 lvm / └─rhel-swap 253:6 0 2G 0 lvm [SWAP] sde 8:64 0 2G 0 disk └─mpathb 253:7 0 2G 0 mpath sdf 8:80 0 2G 0 disk └─mpathb 253:7 0 2G 0 mpath sr0 11:0 1 1024M 0 rom $ vgs VG #PV #LV #SN Attr VSize VFree data 1 0 0 wz--n- <2.00g <2.00g rhel 1 2 0 wz--n- <19.00g <7.00g $ cat /etc/rear/local.conf ONLY_INCLUDE_VG=("rhel") AUTOEXCLUDE_MULTIPATH=n -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- With original code: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- $ rear mkrescue $ grep -v ^# /var/lib/rear/layout/disklayout.conf | grep "mpathba" --> device excluded even though it hosts /boot -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- With the fix: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- $ rear mkrescue $ grep -v ^# /var/lib/rear/layout/disklayout.conf | grep "mpathba" fs /dev/mapper/mpathba1 /boot xfs ... multipath /dev/mapper/mpathba 5368709120 msdos /dev/sda,/dev/sdb part /dev/mapper/mpathba 1073741824 1048576 primary none /dev/mapper/mpathba1 -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- The root cause behind this is ReaR excludes 'mpathb' (hosting the 'data' VG) through using a non-word matching grep command, which causes 'mpathba' (hosting /boot) to be excluded as well. Version-Release number of selected component (if applicable): rear-2.4 and 2.6 How reproducible: Always, see above. Steps to Reproduce: 1. Install a VM with 3 multipath disks (only first used during installation) "rhel" VG being the system disk (mpatha) 2. Move /boot to disk 3 (initially mpathc) # fdisk /dev/mapper/mpathc # mkfs.xfs /dev/mapper/mpathc1 # mount /dev/mapper/mpathc1 /mnt # tar cf -C /boot . | tar xf -C /mnt # umount /boot # vi /etc/fstab (to change uuid for /boot to new one) 3. Reconfigure "mpathc" in "mpathba" # multipath -f /dev/mapper/mpathc # vi /etc/multipath/bindings # systemctl reload multipathd 4. Create a rescue