Also tried some other builds: redhat-virtualization-host-4.4.0-20200401.0.el8_2 redhat-virtualization-host-4.4.0-20200409.1.el8_2 Upgrade RHVH from the above two builds to redhat-virtualization-host-4.4.0-20200518.0.el8_2 will not enter emergency mode.
Tried to upgrade RHVH from "redhat-virtualization-host-4.4.0-20200518.0.el8_2 " to "redhat-virtualization-host-4.4.1-20200527.0.el8_2", 2 test servers enter emergency mode, the other one succeeds.
I have noticed this machine is using multipath. in /var/log/messages there are 3 device-mapper job that goes for timeout (swap and 2 uuids) systemctl --all --full -type device | grep inactive shows /boot and the /boot/efi this bug is not resembling bz#1859876 but https://bugzilla.redhat.com/show_bug.cgi?id=1859876#c11 mentioning the need for blacklisting local devices checking local devices with device -v2 -d does not list local devices which is good lsblk show only sdb with missing partition mounted on /boot so far following https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/pdf/configuring_device_mapper_multipath/Red_Hat_Enterprise_Linux-8-Configuring_device_mapper_multipath-en-US.pdf I yet to find the reason /boot is not mounted through device-mapper. maybe @Qin will see something.
Checked on the given UEFI machine: 1) it only has one local disk which is not multipath 2) after fresh installation, multipathd.service is inactive, and there is no /etc/multipath.conf file 3) after register to rhvm, multipathd.service is active, /etc/multipath.conf is generated 4) after upgrade from engine side, multipath claimed the device during reboot, the system enters to emergency mode. Pls check https://bugzilla.redhat.com/show_bug.cgi?id=1760223#c33 to see why multipath claims device after host upgrade. Adding the local disk to blacklist can solve this problem: # mkdir /etc/multipath/conf.d # vi /etc/multipath/conf.d/local.conf blacklist { wwid "3600508b1001ceaf040dfabf83d79ced4" }
Upgrade RHVH from redhat-virtualization-host-4.4.1-20200722.0.el8_2 to redhat-virtualization-host-4.4.2-20200812.3.el8_2. System enters emergency mode on all the machines I have tested (these machines have local disks which are not multipath) Test steps: 1. Install RHVH-4.4-20200722.1-RHVH-x86_64-dvd1.iso 2. Set repo and point to "redhat-virtualization-host-4.4.2-20200812.3.el8_2" 3. Add host to RHVM 4. Add nfs storage and create VMs 5. Upgrade host via RHVM UI 6. Focus on the host status after upgrade Test results: After upgrade, the system enter emergency mode.
(In reply to peyu from comment #13) > System enters emergency mode on all the machines I have tested (these machines have local disks which are not multipath) This sounds a different issue than the initially reported one, but may be related.
Raising priority since it's happening on all hosts, should block the release.
(In reply to Qin Yuan from comment #12) > Checked on the given UEFI machine: > 1) it only has one local disk which is not multipath > 2) after fresh installation, multipathd.service is inactive, and there is no > /etc/multipath.conf file > 3) after register to rhvm, multipathd.service is active, /etc/multipath.conf > is generated > 4) after upgrade from engine side, multipath claimed the device during > reboot, the system enters to emergency mode. > > Pls check https://bugzilla.redhat.com/show_bug.cgi?id=1760223#c33 to see why > multipath claims device after host upgrade. > > Adding the local disk to blacklist can solve this problem: > # mkdir /etc/multipath/conf.d > # vi /etc/multipath/conf.d/local.conf > blacklist { > wwid "3600508b1001ceaf040dfabf83d79ced4" > } Did you rebuild initramfs? dracut --force --add multipath This is documented in multipath admin guide: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_device_mapper_multipath/index#proc_multipath-setup-in-initramfs-setting-up-dm-multipath
Please share output of: udevadm info /dev/sd{X} /dev/sd{X} is the local device used for the root file system.
(In reply to Nir Soffer from comment #18) > Please share output of: > > udevadm info /dev/sd{X} > > /dev/sd{X} is the local device used for the root file system. # udevadm info /dev/sda2 P: /devices/pci0000:00/0000:00:01.0/0000:03:00.0/host1/target1:2:0/1:2:0:0/block/sda/sda2 N: sda2 S: disk/by-id/lvm-pv-uuid-pfuelt-KIYO-Ucin-Yvr0-Rjcw-BV2J-Dhe6Np S: disk/by-id/scsi-361866da06631ad0020142e351fe64264-part2 S: disk/by-id/scsi-SDELL_PERC_H730_Mini_006442e61f352e142000ad3166a06d86-part2 S: disk/by-id/wwn-0x61866da06631ad0020142e351fe64264-part2 S: disk/by-partuuid/838bb658-02 S: disk/by-path/pci-0000:03:00.0-scsi-0:2:0:0-part2 E: DEVLINKS=/dev/disk/by-id/scsi-361866da06631ad0020142e351fe64264-part2 /dev/disk/by-partuuid/838bb658-02 /dev/disk/by-id/lvm-pv-uuid-pfuelt-KIYO-Ucin-Yvr0-Rjcw-BV2J-Dhe6Np /dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:0:0-part2 /dev/disk/by-id/wwn-0x61866da06631ad0020142e351fe64264-part2 /dev/disk/by-id/scsi-SDELL_PERC_H730_Mini_006442e61f352e142000ad3166a06d86-part2 E: DEVNAME=/dev/sda2 E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:03:00.0/host1/target1:2:0/1:2:0:0/block/sda/sda2 E: DEVTYPE=partition E: DM_MULTIPATH_DEVICE_PATH=0 E: ID_BUS=scsi E: ID_FS_TYPE=LVM2_member E: ID_FS_USAGE=raid E: ID_FS_UUID=pfuelt-KIYO-Ucin-Yvr0-Rjcw-BV2J-Dhe6Np E: ID_FS_UUID_ENC=pfuelt-KIYO-Ucin-Yvr0-Rjcw-BV2J-Dhe6Np E: ID_FS_VERSION=LVM2 001 E: ID_MODEL=PERC_H730_Mini E: ID_MODEL_ENC=PERC\x20H730\x20Mini\x20\x20 E: ID_PART_ENTRY_DISK=8:0 E: ID_PART_ENTRY_NUMBER=2 E: ID_PART_ENTRY_OFFSET=2099200 E: ID_PART_ENTRY_SCHEME=dos E: ID_PART_ENTRY_SIZE=1168898048 E: ID_PART_ENTRY_TYPE=0x8e E: ID_PART_ENTRY_UUID=838bb658-02 E: ID_PART_TABLE_TYPE=dos E: ID_PART_TABLE_UUID=838bb658 E: ID_PATH=pci-0000:03:00.0-scsi-0:2:0:0 E: ID_PATH_TAG=pci-0000_03_00_0-scsi-0_2_0_0 E: ID_REVISION=4.26 E: ID_SCSI=1 E: ID_SCSI_INQUIRY=1 E: ID_SCSI_SERIAL=006442e61f352e142000ad3166a06d86 E: ID_SERIAL=361866da06631ad0020142e351fe64264 E: ID_SERIAL_SHORT=61866da06631ad0020142e351fe64264 E: ID_TYPE=disk E: ID_VENDOR=DELL E: ID_VENDOR_ENC=DELL\x20\x20\x20\x20 E: ID_WWN=0x61866da06631ad00 E: ID_WWN_VENDOR_EXTENSION=0x20142e351fe64264 E: ID_WWN_WITH_EXTENSION=0x61866da06631ad0020142e351fe64264 E: MAJOR=8 E: MINOR=2 E: PARTN=2 E: SCSI_IDENT_LUN_NAA_REGEXT=61866da06631ad0020142e351fe64264 E: SCSI_IDENT_SERIAL=006442e61f352e142000ad3166a06d86 E: SCSI_MODEL=PERC_H730_Mini E: SCSI_MODEL_ENC=PERC\x20H730\x20Mini\x20\x20 E: SCSI_REVISION=4.26 E: SCSI_TPGS=0 E: SCSI_TYPE=disk E: SCSI_VENDOR=DELL E: SCSI_VENDOR_ENC=DELL\x20\x20\x20\x20 E: SUBSYSTEM=block E: SYSTEMD_ALIAS=/dev/block/8:2 E: SYSTEMD_READY=1 E: SYSTEMD_WANTS=lvm2-pvscan@8:2.service E: TAGS=:systemd: E: UDISKS_IGNORE=1 E: USEC_INITIALIZED=13804594
(In reply to Nir Soffer from comment #17) > (In reply to Qin Yuan from comment #12) > > Checked on the given UEFI machine: > > 1) it only has one local disk which is not multipath > > 2) after fresh installation, multipathd.service is inactive, and there is no > > /etc/multipath.conf file > > 3) after register to rhvm, multipathd.service is active, /etc/multipath.conf > > is generated > > 4) after upgrade from engine side, multipath claimed the device during > > reboot, the system enters to emergency mode. > > > > Pls check https://bugzilla.redhat.com/show_bug.cgi?id=1760223#c33 to see why > > multipath claims device after host upgrade. > > > > Adding the local disk to blacklist can solve this problem: > > # mkdir /etc/multipath/conf.d > > # vi /etc/multipath/conf.d/local.conf > > blacklist { > > wwid "3600508b1001ceaf040dfabf83d79ced4" > > } > > Did you rebuild initramfs? > > dracut --force --add multipath > > This is documented in multipath admin guide: > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/ > html-single/configuring_device_mapper_multipath/index#proc_multipath-setup- > in-initramfs-setting-up-dm-multipath I asked Qin, she did not rebuild initramfs, the blacklist was added before upgrade, imagebased will rebuild initramfs during upgrade.
Proposed short-term resolution for disabling lvm filter setup on imagedbased hosts seems to be rejected: https://gerrit.ovirt.org/#/c/110983/ If it is required to blacklist local devices where should it be done? IMHO if vdsm-tool should do that it should be on the multipath configuration step and not in the lvm filter step.
(In reply to Amit Bawer from comment #22) The patch was not rejected, we have one -1, but the maintainer of this project can merge the patch regardless. We fixed the regression introduced in 4.4.1, if the maintainers do not want to accept the fix, they will have to support the system with this regression. The flow should be something like: - find the host devices (code in lvmfilter.analize() does that) - If the host device is a multipath device we don't need a blacklist. - If the host device is not a multipath device, add multipath blacklist - create lvm filter for the host device So this looks like part of config-lvm-filter. There is one complication, if multipath is already configured and grabbed the local device, it will look like a multipath device, although the actual device is a local device. To avoid this, we can configure lvm filter before running "vdsm-tool configure" instead of after this step. This will not help in the case when multipath was already configured when we run config-lvm-filter, and it already grabbed the local device. In this case we need to: - detect that the device is a local device that should not be used by multipath (Ben should be able to help with this) - After installing the blacklist reconfigure multipath, and analyze the host again to find the host devices, this time they will not be multipath devices. I'm not sure we have a good way to detect if a local device grabbed by multipath should not be grabbed, so the best way is to black list the local device before multipath is configured.
(In reply to Nir Soffer from comment #23) > This will not help in the case when multipath was already configured when we > run config-lvm-filter, and it already grabbed the local device. In this case > we need to: > - detect that the device is a local device that should not be used by > multipath (Ben should be able to help with this) > - After installing the blacklist reconfigure multipath, and analyze the host > again to find the host devices, this time they will not be multipath > devices. > > I'm not sure we have a good way to detect if a local device grabbed by > multipath > should not be grabbed, so the best way is to black list the local device > before > multipath is configured. Ben, could you comment on how to detect if a multipath device is local one?
(In reply to Amit Bawer from comment #24) > Ben, could you comment on how to detect if a multipath device is local one? Unfortunately, there is no automated way that multipath can know if a device is a local device. That's why you need to configure it. Using commands like blkid and lsblk will give you information about the lablels and fs uuids on the devices, as well as what type of information is on the device (if it's a filesystem, or an LVM physical volume, for instance). However, like Nir said, if the device has already been claimed by multipath, instead of seeing what's on the device, the FSTYPE will simply tell you that it is a mpath_member. You would need to look at the multipath device itself, to see the actual information. If you are trying to match up scsi devices on the system with actual hardware devices, looking at the output of "udevadm info -n <device_name> -q all" will show all the information in the udev database for the device. If multipath is already using the device. You can get a lot of information about the device from multipathd show paths format <format_string> Where the format string is made up of path wildcards, which you can view with the command multipathd show wildcards For instance # multipathd show paths format "%d %w %s %P" will show the device name, wwid, vendor, it's product, and revision information, and the protocol used to talk to the device.
Some patches are not merged yet.
The bug can still be reproduced. Test version: RHVM: 4.4.2.6-0.2.el8ev RHVH: Upgrade rhvh from the 4.4.1 GA build (redhat-virtualization-host-4.4.1-20200722.0.el8_2) to the latest build (redhat-virtualization-host-4.4.2-20200915.0.el8_2) Test steps: Refer to Comment 13 Test results: After upgrade, the system enter emergency mode. Filter similar to `filter = ["a|^/dev/sda2$|", "r|.*|"]` can be seen from /etc/lvm/lvm.conf Will attach the host-deployment logs. What is the version of RHVM(or RHV) that solves this issue?
Re-add the needinfo of Comment 28
I think ansible cannot work for node, and the only way that will work is to run "vdsm-tool config-lvm-filter -y" from spec post upgrade scriptlet, in the same way we run "vdsm-tool configure --force". I'm not sure it we need a chroot, but we need to run the code from the upgraded layer, not from the current layer. This can be done by mounting the upgraded layer, and changing PYTHONPATH so vdsm-tool will run code from the mounted upgraded layer. During the run, vdsm-tool access lsblk and lvm to find mounted lvs, and then sysfs and udev database to find devices wwids. I don't think this can work in chroot. If we run the new code on the current layer, it will update lvm filter in /etc/lvm/lvm.conf and blacklist in /etc/multipath/conf.d/vdsm_blacklist.conf. If we copy these changes to the upgraded layer and rebuild initramfs, it should work fine after reboot. The new filter and blacklist will also be stored in the current layer, not sure if we want this.
It might be easier to just solve "which version of vdsm-tool gets invoked" the way Nir suggested in offline conversation before - in rhvh code, and then point to the right/alternative file location instead of /etc. Or even just copy it afterwards by rhvh-specific code.
Just sharing my thoughts here, @nsoffer is right, it wasn't not straight forward to run `vdsm-tool configure` in chroot and we must do so, otherwise the initrd that is created will break the system. To achieve this, we used a hidden, undocumented and rather ugly hack of systemd [1] to allow it to run. Not sure what config-lvm-filter does, but i'm pretty sure we should be able to mount the relevant filesystems to allow it to work in chroot right after `vdsm-tool configure` [1] https://github.com/systemd/systemd/blob/e66d2eeeeb4332ca94aeb62e95ec76f1f17ee9b7/src/basic/virt.c#L657
I don't see how we can avoid that. vdsm's posttrans is not any better, is it even executed? we're just switching one image with the other so it doesn't run at all, does it? nsoffer please elaborate on what else might be missing and is needed. I assume binaries are covered by chroot, /proc and /run are handled...anything else might be missing (that is not missing for vdsm-tool configure stuff which apparently works)?
Michal, that's exactly what I'm saying - we should be safe running it in osupdater right next to vdsm-tool configure, but we need to verify this. Ideally, we would love to run vdsm's %post directly but it's not trivial to achieve this inside chroot - I don't remember why, but I have tried it for sure - it works for some packages but not all of them, so we're only doing that for packages that execute selinux-related binaries in their %post section, which is probably wrong also... Bottom line, we need to re-implement the vdsm %post in the upgrade path, so if config-lvm-filter was added to vdsm's %post, we should add it also, and make sure that it works with chroot.
(In reply to Michal Skrivanek from comment #60) > nsoffer please elaborate on what else might be missing and is needed. I > assume binaries are covered by chroot, /proc and /run are handled...anything > else might be missing (that is not missing for vdsm-tool configure stuff > which apparently works)? I added more details in the patch: https://gerrit.ovirt.org/c/111338#message-5f81876b_96009b79 Lets tests this and see how it works.
Move bug status to "ASSIGNED" according to Comment 67
QE installed RHEL host on multipath and non-multipath machines to reproduce this bug and it has not been reproduced on the RHEL host. However, on the multipath machine, the filter does not change to a multipath id, but it has no effect on the system reboot. Test versions: 1. RHEL-8.2.0-20200404.0-x86_64-dvd1.iso 2. rhv-4.4.1-12 3. RHV 4.4.2-8 Test envoronment: 1. The non-multipath machine with only local disks 2. The multipath machine with one local disk and one FC storage Test steps: 1. Install RHEL 8.2(select Virtulization Host) 2. Setup repositories(rhel_821_host_x86.repo, rhv-4.4.1.repo) Steps refer to http://ci-web.eng.lab.tlv.redhat.com/docs/master/Guide/install_guide/index.html 3. Add host to RHVM Collect output of: - lsblk - lvm dumpconfig devices/filter - ls /etc/multipath/conf.d/vdsm_blacklist.conf (should not exist at this point) 4. Setup rhv-4.4.2-8 repositories 5. Upgrade host via RHVM Collect output of: - lsblk - lvm dumpconfig devices/filter - cat /etc/multipath/conf.d/vdsm_blacklist.conf 6. Check host status via RHVM Test results： 1. Test on the non-multipath machine passed. After upgrade, system reboot successfully. The output after adding host to RHVM ~~~~~~ # rpm -qa | grep vdsm vdsm-common-4.40.22-1.el8ev.noarch vdsm-4.40.22-1.el8ev.x86_64 vdsm-network-4.40.22-1.el8ev.x86_64 vdsm-api-4.40.22-1.el8ev.noarch vdsm-hook-ethtool-options-4.40.22-1.el8ev.noarch vdsm-yajsonrpc-4.40.22-1.el8ev.noarch vdsm-http-4.40.22-1.el8ev.noarch vdsm-hook-vhostmd-4.40.22-1.el8ev.noarch vdsm-client-4.40.22-1.el8ev.noarch vdsm-hook-openstacknet-4.40.22-1.el8ev.noarch vdsm-python-4.40.22-1.el8ev.noarch vdsm-hook-fcoe-4.40.22-1.el8ev.noarch vdsm-jsonrpc-4.40.22-1.el8ev.noarch vdsm-hook-vmfex-dev-4.40.22-1.el8ev.noarch # lvm dumpconfig devices/filter filter=["a|^/dev/sda2$|","r|.*|"] # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 558.4G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 557.4G 0 part ├─rhel-root 253:0 0 50G 0 lvm / ├─rhel-swap 253:1 0 15.7G 0 lvm [SWAP] └─rhel-home 253:2 0 491.7G 0 lvm /home sdc 8:32 1 119.2G 0 disk └─sdc1 8:33 1 119.2G 0 part # ls /etc/multipath/conf.d/vdsm_blacklist.conf ls: cannot access '/etc/multipath/conf.d/vdsm_blacklist.conf': No such file or directory ~~~~~~ The output after upgrade ~~~~~~ # rpm -qa | grep vdsm vdsm-api-4.40.26.3-1.el8ev.noarch vdsm-4.40.26.3-1.el8ev.x86_64 vdsm-network-4.40.26.3-1.el8ev.x86_64 vdsm-client-4.40.26.3-1.el8ev.noarch vdsm-common-4.40.26.3-1.el8ev.noarch vdsm-jsonrpc-4.40.26.3-1.el8ev.noarch vdsm-hook-fcoe-4.40.26.3-1.el8ev.noarch vdsm-yajsonrpc-4.40.26.3-1.el8ev.noarch vdsm-http-4.40.26.3-1.el8ev.noarch vdsm-hook-ethtool-options-4.40.26.3-1.el8ev.noarch vdsm-hook-vhostmd-4.40.26.3-1.el8ev.noarch vdsm-python-4.40.26.3-1.el8ev.noarch vdsm-hook-openstacknet-4.40.26.3-1.el8ev.noarch vdsm-hook-vmfex-dev-4.40.26.3-1.el8ev.noarch # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 558.4G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 557.4G 0 part ├─rhel-root 253:0 0 50G 0 lvm / ├─rhel-swap 253:1 0 15.7G 0 lvm [SWAP] └─rhel-home 253:2 0 491.7G 0 lvm /home sdc 8:32 1 119.2G 0 disk └─sdc1 8:33 1 119.2G 0 part # lvm dumpconfig devices/filter filter=["a|^/dev/disk/by-id/lvm-pv-uuid-y8JAjB-L0v0-ZYjA-mu21-Yv2o-ccTw-0sNwXc$|","r|.*|"] # cat /etc/multipath/conf.d/vdsm_blacklist.conf # This file is managed by vdsm, do not edit! # Any changes made to this file will be overwritten when running: # vdsm-tool config-lvm-filter blacklist { wwid "361866da06631ad0020142e351fe64264" } ~~~~~~ As you can see the local disk has not been changed to multipath disk id, after upgrade, the lvm filter changed to stable name (e.g. /dev/disk/by-id/lvm-pv-uuid-xxx) as expected. 2. Test on the multipath machine(install RHEL host on the FC storage only) also passed. After upgrade, system reboot successfully. But 1) It seems the local disk's name has also become a name like multipath. 2) The name of FC lun has changed when adding host to engine, but the filter is not changed accordingly. The output before adding host to RHVM ~~~~~~ # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 558.9G 0 disk sdb 8:16 0 300G 0 disk └─mpatha 253:0 0 300G 0 mpath ├─mpatha1 253:1 0 1G 0 part /boot └─mpatha2 253:2 0 299G 0 part ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home sdc 8:32 0 300G 0 disk └─mpatha 253:0 0 300G 0 mpath ├─mpatha1 253:1 0 1G 0 part /boot └─mpatha2 253:2 0 299G 0 part ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home ~~~~~~ The output after adding host to RHVM ~~~~~~ # rpm -qa | grep vdsm vdsm-jsonrpc-4.40.22-1.el8ev.noarch vdsm-hook-vmfex-dev-4.40.22-1.el8ev.noarch vdsm-api-4.40.22-1.el8ev.noarch vdsm-client-4.40.22-1.el8ev.noarch vdsm-yajsonrpc-4.40.22-1.el8ev.noarch vdsm-python-4.40.22-1.el8ev.noarch vdsm-4.40.22-1.el8ev.x86_64 vdsm-hook-fcoe-4.40.22-1.el8ev.noarch vdsm-hook-vhostmd-4.40.22-1.el8ev.noarch vdsm-network-4.40.22-1.el8ev.x86_64 vdsm-hook-ethtool-options-4.40.22-1.el8ev.noarch vdsm-hook-openstacknet-4.40.22-1.el8ev.noarch vdsm-http-4.40.22-1.el8ev.noarch vdsm-common-4.40.22-1.el8ev.noarch # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 558.9G 0 disk └─3600508b1001ccb4a1de53313beabbd82 253:6 0 558.9G 0 mpath sdb 8:16 0 300G 0 disk └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath ├─36005076300810b3e00000000000002a5p1 253:1 0 1G 0 part /boot └─36005076300810b3e00000000000002a5p2 253:2 0 299G 0 part ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home sdc 8:32 0 300G 0 disk └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath ├─36005076300810b3e00000000000002a5p1 253:1 0 1G 0 part /boot └─36005076300810b3e00000000000002a5p2 253:2 0 299G 0 part ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home # lvm dumpconfig devices/filter filter=["a|^/dev/mapper/mpatha2$|","r|.*|"] # ls /etc/multipath/conf.d/vdsm_blacklist.conf ls: cannot access '/etc/multipath/conf.d/vdsm_blacklist.conf': No such file or directory ~~~~~~ The output after upgrade ~~~~~~ # rpm -qa | grep vdsm vdsm-hook-vmfex-dev-4.40.26.3-1.el8ev.noarch vdsm-common-4.40.26.3-1.el8ev.noarch vdsm-hook-fcoe-4.40.26.3-1.el8ev.noarch vdsm-hook-vhostmd-4.40.26.3-1.el8ev.noarch vdsm-yajsonrpc-4.40.26.3-1.el8ev.noarch vdsm-http-4.40.26.3-1.el8ev.noarch vdsm-hook-ethtool-options-4.40.26.3-1.el8ev.noarch vdsm-api-4.40.26.3-1.el8ev.noarch vdsm-python-4.40.26.3-1.el8ev.noarch vdsm-4.40.26.3-1.el8ev.x86_64 vdsm-hook-openstacknet-4.40.26.3-1.el8ev.noarch vdsm-network-4.40.26.3-1.el8ev.x86_64 vdsm-client-4.40.26.3-1.el8ev.noarch vdsm-jsonrpc-4.40.26.3-1.el8ev.noarch # lvm dumpconfig devices/filter filter=["a|^/dev/mapper/mpatha2$|","r|.*|"] cat /etc/multipath/conf.d/vdsm_blacklist.conf cat: /etc/multipath/conf.d/vdsm_blacklist.conf: No such file or directory # multipath -ll 36005076300810b3e00000000000002a5 dm-0 IBM,2145 size=300G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 3:0:0:0 sdb 8:16 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 3:0:1:0 sdc 8:32 active ready running 3600508b1001ccb4a1de53313beabbd82 dm-6 HP,LOGICAL VOLUME size=559G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active `- 0:1:0:0 sda 8:0 active ready running ~~~~~~
(In reply to peyu from comment #84) > QE installed RHEL host on multipath and non-multipath machines to reproduce > this bug and it has not been reproduced on the RHEL host. > However, on the multipath machine, the filter does not change to a multipath > id, but it has no effect on the system reboot. > > > Test versions: > 1. RHEL-8.2.0-20200404.0-x86_64-dvd1.iso > 2. rhv-4.4.1-12 > 3. RHV 4.4.2-8 > > Test envoronment: > 1. The non-multipath machine with only local disks > 2. The multipath machine with one local disk and one FC storage > > Test steps: > 1. Install RHEL 8.2(select Virtulization Host) > 2. Setup repositories(rhel_821_host_x86.repo, rhv-4.4.1.repo) > Steps refer to > http://ci-web.eng.lab.tlv.redhat.com/docs/master/Guide/install_guide/index. > html > 3. Add host to RHVM > > Collect output of: > - lsblk > - lvm dumpconfig devices/filter > - ls /etc/multipath/conf.d/vdsm_blacklist.conf (should not exist at this > point) > > 4. Setup rhv-4.4.2-8 repositories > 5. Upgrade host via RHVM > > Collect output of: > - lsblk > - lvm dumpconfig devices/filter > - cat /etc/multipath/conf.d/vdsm_blacklist.conf > > 6. Check host status via RHVM > > > > Test results： > 1. Test on the non-multipath machine passed. After upgrade, system reboot > successfully. > > The output after adding host to RHVM > ~~~~~~ > # rpm -qa | grep vdsm > vdsm-common-4.40.22-1.el8ev.noarch > vdsm-4.40.22-1.el8ev.x86_64 > vdsm-network-4.40.22-1.el8ev.x86_64 > vdsm-api-4.40.22-1.el8ev.noarch > vdsm-hook-ethtool-options-4.40.22-1.el8ev.noarch > vdsm-yajsonrpc-4.40.22-1.el8ev.noarch > vdsm-http-4.40.22-1.el8ev.noarch > vdsm-hook-vhostmd-4.40.22-1.el8ev.noarch > vdsm-client-4.40.22-1.el8ev.noarch > vdsm-hook-openstacknet-4.40.22-1.el8ev.noarch > vdsm-python-4.40.22-1.el8ev.noarch > vdsm-hook-fcoe-4.40.22-1.el8ev.noarch > vdsm-jsonrpc-4.40.22-1.el8ev.noarch > vdsm-hook-vmfex-dev-4.40.22-1.el8ev.noarch > > # lvm dumpconfig devices/filter > filter=["a|^/dev/sda2$|","r|.*|"] > > # lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda 8:0 0 558.4G 0 disk > ├─sda1 8:1 0 1G 0 part /boot > └─sda2 8:2 0 557.4G 0 part > ├─rhel-root 253:0 0 50G 0 lvm / > ├─rhel-swap 253:1 0 15.7G 0 lvm [SWAP] > └─rhel-home 253:2 0 491.7G 0 lvm /home > sdc 8:32 1 119.2G 0 disk > └─sdc1 8:33 1 119.2G 0 part > > # ls /etc/multipath/conf.d/vdsm_blacklist.conf > ls: cannot access '/etc/multipath/conf.d/vdsm_blacklist.conf': No such file > or directory > ~~~~~~ > > The output after upgrade > ~~~~~~ > # rpm -qa | grep vdsm > vdsm-api-4.40.26.3-1.el8ev.noarch > vdsm-4.40.26.3-1.el8ev.x86_64 > vdsm-network-4.40.26.3-1.el8ev.x86_64 > vdsm-client-4.40.26.3-1.el8ev.noarch > vdsm-common-4.40.26.3-1.el8ev.noarch > vdsm-jsonrpc-4.40.26.3-1.el8ev.noarch > vdsm-hook-fcoe-4.40.26.3-1.el8ev.noarch > vdsm-yajsonrpc-4.40.26.3-1.el8ev.noarch > vdsm-http-4.40.26.3-1.el8ev.noarch > vdsm-hook-ethtool-options-4.40.26.3-1.el8ev.noarch > vdsm-hook-vhostmd-4.40.26.3-1.el8ev.noarch > vdsm-python-4.40.26.3-1.el8ev.noarch > vdsm-hook-openstacknet-4.40.26.3-1.el8ev.noarch > vdsm-hook-vmfex-dev-4.40.26.3-1.el8ev.noarch > > # lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda 8:0 0 558.4G 0 disk > ├─sda1 8:1 0 1G 0 part /boot > └─sda2 8:2 0 557.4G 0 part > ├─rhel-root 253:0 0 50G 0 lvm / > ├─rhel-swap 253:1 0 15.7G 0 lvm [SWAP] > └─rhel-home 253:2 0 491.7G 0 lvm /home > sdc 8:32 1 119.2G 0 disk > └─sdc1 8:33 1 119.2G 0 part > > # lvm dumpconfig devices/filter > filter=["a|^/dev/disk/by-id/lvm-pv-uuid-y8JAjB-L0v0-ZYjA-mu21-Yv2o-ccTw- > 0sNwXc$|","r|.*|"] > > # cat /etc/multipath/conf.d/vdsm_blacklist.conf > # This file is managed by vdsm, do not edit! > # Any changes made to this file will be overwritten when running: > # vdsm-tool config-lvm-filter > > blacklist { > wwid "361866da06631ad0020142e351fe64264" > } > ~~~~~~ > > As you can see the local disk has not been changed to multipath disk id, There are no mpath local disks on this setup so nothing would change from user-friendly "mpatha{x}" names to WWN. > after upgrade, the lvm filter changed to stable name (e.g. > /dev/disk/by-id/lvm-pv-uuid-xxx) as expected. > > > 2. Test on the multipath machine(install RHEL host on the FC storage only) > also passed. After upgrade, system reboot successfully. But > 1) It seems the local disk's name has also become a name like multipath. > 2) The name of FC lun has changed when adding host to engine, but the filter > is not changed accordingly. > > The output before adding host to RHVM > ~~~~~~ > # lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda 8:0 0 558.9G 0 disk > sdb 8:16 0 300G 0 disk > └─mpatha 253:0 0 300G 0 mpath > ├─mpatha1 253:1 0 1G 0 part /boot > └─mpatha2 253:2 0 299G 0 part > ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / > ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] > └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home > sdc 8:32 0 300G 0 disk > └─mpatha 253:0 0 300G 0 mpath > ├─mpatha1 253:1 0 1G 0 part /boot > └─mpatha2 253:2 0 299G 0 part > ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / > ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] > └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home > ~~~~~~ > > > The output after adding host to RHVM > ~~~~~~ > # rpm -qa | grep vdsm > vdsm-jsonrpc-4.40.22-1.el8ev.noarch > vdsm-hook-vmfex-dev-4.40.22-1.el8ev.noarch > vdsm-api-4.40.22-1.el8ev.noarch > vdsm-client-4.40.22-1.el8ev.noarch > vdsm-yajsonrpc-4.40.22-1.el8ev.noarch > vdsm-python-4.40.22-1.el8ev.noarch > vdsm-4.40.22-1.el8ev.x86_64 > vdsm-hook-fcoe-4.40.22-1.el8ev.noarch > vdsm-hook-vhostmd-4.40.22-1.el8ev.noarch > vdsm-network-4.40.22-1.el8ev.x86_64 > vdsm-hook-ethtool-options-4.40.22-1.el8ev.noarch > vdsm-hook-openstacknet-4.40.22-1.el8ev.noarch > vdsm-http-4.40.22-1.el8ev.noarch > vdsm-common-4.40.22-1.el8ev.noarch > > # lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda 8:0 0 558.9G 0 disk > └─3600508b1001ccb4a1de53313beabbd82 253:6 0 558.9G 0 mpath > sdb 8:16 0 300G 0 disk > └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath > ├─36005076300810b3e00000000000002a5p1 253:1 0 1G 0 part /boot > └─36005076300810b3e00000000000002a5p2 253:2 0 299G 0 part > ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / > ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] > └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home > sdc 8:32 0 300G 0 disk > └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath > ├─36005076300810b3e00000000000002a5p1 253:1 0 1G 0 part /boot > └─36005076300810b3e00000000000002a5p2 253:2 0 299G 0 part > ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / > ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] > └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home > > # lvm dumpconfig devices/filter > filter=["a|^/dev/mapper/mpatha2$|","r|.*|"] > > # ls /etc/multipath/conf.d/vdsm_blacklist.conf > ls: cannot access '/etc/multipath/conf.d/vdsm_blacklist.conf': No such file > or directory > > > ~~~~~~ > > The output after upgrade > ~~~~~~ > # rpm -qa | grep vdsm > vdsm-hook-vmfex-dev-4.40.26.3-1.el8ev.noarch > vdsm-common-4.40.26.3-1.el8ev.noarch > vdsm-hook-fcoe-4.40.26.3-1.el8ev.noarch > vdsm-hook-vhostmd-4.40.26.3-1.el8ev.noarch > vdsm-yajsonrpc-4.40.26.3-1.el8ev.noarch > vdsm-http-4.40.26.3-1.el8ev.noarch > vdsm-hook-ethtool-options-4.40.26.3-1.el8ev.noarch > vdsm-api-4.40.26.3-1.el8ev.noarch > vdsm-python-4.40.26.3-1.el8ev.noarch > vdsm-4.40.26.3-1.el8ev.x86_64 > vdsm-hook-openstacknet-4.40.26.3-1.el8ev.noarch > vdsm-network-4.40.26.3-1.el8ev.x86_64 > vdsm-client-4.40.26.3-1.el8ev.noarch > vdsm-jsonrpc-4.40.26.3-1.el8ev.noarch > > # lvm dumpconfig devices/filter > filter=["a|^/dev/mapper/mpatha2$|","r|.*|"] What RHVM version was used to upgrade from here? if its 4.4.2.6 or later (since it calls the lvm filter config as part of the ansible playbook for host-upgrade), then its possible that the lvmfilter fix for replacing unstable names from 4.4.1 with their stable names is missing a case where the unstable name refers to a non-existing device, like in /dev/mapper/mpatha2 seen here when friendly-names were already disabled for the multipath configuration. > > cat /etc/multipath/conf.d/vdsm_blacklist.conf > cat: /etc/multipath/conf.d/vdsm_blacklist.conf: No such file or directory > > # multipath -ll > 36005076300810b3e00000000000002a5 dm-0 IBM,2145 > size=300G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw > |-+- policy='service-time 0' prio=50 status=active > | `- 3:0:0:0 sdb 8:16 active ready running > `-+- policy='service-time 0' prio=10 status=enabled > `- 3:0:1:0 sdc 8:32 active ready running > 3600508b1001ccb4a1de53313beabbd82 dm-6 HP,LOGICAL VOLUME > size=559G features='1 queue_if_no_path' hwhandler='0' wp=rw > `-+- policy='service-time 0' prio=1 status=active > `- 0:1:0:0 sda 8:0 active ready running > ~~~~~~
(In reply to Amit Bawer from comment #85) > (In reply to peyu from comment #84) > > QE installed RHEL host on multipath and non-multipath machines to reproduce > > this bug and it has not been reproduced on the RHEL host. > > However, on the multipath machine, the filter does not change to a multipath > > id, but it has no effect on the system reboot. > > > > > > Test versions: > > 1. RHEL-8.2.0-20200404.0-x86_64-dvd1.iso > > 2. rhv-4.4.1-12 > > 3. RHV 4.4.2-8 > > > > Test envoronment: > > 1. The non-multipath machine with only local disks > > 2. The multipath machine with one local disk and one FC storage > > > > Test steps: > > 1. Install RHEL 8.2(select Virtulization Host) > > 2. Setup repositories(rhel_821_host_x86.repo, rhv-4.4.1.repo) > > Steps refer to > > http://ci-web.eng.lab.tlv.redhat.com/docs/master/Guide/install_guide/index. > > html > > 3. Add host to RHVM > > > > Collect output of: > > - lsblk > > - lvm dumpconfig devices/filter > > - ls /etc/multipath/conf.d/vdsm_blacklist.conf (should not exist at this > > point) > > > > 4. Setup rhv-4.4.2-8 repositories > > 5. Upgrade host via RHVM > > > > Collect output of: > > - lsblk > > - lvm dumpconfig devices/filter > > - cat /etc/multipath/conf.d/vdsm_blacklist.conf > > > > 6. Check host status via RHVM > > > > > > > > Test results： > > 1. Test on the non-multipath machine passed. After upgrade, system reboot > > successfully. > > > > The output after adding host to RHVM > > ~~~~~~ > > # rpm -qa | grep vdsm > > vdsm-common-4.40.22-1.el8ev.noarch > > vdsm-4.40.22-1.el8ev.x86_64 > > vdsm-network-4.40.22-1.el8ev.x86_64 > > vdsm-api-4.40.22-1.el8ev.noarch > > vdsm-hook-ethtool-options-4.40.22-1.el8ev.noarch > > vdsm-yajsonrpc-4.40.22-1.el8ev.noarch > > vdsm-http-4.40.22-1.el8ev.noarch > > vdsm-hook-vhostmd-4.40.22-1.el8ev.noarch > > vdsm-client-4.40.22-1.el8ev.noarch > > vdsm-hook-openstacknet-4.40.22-1.el8ev.noarch > > vdsm-python-4.40.22-1.el8ev.noarch > > vdsm-hook-fcoe-4.40.22-1.el8ev.noarch > > vdsm-jsonrpc-4.40.22-1.el8ev.noarch > > vdsm-hook-vmfex-dev-4.40.22-1.el8ev.noarch > > > > # lvm dumpconfig devices/filter > > filter=["a|^/dev/sda2$|","r|.*|"] > > > > # lsblk > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > sda 8:0 0 558.4G 0 disk > > ├─sda1 8:1 0 1G 0 part /boot > > └─sda2 8:2 0 557.4G 0 part > > ├─rhel-root 253:0 0 50G 0 lvm / > > ├─rhel-swap 253:1 0 15.7G 0 lvm [SWAP] > > └─rhel-home 253:2 0 491.7G 0 lvm /home > > sdc 8:32 1 119.2G 0 disk > > └─sdc1 8:33 1 119.2G 0 part > > > > # ls /etc/multipath/conf.d/vdsm_blacklist.conf > > ls: cannot access '/etc/multipath/conf.d/vdsm_blacklist.conf': No such file > > or directory > > ~~~~~~ > > > > The output after upgrade > > ~~~~~~ > > # rpm -qa | grep vdsm > > vdsm-api-4.40.26.3-1.el8ev.noarch > > vdsm-4.40.26.3-1.el8ev.x86_64 > > vdsm-network-4.40.26.3-1.el8ev.x86_64 > > vdsm-client-4.40.26.3-1.el8ev.noarch > > vdsm-common-4.40.26.3-1.el8ev.noarch > > vdsm-jsonrpc-4.40.26.3-1.el8ev.noarch > > vdsm-hook-fcoe-4.40.26.3-1.el8ev.noarch > > vdsm-yajsonrpc-4.40.26.3-1.el8ev.noarch > > vdsm-http-4.40.26.3-1.el8ev.noarch > > vdsm-hook-ethtool-options-4.40.26.3-1.el8ev.noarch > > vdsm-hook-vhostmd-4.40.26.3-1.el8ev.noarch > > vdsm-python-4.40.26.3-1.el8ev.noarch > > vdsm-hook-openstacknet-4.40.26.3-1.el8ev.noarch > > vdsm-hook-vmfex-dev-4.40.26.3-1.el8ev.noarch > > > > # lsblk > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > sda 8:0 0 558.4G 0 disk > > ├─sda1 8:1 0 1G 0 part /boot > > └─sda2 8:2 0 557.4G 0 part > > ├─rhel-root 253:0 0 50G 0 lvm / > > ├─rhel-swap 253:1 0 15.7G 0 lvm [SWAP] > > └─rhel-home 253:2 0 491.7G 0 lvm /home > > sdc 8:32 1 119.2G 0 disk > > └─sdc1 8:33 1 119.2G 0 part > > > > # lvm dumpconfig devices/filter > > filter=["a|^/dev/disk/by-id/lvm-pv-uuid-y8JAjB-L0v0-ZYjA-mu21-Yv2o-ccTw- > > 0sNwXc$|","r|.*|"] > > > > # cat /etc/multipath/conf.d/vdsm_blacklist.conf > > # This file is managed by vdsm, do not edit! > > # Any changes made to this file will be overwritten when running: > > # vdsm-tool config-lvm-filter > > > > blacklist { > > wwid "361866da06631ad0020142e351fe64264" > > } > > ~~~~~~ > > > > As you can see the local disk has not been changed to multipath disk id, > > There are no mpath local disks on this setup so nothing would change from > user-friendly "mpatha{x}" names to WWN. > > > after upgrade, the lvm filter changed to stable name (e.g. > > /dev/disk/by-id/lvm-pv-uuid-xxx) as expected. > > > > > > 2. Test on the multipath machine(install RHEL host on the FC storage only) > > also passed. After upgrade, system reboot successfully. But > > 1) It seems the local disk's name has also become a name like multipath. > > 2) The name of FC lun has changed when adding host to engine, but the filter > > is not changed accordingly. > > > > The output before adding host to RHVM > > ~~~~~~ > > # lsblk > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > sda 8:0 0 558.9G 0 disk > > sdb 8:16 0 300G 0 disk > > └─mpatha 253:0 0 300G 0 mpath > > ├─mpatha1 253:1 0 1G 0 part /boot > > └─mpatha2 253:2 0 299G 0 part > > ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / > > ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] > > └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home > > sdc 8:32 0 300G 0 disk > > └─mpatha 253:0 0 300G 0 mpath > > ├─mpatha1 253:1 0 1G 0 part /boot > > └─mpatha2 253:2 0 299G 0 part > > ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / > > ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] > > └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home > > ~~~~~~ > > > > > > The output after adding host to RHVM > > ~~~~~~ > > # rpm -qa | grep vdsm > > vdsm-jsonrpc-4.40.22-1.el8ev.noarch > > vdsm-hook-vmfex-dev-4.40.22-1.el8ev.noarch > > vdsm-api-4.40.22-1.el8ev.noarch > > vdsm-client-4.40.22-1.el8ev.noarch > > vdsm-yajsonrpc-4.40.22-1.el8ev.noarch > > vdsm-python-4.40.22-1.el8ev.noarch > > vdsm-4.40.22-1.el8ev.x86_64 > > vdsm-hook-fcoe-4.40.22-1.el8ev.noarch > > vdsm-hook-vhostmd-4.40.22-1.el8ev.noarch > > vdsm-network-4.40.22-1.el8ev.x86_64 > > vdsm-hook-ethtool-options-4.40.22-1.el8ev.noarch > > vdsm-hook-openstacknet-4.40.22-1.el8ev.noarch > > vdsm-http-4.40.22-1.el8ev.noarch > > vdsm-common-4.40.22-1.el8ev.noarch > > > > # lsblk > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > sda 8:0 0 558.9G 0 disk > > └─3600508b1001ccb4a1de53313beabbd82 253:6 0 558.9G 0 mpath > > sdb 8:16 0 300G 0 disk > > └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath > > ├─36005076300810b3e00000000000002a5p1 253:1 0 1G 0 part /boot > > └─36005076300810b3e00000000000002a5p2 253:2 0 299G 0 part > > ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / > > ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] > > └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home > > sdc 8:32 0 300G 0 disk > > └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath > > ├─36005076300810b3e00000000000002a5p1 253:1 0 1G 0 part /boot > > └─36005076300810b3e00000000000002a5p2 253:2 0 299G 0 part > > ├─rhel_hp--dl385g8--03-root 253:3 0 50G 0 lvm / > > ├─rhel_hp--dl385g8--03-swap 253:4 0 30G 0 lvm [SWAP] > > └─rhel_hp--dl385g8--03-home 253:5 0 219G 0 lvm /home > > > > # lvm dumpconfig devices/filter > > filter=["a|^/dev/mapper/mpatha2$|","r|.*|"] > > > > # ls /etc/multipath/conf.d/vdsm_blacklist.conf > > ls: cannot access '/etc/multipath/conf.d/vdsm_blacklist.conf': No such file > > or directory > > > > > > ~~~~~~ > > > > The output after upgrade > > ~~~~~~ > > # rpm -qa | grep vdsm > > vdsm-hook-vmfex-dev-4.40.26.3-1.el8ev.noarch > > vdsm-common-4.40.26.3-1.el8ev.noarch > > vdsm-hook-fcoe-4.40.26.3-1.el8ev.noarch > > vdsm-hook-vhostmd-4.40.26.3-1.el8ev.noarch > > vdsm-yajsonrpc-4.40.26.3-1.el8ev.noarch > > vdsm-http-4.40.26.3-1.el8ev.noarch > > vdsm-hook-ethtool-options-4.40.26.3-1.el8ev.noarch > > vdsm-api-4.40.26.3-1.el8ev.noarch > > vdsm-python-4.40.26.3-1.el8ev.noarch > > vdsm-4.40.26.3-1.el8ev.x86_64 > > vdsm-hook-openstacknet-4.40.26.3-1.el8ev.noarch > > vdsm-network-4.40.26.3-1.el8ev.x86_64 > > vdsm-client-4.40.26.3-1.el8ev.noarch > > vdsm-jsonrpc-4.40.26.3-1.el8ev.noarch > > > > # lvm dumpconfig devices/filter > > filter=["a|^/dev/mapper/mpatha2$|","r|.*|"] > > What RHVM version was used to upgrade from here? > if its 4.4.2.6 or later (since it calls the lvm filter config as part of the > ansible playbook for host-upgrade), > then its possible that the lvmfilter fix for replacing unstable names from > 4.4.1 with their stable names is missing > a case where the unstable name refers to a non-existing device, like in > /dev/mapper/mpatha2 seen here when friendly-names > were already disabled for the multipath configuration. Could you share the deploy and upgrade logs from this test?
Continue to Comment 84 Test version: RHVM：4.4.2.6-0.2.el8ev 3. Test on the multipath machine(install RHEL host on the local disk only) passed. The output after adding host to RHVM ~~~~~~ # rpm -qa | grep vdsm vdsm-hook-vmfex-dev-4.40.22-1.el8ev.noarch vdsm-hook-vhostmd-4.40.22-1.el8ev.noarch vdsm-client-4.40.22-1.el8ev.noarch vdsm-hook-openstacknet-4.40.22-1.el8ev.noarch vdsm-http-4.40.22-1.el8ev.noarch vdsm-common-4.40.22-1.el8ev.noarch vdsm-yajsonrpc-4.40.22-1.el8ev.noarch vdsm-4.40.22-1.el8ev.x86_64 vdsm-hook-fcoe-4.40.22-1.el8ev.noarch vdsm-network-4.40.22-1.el8ev.x86_64 vdsm-jsonrpc-4.40.22-1.el8ev.noarch vdsm-hook-ethtool-options-4.40.22-1.el8ev.noarch vdsm-python-4.40.22-1.el8ev.noarch vdsm-api-4.40.22-1.el8ev.noarch # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 558.9G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 557.9G 0 part ├─rhel-root 253:1 0 50G 0 lvm / ├─rhel-swap 253:4 0 31.5G 0 lvm [SWAP] └─rhel-home 253:5 0 476.4G 0 lvm /home sdb 8:16 0 300G 0 disk └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath ├─36005076300810b3e00000000000002a5p1 253:2 0 1G 0 part └─36005076300810b3e00000000000002a5p2 253:3 0 299G 0 part sdc 8:32 0 300G 0 disk └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath ├─36005076300810b3e00000000000002a5p1 253:2 0 1G 0 part └─36005076300810b3e00000000000002a5p2 253:3 0 299G 0 part # lvm dumpconfig devices/filter filter=["a|^/dev/sda2$|","r|.*|"] ls /etc/multipath/conf.d/vdsm_blacklist.conf ls: cannot access '/etc/multipath/conf.d/vdsm_blacklist.conf': No such file or directory ~~~~~~ The output after upgrade ~~~~~~ # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 558.9G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 557.9G 0 part ├─rhel-root 253:1 0 50G 0 lvm / ├─rhel-swap 253:4 0 31.5G 0 lvm [SWAP] └─rhel-home 253:5 0 476.4G 0 lvm /home sdb 8:16 0 300G 0 disk └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath ├─36005076300810b3e00000000000002a5p1 253:2 0 1G 0 part └─36005076300810b3e00000000000002a5p2 253:3 0 299G 0 part sdc 8:32 0 300G 0 disk └─36005076300810b3e00000000000002a5 253:0 0 300G 0 mpath ├─36005076300810b3e00000000000002a5p1 253:2 0 1G 0 part └─36005076300810b3e00000000000002a5p2 253:3 0 299G 0 part # lvm dumpconfig devices/filter filter=["a|^/dev/disk/by-id/lvm-pv-uuid-XLrnW0-Cas1-JRMB-RMfj-aLH8-1h32-QwnuJu$|","r|.*|"] # cat /etc/multipath/conf.d/vdsm_blacklist.conf # This file is managed by vdsm, do not edit! # Any changes made to this file will be overwritten when running: # vdsm-tool config-lvm-filter blacklist { wwid "3600508b1001ccb4a1de53313beabbd82" } ~~~~~~ 4. Test on the multipath machine(select both local disk and FC storage, when installing RHEL host) passed. The output after adding host to RHVM ~~~~~~ # rpm -qa | grep vdsm vdsm-jsonrpc-4.40.22-1.el8ev.noarch vdsm-hook-vmfex-dev-4.40.22-1.el8ev.noarch vdsm-api-4.40.22-1.el8ev.noarch vdsm-client-4.40.22-1.el8ev.noarch vdsm-yajsonrpc-4.40.22-1.el8ev.noarch vdsm-python-4.40.22-1.el8ev.noarch vdsm-4.40.22-1.el8ev.x86_64 vdsm-hook-fcoe-4.40.22-1.el8ev.noarch vdsm-hook-vhostmd-4.40.22-1.el8ev.noarch vdsm-network-4.40.22-1.el8ev.x86_64 vdsm-hook-ethtool-options-4.40.22-1.el8ev.noarch vdsm-hook-openstacknet-4.40.22-1.el8ev.noarch vdsm-http-4.40.22-1.el8ev.noarch vdsm-common-4.40.22-1.el8ev.noarch # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 557.9G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 556.9G 0 part ├─rhel-root 253:1 0 50G 0 lvm / └─rhel-home 253:4 0 1.5T 0 lvm /home sdb 8:16 0 1T 0 disk └─36005076300810b3e00000000000002b6 253:0 0 1T 0 mpath └─36005076300810b3e00000000000002b6p1 253:2 0 1024G 0 part ├─rhel-swap 253:3 0 4G 0 lvm [SWAP] └─rhel-home 253:4 0 1.5T 0 lvm /home sdc 8:32 0 1T 0 disk └─36005076300810b3e00000000000002b6 253:0 0 1T 0 mpath └─36005076300810b3e00000000000002b6p1 253:2 0 1024G 0 part ├─rhel-swap 253:3 0 4G 0 lvm [SWAP] └─rhel-home 253:4 0 1.5T 0 lvm /home # lvmconfig | grep filter filter=["a|^/dev/mapper/mpatha1$|","a|^/dev/sda2$|","r|.*|"] ~~~~~~ The output after upgrade ~~~~~~ # rpm -qa | grep vdsm vdsm-hook-vmfex-dev-4.40.26.3-1.el8ev.noarch vdsm-common-4.40.26.3-1.el8ev.noarch vdsm-hook-fcoe-4.40.26.3-1.el8ev.noarch vdsm-hook-vhostmd-4.40.26.3-1.el8ev.noarch vdsm-yajsonrpc-4.40.26.3-1.el8ev.noarch vdsm-http-4.40.26.3-1.el8ev.noarch vdsm-hook-ethtool-options-4.40.26.3-1.el8ev.noarch vdsm-api-4.40.26.3-1.el8ev.noarch vdsm-python-4.40.26.3-1.el8ev.noarch vdsm-4.40.26.3-1.el8ev.x86_64 vdsm-hook-openstacknet-4.40.26.3-1.el8ev.noarch vdsm-network-4.40.26.3-1.el8ev.x86_64 vdsm-client-4.40.26.3-1.el8ev.noarch vdsm-jsonrpc-4.40.26.3-1.el8ev.noarch # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 557.9G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 556.9G 0 part ├─rhel-root 253:1 0 50G 0 lvm / └─rhel-home 253:4 0 1.5T 0 lvm /home sdb 8:16 0 1T 0 disk └─36005076300810b3e00000000000002b6 253:0 0 1T 0 mpath └─36005076300810b3e00000000000002b6p1 253:2 0 1024G 0 part ├─rhel-swap 253:3 0 4G 0 lvm [SWAP] └─rhel-home 253:4 0 1.5T 0 lvm /home sdc 8:32 0 1T 0 disk └─36005076300810b3e00000000000002b6 253:0 0 1T 0 mpath └─36005076300810b3e00000000000002b6p1 253:2 0 1024G 0 part ├─rhel-swap 253:3 0 4G 0 lvm [SWAP] └─rhel-home 253:4 0 1.5T 0 lvm /home # lvmconfig | grep filter filter=["a|^/dev/mapper/mpatha1$|","a|^/dev/sda2$|","r|.*|"] ~~~~~~
Created attachment 1715986 [details] RHEL host log
Created attachment 1715989 [details] host deploy logs, generated on Sep22,23 is related to RHEL host
(In reply to peyu from comment #87) > 4. Test on the multipath machine(select both local disk and FC storage, when > installing RHEL host) passed. > > The output after adding host to RHVM > ~~~~~~ > # rpm -qa | grep vdsm > vdsm-jsonrpc-4.40.22-1.el8ev.noarch > vdsm-hook-vmfex-dev-4.40.22-1.el8ev.noarch > vdsm-api-4.40.22-1.el8ev.noarch > vdsm-client-4.40.22-1.el8ev.noarch > vdsm-yajsonrpc-4.40.22-1.el8ev.noarch > vdsm-python-4.40.22-1.el8ev.noarch > vdsm-4.40.22-1.el8ev.x86_64 > vdsm-hook-fcoe-4.40.22-1.el8ev.noarch > vdsm-hook-vhostmd-4.40.22-1.el8ev.noarch > vdsm-network-4.40.22-1.el8ev.x86_64 > vdsm-hook-ethtool-options-4.40.22-1.el8ev.noarch > vdsm-hook-openstacknet-4.40.22-1.el8ev.noarch > vdsm-http-4.40.22-1.el8ev.noarch > vdsm-common-4.40.22-1.el8ev.noarch > > # lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda 8:0 0 557.9G 0 disk > ├─sda1 8:1 0 1G 0 part /boot > └─sda2 8:2 0 556.9G 0 part > ├─rhel-root 253:1 0 50G 0 lvm / > └─rhel-home 253:4 0 1.5T 0 lvm /home > sdb 8:16 0 1T 0 disk > └─36005076300810b3e00000000000002b6 253:0 0 1T 0 mpath > └─36005076300810b3e00000000000002b6p1 253:2 0 1024G 0 part > ├─rhel-swap 253:3 0 4G 0 lvm [SWAP] > └─rhel-home 253:4 0 1.5T 0 lvm /home > sdc 8:32 0 1T 0 disk > └─36005076300810b3e00000000000002b6 253:0 0 1T 0 mpath > └─36005076300810b3e00000000000002b6p1 253:2 0 1024G 0 part > ├─rhel-swap 253:3 0 4G 0 lvm [SWAP] > └─rhel-home 253:4 0 1.5T 0 lvm /home > > > # lvmconfig | grep filter > filter=["a|^/dev/mapper/mpatha1$|","a|^/dev/sda2$|","r|.*|"] > ~~~~~~ > > The output after upgrade > ~~~~~~ > # rpm -qa | grep vdsm > vdsm-hook-vmfex-dev-4.40.26.3-1.el8ev.noarch > vdsm-common-4.40.26.3-1.el8ev.noarch > vdsm-hook-fcoe-4.40.26.3-1.el8ev.noarch > vdsm-hook-vhostmd-4.40.26.3-1.el8ev.noarch > vdsm-yajsonrpc-4.40.26.3-1.el8ev.noarch > vdsm-http-4.40.26.3-1.el8ev.noarch > vdsm-hook-ethtool-options-4.40.26.3-1.el8ev.noarch > vdsm-api-4.40.26.3-1.el8ev.noarch > vdsm-python-4.40.26.3-1.el8ev.noarch > vdsm-4.40.26.3-1.el8ev.x86_64 > vdsm-hook-openstacknet-4.40.26.3-1.el8ev.noarch > vdsm-network-4.40.26.3-1.el8ev.x86_64 > vdsm-client-4.40.26.3-1.el8ev.noarch > vdsm-jsonrpc-4.40.26.3-1.el8ev.noarch > > # lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda 8:0 0 557.9G 0 disk > ├─sda1 8:1 0 1G 0 part /boot > └─sda2 8:2 0 556.9G 0 part > ├─rhel-root 253:1 0 50G 0 lvm / > └─rhel-home 253:4 0 1.5T 0 lvm /home > sdb 8:16 0 1T 0 disk > └─36005076300810b3e00000000000002b6 253:0 0 1T 0 mpath > └─36005076300810b3e00000000000002b6p1 253:2 0 1024G 0 part > ├─rhel-swap 253:3 0 4G 0 lvm [SWAP] > └─rhel-home 253:4 0 1.5T 0 lvm /home > sdc 8:32 0 1T 0 disk > └─36005076300810b3e00000000000002b6 253:0 0 1T 0 mpath > └─36005076300810b3e00000000000002b6p1 253:2 0 1024G 0 part > ├─rhel-swap 253:3 0 4G 0 lvm [SWAP] > └─rhel-home 253:4 0 1.5T 0 lvm /home > > # lvmconfig | grep filter > filter=["a|^/dev/mapper/mpatha1$|","a|^/dev/sda2$|","r|.*|"] > ~~~~~~ For the last test case, config-lvm-filter call on upgrade will only warn with recommended configuration and will not reconfigure the filter in-place since "/dev/mapper/mpatha1" has no match in the devlinks for its pv-uuid link (it is already unreferenced at this stage), probably expected. Taken from attachment 1715989 [details]/ovirt-host-mgmt-ansible-20200923082559-10.73.130.127-d304a1f3-e8d7-48a7-b5e3-5e25bcd4fb4c.log 2020-09-23 08:31:18 CST - TASK [ovirt-host-upgrade : Configure LVM filter] ******************************* 2020-09-23 08:31:18 CST - fatal: [10.73.130.127]: FAILED! => {"changed": true, "cmd": ["vdsm-tool", "config-lvm-filter", "-y"], "delta": "0:00:00.736812", "end": "2020-09-22 12:31:35.606541", "msg": "non-zero return code", "rc": 3, "start": "2020-09-22 12:31:34.869729", "stderr": "", "stderr_lines": [], "stdout": "Analyzing host...\nFound these mounted logical volumes on this host:\n\n logical volume: /dev/mapper/rhel-home\n mountpoint: /home\n devices: /dev/disk/by-id/lvm-pv-uuid-2oJ0N3-AIz8-yrv2-Bswv-JDo1-gJEa-2t6gS0, /dev/disk/by-id/lvm-pv-uuid-EfvfHD-KdCh-H0Dn-f1SH-Gu7e-we4O-4WNJ6p\n\n logical volume: /dev/mapper/rhel-root\n mountpoint: /\n devices: /dev/disk/by-id/lvm-pv-uuid-2oJ0N3-AIz8-yrv2-Bswv-JDo1-gJEa-2t6gS0, /dev/disk/by-id/lvm-pv-uuid-EfvfHD-KdCh-H0Dn-f1SH-Gu7e-we4O-4WNJ6p\n\n logical volume: /dev/mapper/rhel-swap\n mountpoint: [SWAP]\n devices: /dev/disk/by-id/lvm-pv-uuid-2oJ0N3-AIz8-yrv2-Bswv-JDo1-gJEa-2t6gS0, /dev/disk/by-id/lvm-pv-uuid-EfvfHD-KdCh-H0Dn-f1SH-Gu7e-we4O-4WNJ6p\n\nThis is the recommended LVM filter for this host:\n\n filter = [ \"a|^/dev/disk/by-id/lvm-pv-uuid-2oJ0N3-AIz8-yrv2-Bswv-JDo1-gJEa-2t6gS0$|\", \"a|^/dev/disk/by-id/lvm-pv-uuid-EfvfHD-KdCh-H0Dn-f1SH-Gu7e-we4O-4WNJ6p$|\", \"r|.*|\" ]\n\nThis filter allows LVM to access the local devices used by the\nhypervisor, but not shared storage owned by Vdsm. If you add a new\ndevice to the volume group, you will need to edit the filter manually.\n\nThis is the current LVM filter:\n\n filter = [ \"a|^/dev/mapper/mpatha1$|\", \"a|^/dev/sda2$|\", \"r|.*|\" ]\n\nTo use the recommended filter we need to add multipath\nblacklist in /etc/multipath/conf.d/vdsm_blacklist.conf:\n\n blacklist {\n wwid \"3600605b00e6c476023918a0360fdff86\"\n }\n\n\nWARNING: The current LVM filter does not match the recommended filter,\nVdsm cannot configure the filter automatically.\n\nPlease edit /etc/lvm/lvm.conf and set the 'filter' option in the\n'devices' section to the recommended value.\n\nMake sure /etc/multipath/conf.d/vdsm_blacklist.conf is set with the\nrecommended 'blacklist' section.\n\nIt is recommended to reboot to verify the new configuration.", "stdout_lines": ["Analyzing host...", "Found these mounted logical volumes on this host:", "", " logical volume: /dev/mapper/rhel-home", " mountpoint: /home", " devices: /dev/disk/by-id/lvm-pv-uuid-2oJ0N3-AIz8-yrv2-Bswv-JDo1-gJEa-2t6gS0, /dev/disk/by-id/lvm-pv-uuid-EfvfHD-KdCh-H0Dn-f1SH-Gu7e-we4O-4WNJ6p", "", " logical volume: /dev/mapper/rhel-root", " mountpoint: /", " devices: /dev/disk/by-id/lvm-pv-uuid-2oJ0N3-AIz8-yrv2-Bswv-JDo1-gJEa-2t6gS0, /dev/disk/by-id/lvm-pv-uuid-EfvfHD-KdCh-H0Dn-f1SH-Gu7e-we4O-4WNJ6p", "", " logical volume: /dev/mapper/rhel-swap", " mountpoint: [SWAP]", " devices: /dev/disk/by-id/lvm-pv-uuid-2oJ0N3-AIz8-yrv2-Bswv-JDo1-gJEa-2t6gS0, /dev/disk/by-id/lvm-pv-uuid-EfvfHD-KdCh-H0Dn-f1SH-Gu7e-we4O-4WNJ6p", "", "This is the recommended LVM filter for this host:", "", " filter = [ \"a|^/dev/disk/by-id/lvm-pv-uuid-2oJ0N3-AIz8-yrv2-Bswv-JDo1-gJEa-2t6gS0$|\", \"a|^/dev/disk/by-id/lvm-pv-uuid-EfvfHD-KdCh-H0Dn-f1SH-Gu7e-we4O-4WNJ6p$|\", \"r|.*|\" ]", "", "This filter allows LVM to access the local devices used by the", "hypervisor, but not shared storage owned by Vdsm. If you add a new", "device to the volume group, you will need to edit the filter manually.", "", "This is the current LVM filter:", "", " filter = [ \"a|^/dev/mapper/mpatha1$|\", \"a|^/dev/sda2$|\", \"r|.*|\" ]", "", "To use the recommended filter we need to add multipath", "blacklist in /etc/multipath/conf.d/vdsm_blacklist.conf:", "", " blacklist {", " wwid \"3600605b00e6c476023918a0360fdff86\"", " }", "", "", "WARNING: The current LVM filter does not match the recommended filter,", "Vdsm cannot configure the filter automatically.", "", "Please edit /etc/lvm/lvm.conf and set the 'filter' option in the", "'devices' section to the recommended value.", "", "Make sure /etc/multipath/conf.d/vdsm_blacklist.conf is set with the", "recommended 'blacklist' section.", "", "It is recommended to reboot to verify the new configuration."]
Here is the KCS: https://access.redhat.com/solutions/5428651. However, I suspect a simpler filter will do as well for the workaround in the KCS: filter = [ "a|.*|" ]
(In reply to Marina Kalinin from comment #91) > Here is the KCS: https://access.redhat.com/solutions/5428651. > However, I suspect a simpler filter will do as well for the workaround in > the KCS: > filter = [ "a|.*|" ] Thanks, using filter = [ "a|.*|" ] will not get it replaced by vdsm-tool config-lvm-filter -y since its too generic: # grep filter /etc/lvm/lvm.conf | grep -v \# filter = ["a|.*|"] # vdsm-tool config-lvm-filter -y Analyzing host... Found these mounted logical volumes on this host: logical volume: /dev/mapper/cl-root mountpoint: / devices: /dev/disk/by-id/lvm-pv-uuid-DXgufc-7riC-TqhU-f8yH-EfZt-ivvH-TVcnEQ logical volume: /dev/mapper/cl-swap mountpoint: [SWAP] devices: /dev/disk/by-id/lvm-pv-uuid-DXgufc-7riC-TqhU-f8yH-EfZt-ivvH-TVcnEQ This is the recommended LVM filter for this host: filter = [ "a|^/dev/disk/by-id/lvm-pv-uuid-DXgufc-7riC-TqhU-f8yH-EfZt-ivvH-TVcnEQ$|", "r|.*|" ] This filter allows LVM to access the local devices used by the hypervisor, but not shared storage owned by Vdsm. If you add a new device to the volume group, you will need to edit the filter manually. This is the current LVM filter: filter = [ "a|.*|" ] To use the recommended filter we need to add multipath blacklist in /etc/multipath/conf.d/vdsm_blacklist.conf: blacklist { wwid "QEMU_HARDDISK_QM00003" } WARNING: The current LVM filter does not match the recommended filter, Vdsm cannot configure the filter automatically. Please edit /etc/lvm/lvm.conf and set the 'filter' option in the 'devices' section to the recommended value. Make sure /etc/multipath/conf.d/vdsm_blacklist.conf is set with the recommended 'blacklist' section. It is recommended to reboot to verify the new configuration. For the KCS we can choose no-filter, i.e. commenting out the stale filter line in /etc/lvm/lvm.conf so config lvm filter will be able to replace it after it: # grep filter /etc/lvm/lvm.conf | grep -v \# (no filter) # vdsm-tool config-lvm-filter -y Analyzing host... Found these mounted logical volumes on this host: logical volume: /dev/mapper/cl-root mountpoint: / devices: /dev/disk/by-id/lvm-pv-uuid-DXgufc-7riC-TqhU-f8yH-EfZt-ivvH-TVcnEQ logical volume: /dev/mapper/cl-swap mountpoint: [SWAP] devices: /dev/disk/by-id/lvm-pv-uuid-DXgufc-7riC-TqhU-f8yH-EfZt-ivvH-TVcnEQ This is the recommended LVM filter for this host: filter = [ "a|^/dev/disk/by-id/lvm-pv-uuid-DXgufc-7riC-TqhU-f8yH-EfZt-ivvH-TVcnEQ$|", "r|.*|" ] This filter allows LVM to access the local devices used by the hypervisor, but not shared storage owned by Vdsm. If you add a new device to the volume group, you will need to edit the filter manually. To use the recommended filter we need to add multipath blacklist in /etc/multipath/conf.d/vdsm_blacklist.conf: blacklist { wwid "QEMU_HARDDISK_QM00003" } Configuration completed successfully! Please reboot to verify the configuration.
Doc text looks good to me. Thank you, Eli. For the KCS, we are still having a discussion over email. I will update it once we get to consensus. But generally speaking the KCS is already good enough.
I'm not sure if it is still relevant but I hit this issue on RHEL 8.2.1 based host which was installed from CDN via subscription) in version 4.4.1 during upgrade to latest 4.4.2 (tested it yesterday) My steps: 1) I installed RHEL 8.2.1 (from ISO) on a device which became a multipath device. - For installing an OS on a multipath device, it's better to have a single path device, install OS and then extend the number of paths to that device. I followed a documentation how it should be done [1] 2) I registered the machine with CDN and enable necessary repositories for 4.4.1 host 3) I added 4.4.1 host into 4.4.1 engine which was also installed from CDN 4) I checked via SSH Management in webadmin that the host could be restarted after the host is present in engine - here everything works, host was successfully restarted and it boots into system normally 5) I added repositories from last 4.4.2 build: 4.4.2-8 6) Did upgrade check and upgrade on that host from webadmin... (the reboot box after upgrade was checked by default) 7) After reboot, the host entered into emergency mode [1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_device_mapper_multipath/assembly_managing-mpath-configuring-device-mapper-multipath#proc_move-root-to-mpath-managing-mpath [2] Checking multipath after step 1) # multipath -ll mpatha (36000d310047d18000000000000000012) dm-0 COMPELNT,Compellent Vol size=75G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 0:0:0:0 sda 8:0 active ready running `- 0:0:0:1 sdb 8:16 active ready running
(In reply to Petr Kubica from comment #94) > I'm not sure if it is still relevant but I hit this issue on RHEL 8.2.1 > based host which was installed from CDN via subscription) in version 4.4.1 > during upgrade to latest 4.4.2 (tested it yesterday) > > My steps: > 1) I installed RHEL 8.2.1 (from ISO) on a device which became a multipath > device. > - For installing an OS on a multipath device, it's better to have a single > path device, install OS and then extend the number of paths to that device. > I followed a documentation how it should be done [1] > 2) I registered the machine with CDN and enable necessary repositories for > 4.4.1 host > 3) I added 4.4.1 host into 4.4.1 engine which was also installed from CDN > 4) I checked via SSH Management in webadmin that the host could be restarted > after the host is present in engine - here everything works, host was > successfully restarted and it boots into system normally > 5) I added repositories from last 4.4.2 build: 4.4.2-8 > 6) Did upgrade check and upgrade on that host from webadmin... (the reboot > box after upgrade was checked by default) > 7) After reboot, the host entered into emergency mode > > [1] > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/ > html/configuring_device_mapper_multipath/assembly_managing-mpath-configuring- > device-mapper-multipath#proc_move-root-to-mpath-managing-mpath > > [2] Checking multipath after step 1) > # multipath -ll > mpatha (36000d310047d18000000000000000012) dm-0 COMPELNT,Compellent Vol > size=75G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw > `-+- policy='service-time 0' prio=50 status=active > |- 0:0:0:0 sda 8:0 active ready running > `- 0:0:0:1 sdb 8:16 active ready running 1. Could you share the host-deploy and upgrade (mgmt) logs from the engine? 2. Were you able to recover and reboot by: 2.1. Mounting the root file system 2.2. Editing etc/lvm/lvm.conf in the mounted fs and removing the bad lvm filter - Please share the current lvm filter before doing 2.2. 3. After the host boots, running "vdsm-tool config-lvm-filter" to configure the right filter 4. Rebooting to see its okay
(In reply to Petr Kubica from comment #94) > 6) Did upgrade check and upgrade on that host from webadmin... (the reboot > box after upgrade was checked by default) > 7) After reboot, the host entered into emergency mode Wasn't an upgrade to engine 4.4.2 needed before upgrading hosts? There was a fix on engine side as well for this issue.
(In reply to Amit Bawer from comment #95) > 1. Could you share the host-deploy and upgrade (mgmt) logs from the engine? > 2. Were you able to recover and reboot by: > > 2.1. Mounting the root file system > 2.2. Editing etc/lvm/lvm.conf in the mounted fs and removing the bad lvm > filter > - Please share the current lvm filter before doing 2.2. > > 3. After the host boots, running "vdsm-tool config-lvm-filter" to configure > the right filter Since there was an initramfs update here according to manual mentioned in [1] of comment 94, and what have probably caused the boot failure since the initramfs has included the bad lvm filter, you'll need to re-update it with: # dracut -f after step 3, before rebooting again. > 4. Rebooting to see its okay
(In reply to Amit Bawer from comment #97) > Since there was an initramfs update here according to manual mentioned in > [1] of comment 94, > and what have probably caused the boot failure since the initramfs has > included the bad lvm filter, > you'll need to re-update it with: > > # dracut -f # dracut --force --add multipath to cover multipath config changes as well. > > after step 3, before rebooting again. > > > 4. Rebooting to see its okay
QE verified this issue on multipath machine using the KCS(https://access.redhat.com/solutions/5428651), it works. Test version: RHVM: 4.4.2.6-0.2.el8ev RHVH: redhat-virtualization-host-4.4.2-20200918.0.el8_2 Test steps: 1. Install RHVH-4.4-20200722.1-RHVH-x86_64-dvd1.iso 2. Add host to RHVM 3. Remove current lvm filter 4. Reboot 5. Set up repo and point to "redhat-virtualization-host-4.4.2-20200918.0.el8_2" 6. Upgrade host via RHVM UI 7. Run `vdsm-tool config-lvm-filter` to confirm there is a new filter in place Test results: 1. Host upgrade is successful, and system reboot normally. 2. In step 6, after upgrade, a new filter is generated, as shown below: ~~~~~~ # lvm dumpconfig devices/filter filter=["a|^/dev/disk/by-id/lvm-pv-uuid-SK9sV3-3dh0-MRR2-2NEi-m3HC-YjVS-ty80Py$|","a|^/dev/disk/by-id/lvm-pv-uuid-dDdqNq-sOlT-MTeu-fKqK-q9fQ-UjMZ-YYgAs7$|","r|.*|"] ~~~~~~ Will move the bug status to "VERIFIED"
[this is applicable both to RHEL and RHVH hosts, using RHVH erratum for the notice] [ also note kbase in comment #102 ] Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Virtualization security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:4172