Description of problem: Guest LVs created on raw volumes are auto activated on the hypervisor with FC storage The host still autoactivates everything at boot, LVM freaks out seeing all the guest VGs/LVs many of which have duplicated names/guids (because of cloned VMs). This breaks: 1. RHVH installation - anaconda fails on LVM errors if storage is connected to server 2. Upgrade - nodectl/imgbase used to fail on LVM errors, not sure if still fails or ignores the errors 3. VDSM - cannot correctly deactivate powered off VM disks, because guest LVs are still active on them 4. boot/shutdown issues as lvm services fail to start/stop correctly Version-Release number of selected component (if applicable): rhvh-4.1-0.20170421 How reproducible: Steps to Reproduce: 1.install host and add it to manager 2.add fc storage to datacenter 3.create vm on raw device with lvm on top of it 4.shutdown the vm, put host into maintenance 6, reboot the host Actual results: guest vg and lv are autoactivated on the boot Expected results: guest vg and lv are not autoactivated on the boot Additional info:
There are several bugs here: 1) We use anaconda from platform, so I'd expect this bug to also be reproducible on RHEL. Can you please open a bug against Anacodna to track this? 2) nodectl/imgbase now ignores LVM errors. 3/4) These look like duplicates of rhbz#1374545, but that shipped with 4.1.1-1. Nir, can you please look at this? RHVH doesn't have any special/different handling for LVs, so I'd expect this to be vdsm/storage.
(In reply to Ryan Barry from comment #1) > There are several bugs here: > > 1) We use anaconda from platform, so I'd expect this bug to also be > reproducible on RHEL. Can you please open a bug against Anacodna to track > this? I agree, issues with installing stuff on a host with connected FC storage is platform issue, we need a RHEL bug for this. > 2) nodectl/imgbase now ignores LVM errors. I don't know what are these errors but failing on lvm errors sounds reasonable, ignoring lvm errors is very risky. > 3/4) These look like duplicates of rhbz#1374545, but that shipped with > 4.1.1-1. Nir, can you please look at this? RHVH doesn't have any > special/different handling for LVs, so I'd expect this to be vdsm/storage. We handled only iSCSI issues so far, we have couple of bugs open for FC issues, see bug 1450114.
Marian, I would like to focus on single issue in this bug. If you look in the tracker bug 1450114 we already have bugs about: - duplicate vg/lvs - bug 1130527 - slow boot - bug 1400446 - dirty luns not usable - bug 1253640 We lack a bug about thin_check breaking thin guest lvs with FC storage (see bug 1371939). Do you have a relevant case for this issue? If not, maybe the case you added is relevant to one of the other bugs?
(In reply to Nir Soffer from comment #6) > (In reply to Ryan Barry from comment #1) > I agree, issues with installing stuff on a host with connected FC storage > is platform issue, we need a RHEL bug for this. I'll open a platform bug > I don't know what are these errors but failing on lvm errors sounds > reasonable, > ignoring lvm errors is very risky. Well, we're not strictly ignoring LVM errors. Simply that with the lvmetad changes, stderr throws out a lot of messages. We're dropping that to get a clean parse, not non-zero return code. > We handled only iSCSI issues so far, we have couple of bugs open for FC > issues, > see bug 1450114. Since this bug is probably not RHVH specific, do you want to grab this for vdsm/storage?
(In reply to Ryan Barry from comment #8) > Since this bug is probably not RHVH specific, do you want to grab this for > vdsm/storage? Lets understand first what is the unique issue in this bug that is not tracked already in one of the other bugs. Waiting for Marian reply on comment 7.
Hello Nir, i think none of the bugs in comment #7 is relevant for this one, guest lvs that are autoactivated are not direct luns assigned to the vms, they are from data storage domain and they are raw devices.
Based on comment 10, moving this to vdsm.
4.1.4 is planned as a minimal, fast, z-stream version to fix any open issues we may have in supporting the upcoming EL 7.4. Pushing out anything unrelated, although if there's a minimal/trival, SAFE fix that's ready on time, we can consider introducing it in 4.1.4.
Hello Nir, thank you for responding, appreciate that even more as you are on PTO. Would it be possible that we would add this as default setting to our rhvh hosts? I think it should not break anything as we are activating rhv storage domains with vdsm. Or at we should highlight this in the documentation. What do you think?
I'm also on PTO, but from a RHVH perspective, we'd rather see this fixed from vdsm or platform, since this problem is not at all unique to RHVH. Highlighting this in the documentation would be ideal, though
(In reply to Marian Jankular from comment #16) > Hello Nir, > > thank you for responding, appreciate that even more as you are on PTO. > Would it be possible that we would add this as default setting to our rhvh > hosts? I think it should not break anything as we are activating rhv storage > domains with vdsm. Or at we should highlight this in the documentation. > > What do you think? There's no magic solution here. The current solution is manual configuration. A better, longer term solution would be a script (Ansible?) that creates this LVM filter and keeps it up-to-date.
*** Bug 1507691 has been marked as a duplicate of this bug. ***
Current state: Vdsm includes now a new vdsm-tool configure-lvm-filter command. This command analyze the current host and recommends the lvm filter for this host. Here is an example run on this command: # vdsm-tool config-lvm-filter Found these mounted logical volumes on this host: logical volume: /dev/mapper/vg0-lv_home mountpoint: /home devices: /dev/vda2 logical volume: /dev/mapper/vg0-lv_root mountpoint: / devices: /dev/vda2 logical volume: /dev/mapper/vg0-lv_swap mountpoint: [SWAP] devices: /dev/vda2 This is the recommended LVM filter for this host: filter = [ "a|^/dev/vda2$|", "r|.*|" ] To use this LVM filter please edit /etc/lvm/lvm.conf and set the 'filter' option in the 'devices' section. It is recommended to reboot the hypervisor to verify the filter. This filter will allow LVM to access the local devices used by the hypervisor, but not shared storage owned by Vdsm. If you want to add another local device you will have to add the device manually to the LVM filter. The next step is analyzing the currently configured lvm filter, and configuring the lvm filter when a user confirm. We hope to perform this configuation automatically as part of host deployment.
I posted 2 additional patches, improving this configuration tool. `vdsm-tool config-lvm-filter` analyzes the current LVM configuation to decide if a filter should to be configured, and to configure the filter automatically if possible. If the host is already configured, the tool does nothing: # vdsm-tool config-lvm-filter Analyzing host... LVM filter is already configured for Vdsm On a host not configured yet, the tool configure LVM automatically after the user confirm the operation. # vdsm-tool config-lvm-filter Analyzing host... Found these mounted logical volumes on this host: logical volume: /dev/mapper/vg0-lv_home mountpoint: /home devices: /dev/vda2 logical volume: /dev/mapper/vg0-lv_root mountpoint: / devices: /dev/vda2 logical volume: /dev/mapper/vg0-lv_swap mountpoint: [SWAP] devices: /dev/vda2 This is the recommended LVM filter for this host: filter = [ "a|^/dev/vda2$|", "r|.*|" ] This filter will allow LVM to access the local devices used by the hypervisor, but not shared storage owned by Vdsm. If you add a new device to the volume group, you will need to edit the filter manually. Configure LVM filter? [yes,NO] ? [NO/yes] yes Configuration completed successfully! Please reboot to verify the LVM configuration. If the host configuration does not match Vdsm required configuration, the user need to configure LVM filter manually. # vdsm-tool config-lvm-filter Analyzing host... Found these mounted logical volumes on this host: logical volume: /dev/mapper/vg0-lv_home mountpoint: /home devices: /dev/vda2 logical volume: /dev/mapper/vg0-lv_root mountpoint: / devices: /dev/vda2 logical volume: /dev/mapper/vg0-lv_swap mountpoint: [SWAP] devices: /dev/vda2 This is the recommended LVM filter for this host: filter = [ "a|^/dev/vda2$|", "r|.*|" ] This filter will allow LVM to access the local devices used by the hypervisor, but not shared storage owned by Vdsm. If you add a new device to the volume group, you will need to edit the filter manually. This is the current LVM filter: filter = [ "a|^/dev/vda2$|", "a|^/dev/vdb1$|", "r|.*|" ] WARNING: The current LVM filter does not match the recommended filter, Vdsm cannot configure the filter automatically. Please edit /etc/lvm/lvm.conf and set the 'filter' option in the 'devices' section to the recommended value. It is recommend to reboot after changing LVM filter. This completes the work planned for 4.2.0. After additional testing, we can integrate this into host deployment, so in the common cases, LVM filter will be configured automatically. In some cases, when a user configured in compatible LVM filter the user will have to modify the configuration manually.
2 Additional patches merged fixing issues we found in the last week.
Moving back to modified as all patches were merged in master for couple of weeks.
(In reply to Nir Soffer from comment #24) > Moving back to modified as all patches were merged in master for couple of > weeks. I still see 2 patches in POST in above referenced patches: oVirt gerrit 85348 ovirt-4.2.0 POST lvmfilter: Filter out the master LV on the SPM oVirt gerrit 85347 ovirt-4.2.0 POST lvmfilter: Remove double [yes/No]? Please cross check. Also, this is targeted 4.3.0, if it's merged in master it should probably be re-targeted to 4.2.1
We have bug 1523152 for 4.2.1.
Verified with the following code: -------------------------------------- ovirt-engine-4.2.1.3-0.1.el7.noarch vdsm-4.20.17-32.git9b853be.el7.centos.x86_64 Verified with the following scenario: -------------------------------------- 1. Created 1 VM with a Direct FC LUN attached as well as a regular block disk in addition to the OS disk 2. Start the VM and create a pv, vg and lv on the direct lun and the block disk with the names guest_vg, guest_lv and guest_vg1, guest_lv1 3. Rebooted the host >>>>> 2 sets of duplicate LVs are reported as active 4. Add the filter with vdsm-tool config-lvm-filter and reboot the host again >>>>> when the host comes up the Guest lvs are no longer displayed and are not active Moving to VERIFIED!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:1489
BZ<2>Jira Resync