Description of problem: When adding a host to a data center, we run "vdsm-tool configure" on the host, configuring libvirt, qemu, multipath, lvm and more. When starting vdsm, we check that a host is configured, and fail the operation if the host need to be configured. We want to integrated lvm filter configuration into this configuration step, so the required configuration is transparent for users, and there is no way to run vdsm with incorrect configuration. We assume that in the common case will be: 1. user add host 2. host does not have lvm filter yet 3. vdsm-tool configure will configure lvm filter When vdsm starts, vdsm-tool is-configured will: 1. check if a host has proper lvm filter 2. fail if filter is missing or does match the recommended one Another common case is: 1. user add host 2. vdsm-tool configure detect that lvm filter is already configured We assume that there will be also rare cases of hosts with existing lvm filter which does not match vdsm recommended filter. This may be a sloppy filter does not using anchores (e..g. "a|/dev/sda2|" instead of "a|^/dev/sda2$|") or maybe some other reason we are not aware yet. In this case vdsm-tool config-lvm-filter refuse to configure the host automaticaly and the user will have to configure the host manually. In this case: - adding a host will fail - starting vdsm fail fail In both cases user will have to fix the host filter manually using vdsm-tool config-lvm-filter. Issues to think about: - Users that use only file storage do not care about lvm configuration, but their installation may fail because of it - We recommend to reboot a host after configuring a filter for testing the filter operation. Are we ok to reboot a host after configuring it? - Configuring a filter when adding a host does not solve the problem of upgrading existing host to new version supporting automatic configuration of the lvm filter. we will track this issue in another bug.
Scheduling for 4.2.2 for discussion.
Failing installation if LVM filter cannot be configured may be an issue, in particular for user that do not use block storage, but may use lvm locally in a way which is not supported by the current lvm filter generator. I discussed a less strict lvm filter with Dan, that instead of including only the mounted logical volumes, will include any logical volume which is not an oVirt logical volume, or a guest logical volume using an oVirt logical volume as a physical volume. The solution can work like this: - find all lvs and their vgs - find all vgs and and their pvs - filter out ovirt vgs by the RHAT_storage_domain tag - filter out vgs which one of their pvs is an lv in ovirt vg (guest lvs activate on a host). - create a filter for the rest of the vgs devices This may be tricky since guest vgs are likely to be partial (referencing devices from a guest, not available on a host), and lvm commands may fail when accessing these vgs. This decreases the chance that LVM filter will break some other application on a host, using LVM in a way that we do not expect, and allow fully automatic configuration of hosts.
I would like to know if LVM filter integration to "vdsm configure" is going to make it in 4.3? Is there a progress on this? I have a customer asking the status.
This bug has not been marked as blocker for oVirt 4.3.0. Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.
(In reply to nijin ashok from comment #3) > I would like to know if LVM filter integration to "vdsm configure" is going > to make it in 4.3? Is there a progress on this? I have a customer asking the > status. It may be in 4.3.z if the work will be scheduled, currently I don't know about plans to include it. Until this is integrated in host deploy flow, customers can configure the filter on all hosts using ansible or other tools to run commands on mulitple hosts. $ cat /etc/ansible/hosts [my-group] host1.foo.com host2.foo.com $ ansible my-group -u root -a "sh -c 'echo yes | vdsm-tool config-lvm-filter'" host1.foo.com | SUCCESS | rc=0 >> Analyzing host... Found these mounted logical volumes on this host: logical volume: /dev/mapper/vg0-lv_home mountpoint: /home devices: /dev/vda2 logical volume: /dev/mapper/vg0-lv_root mountpoint: / devices: /dev/vda2 logical volume: /dev/mapper/vg0-lv_swap mountpoint: [SWAP] devices: /dev/vda2 This is the recommended LVM filter for this host: filter = [ "a|^/dev/vda2$|", "r|.*|" ] This filter allows LVM to access the local devices used by the hypervisor, but not shared storage owned by Vdsm. If you add a new device to the volume group, you will need to edit the filter manually. Configure LVM filter? [yes,NO] Configuration completed successfully! Please reboot to verify the LVM configuration. host2.foo.com | SUCCESS | rc=0 >> Analyzing host... LVM filter is already configured for Vdsm
*** Bug 1739296 has been marked as a duplicate of this bug. ***
I think recent issues seen in QE storage environment and OST are related to missing or wrong lvm filter. Probably recent changes in LVM in RHEL 8 are related. I discussed this with David Teigland and our conclusion is that we must have proper lvm filter on a host. Bumping priority.
This sounds like a paper cut we might want to get in ovirt-4.4.1, can you please consider if there is capacity?
(In reply to Nir Soffer from comment #2) > The solution can work like this: > > - find all lvs and their vgs > - find all vgs and and their pvs > - filter out ovirt vgs by the RHAT_storage_domain tag > - filter out vgs which one of their pvs is an lv in ovirt vg > (guest lvs activate on a host). > - create a filter for the rest of the vgs devices This will not work since we want to block also LUNs which are not yet part of RHV storage domain, or used as direct LUN.
can that be "solved" by a clear warning in UI when you are adding a new SD or adding Direct LUN? For the former it would need to just rerun config-lvm-filter on relevant hosts, right? Is that enough to run it *after* the SD is added? For Direct LUN we can keep it manual
(In reply to Michal Skrivanek from comment #18) > can that be "solved" by a clear warning in UI when you are adding a new SD > or adding Direct LUN? For the former it would need to just rerun > config-lvm-filter on relevant hosts, right? Is that enough to run it *after* > the SD is added? > For Direct LUN we can keep it manual No, missing filter can cause trouble for any LUN even before it is used for storage domain or direct LUN, including managed LUNs added via cinderlib. Configuration must happen before vdsm enables usage of block storage, which currently is when vdsm starts, since vdsm is not modular enough. The best time to configure the filter is when a host is added to the system, or when provisioning a host, but the installer does not support yet adding a proper lvm filter. LVM folks are working on solving this, see bug 1749513.
But regarding warnings the user about misconfigured system, showing a warning about missing or unexpected lvm filter when creating/editing attaching direct LUN can be useful. The issue with such warning is how to turn it off if the user knows that the filter is correct because they have special setup that requires different filter.
from offline email thread: Assuming you have a clean host and you’re deploying host we will add the default filter (i.e. allow all currently used local devices, reject everything else) And then if anyone needs to make other device accessible to the host they have to edit the filter, once they do that we may be issuing a warning(only during host deploy) but otherwise the changes are honored. This gives people a much better chance to be safe by default and to stop&think before making changes for advanced configuration.
*** Bug 1696634 has been marked as a duplicate of this bug. ***
With https://gerrit.ovirt.org/c/109755/ it is easier now to configure automatically: vdsm-tool config-lvm-filter -y We probably need to fix the exit code so program running the command can detect: - 0: host was configured or was already configured - 1: general error, host status is unknown (e.g. running lvm command failed) - 2: incorrect usage (e.g. non-existing command line option) - 3: tool cannot configure the host, manual configuration is required. should log a warning in engine event log.
git tag --contains 2d78b70bf58942780169012ef3429cf8f96a81a6 v4.40.21
Suggested verifications: 1. Deploy a fresh el8 host from engine. See that lvm filter is configured on host after deployment: "lvmconfig | grep filter" should show "accept" expression only for /dev/sda2. 2. Re-Deploy same host from engine. See that no errors for LVM configuration in event log and that lvm filter on host is the same. 3. Manually change the lvm filter on the host in /etc/lvm/lvm.conf , set it to something different than already set there, verify with lvm command its legit (see that lvs command does not drop errors to console for bad filter). Then Re-Deploy this host from engine, see that event viewer has "Error on LVM configuration" phase for this attempt and that lvm filter is still the same on host as you have set it. When you will check host-deploy log over engine for this run you will see that LVM configuration step had rc=3. 4. Repeat (3) with a broken filter, i.e. change the lvm filter in /etc/lvm/lvm.conf to some invalid expression so lvs command will throw an error after you did it. Then try re-deploy the host, you'll see again "Error on LVM configuration" in event viewer, but this time when you will check host-deploy log over engine for this run you will see that LVM configuration step got rc=1.
Verified on: ovirt-engine-4.4.1.7-0.3.el8ev.noarch vdsm-4.40.22-1.el8ev.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RHV RHEL Host (ovirt-host) 4.4), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:3246