Bug 1374545 - Guest LVs created in ovirt raw volumes are auto activated on the hypervisor in RHEL 7
Summary: Guest LVs created in ovirt raw volumes are auto activated on the hypervisor i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.6.7
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ovirt-4.1.1
: ---
Assignee: Nir Soffer
QA Contact: Natalie Gavrielov
URL:
Whiteboard:
: 1326828 1374549 1421424 (view as bug list)
Depends On:
Blocks: 1130527 1202595 1253640 1303940 1313588 1325844 1326828 1331978 1342786 1371939 1373118 1374549 1377157 1398918 1400446 1400528 1403839 1411197 1454287
TreeView+ depends on / blocked
 
Reported: 2016-09-09 00:40 UTC by Germano Veit Michel
Modified: 2019-12-16 06:41 UTC (History)
41 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1398918 (view as bug list)
Environment:
Last Closed: 2017-04-25 00:43:56 UTC
oVirt Team: Storage
Target Upstream Version:


Attachments (Terms of Use)
Script for generating lvm filter (2.82 KB, text/x-python)
2017-01-19 19:11 UTC, Nir Soffer
no flags Details
Script for generating lvm filter (3.07 KB, text/x-python)
2017-01-20 12:46 UTC, Nir Soffer
no flags Details
Script for generating lvm filter (2.19 KB, text/x-python)
2017-01-23 12:50 UTC, Nir Soffer
no flags Details
Script for generating lvm filter (2.85 KB, text/x-python)
2017-01-23 13:59 UTC, Nir Soffer
no flags Details
vgs -o vg_name,pv_name --noheading --select 'vg_tags != {RHAT_storage_domain}' (384 bytes, text/plain)
2017-01-24 04:34 UTC, Germano Veit Michel
no flags Details
command issued for creating pv,vg and lv in verification process, step 3 comment 165 (352.19 KB, image/png)
2017-04-13 15:12 UTC, Natalie Gavrielov
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1236161 'medium' 'CLOSED' 'VDSM response to path preparation during live VM migration was very slow' 2019-11-25 18:45:43 UTC
Red Hat Bugzilla 1334726 'unspecified' 'CLOSED' '[PPC][rhevm-3.6.6-0.1] RHEL 7.2 vm with copied disks enters emergency mode when booted.' 2019-11-25 18:45:43 UTC
Red Hat Bugzilla 1358348 None None None 2019-11-25 18:45:43 UTC
Red Hat Bugzilla 1361549 'unspecified' 'CLOSED' '[PPC][rhevm-3.6.6-0.1] RHEL 7.2 vm with copied disks enters emergency mode when booted.' 2019-11-25 18:45:43 UTC
Red Hat Bugzilla 1377157 'unspecified' 'CLOSED' 'LVM cache not updating and reporting unknown device' 2019-11-25 18:45:43 UTC
Red Hat Bugzilla 1386732 'unspecified' 'CLOSED' '[scale] Unable to add more disks "VolumeCreationError: Error creating a new volume" (limit of ~1950 disks per SD)' 2019-11-25 18:45:43 UTC
Red Hat Bugzilla 1425233 'unspecified' 'CLOSED' 'Ensure that the HE 3.5 -> 3.6 upgrade procedure is still working if executed on 4.1 host with jsonrpc' 2019-11-25 18:45:44 UTC
Red Hat Bugzilla 1462792 'medium' 'NEW' '[BLOCKED on platform bug 1546538] systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is ma... 2019-11-25 18:45:44 UTC
Red Hat Knowledge Base (Solution) 2610081 None None None 2016-09-12 06:40:26 UTC
Red Hat Knowledge Base (Solution) 2662261 None None None 2016-09-28 02:26:23 UTC
Red Hat Product Errata RHEA-2017:0998 normal SHIPPED_LIVE VDSM bug fix and enhancement update 4.1 GA 2017-04-18 20:11:39 UTC
oVirt gerrit 64328 'None' 'ABANDONED' 'tests: Rename lvmTests to new naming convention' 2019-11-25 18:45:46 UTC
oVirt gerrit 64329 'None' 'MERGED' 'tests: Add loopback module' 2019-11-25 18:45:46 UTC
oVirt gerrit 64330 'None' 'ABANDONED' 'guest-lvs: Add failing test for guest lvs' 2019-11-25 18:45:47 UTC
oVirt gerrit 64367 'None' 'ABANDONED' 'guest-lvs: Add lvm bootstrap tests' 2019-11-25 18:45:47 UTC
oVirt gerrit 64368 'None' 'ABANDONED' 'guest-lvs: Deactivate guest lvs during bootstrap' 2019-11-25 18:45:48 UTC
oVirt gerrit 64369 'None' 'ABANDONED' 'guest-lvs: Deactivate guest lvs during deactivation' 2019-11-25 18:45:47 UTC
oVirt gerrit 64370 'None' 'ABANDONED' 'guest-lvs: Skip foreign vgs during bootstrap' 2019-11-25 18:45:47 UTC
oVirt gerrit 66893 'None' 'MERGED' 'lvm: Add vdsm local lvm configuration' 2019-11-25 18:45:47 UTC
oVirt gerrit 70899 'None' 'ABANDONED' 'gen-lvm-filter: Add helper to generate LVM filter' 2019-11-25 18:45:47 UTC


Description Germano Veit Michel 2016-09-09 00:40:40 UTC
Description of problem:
When a hypervisor with FC storage is rebooted, the host sees all the LVs of the storage domain, fine. But then it LVM PV scan scans all these devices and finds the VM's LVM metadata in case it's raw disk! So we end up with dm-xxx in the host pointing to the VM's internal LVs. These devices are never cleared/removed so it ends up with lot's of stale LVs and mapper devices, which can lead to disk corruption.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160711.0.el7ev)

How reproducible:
100% on customer side

Steps to Reproduce:
1. Reboot host

Actual results:
- vdsm will fail to lvchange -an
- stale LVs in all hosts

Expected results:
- LV only open in the host running the VM
- No child dm-xxx devices from VM internal LVM.

Additional info:

Logs Host booting:
lvm: 140 logical volume(s) in volume group "b442d48e-0398-4cad-b9bf-992a3e663573" now active <--- storage domain
systemd: Started LVM2 PV scan on device 253:249.  <----- dm-249 is the LV of the VM disk (raw, prealloc)
lvm: 1 logical volume(s) in volume group "vg_pgsql" now active <--- this is VM internal!

Having this VM internal child, will make vdsm's lvchange -an fail when the VM stops.

$ cat sos_commands/devicemapper/dmsetup_info_-c | egrep 'pgsql|5853cd' | awk -F' ' '{print $1" dm-"$3}'
b442d48e--0398--4cad--b9bf--992a3e663573-5853cdf8--7b84--487e--ab70--827bf5b00140 dm-249 <--- LV of VM image
vg_pgsql-lv_pgsql dm-272                                                                 <--- internal VM stuff

vg_pgsql-lv_pgsql (253:272)
 `-b442d48e--0398--4cad--b9bf--992a3e663573-5853cdf8--7b84--487e--ab70--827bf5b...
    `-36000144000000010706222888cc3683f (253:107)
       |- (131:880)
       |- (67:992)
       |- (132:592)
       |- (68:704)
       |- (133:304)
       |- (69:416)
       |- (134:16)
       `- (70:128)

So in the HOST we have dm-272 which is VMs internal business relying on dm-249, which is the VM disk (LV). vdsm fails to deactivate dm-249 as well. The result is a full Data Center where ALL hosts have these LVs active, asking for trouble.

jsonrpc.Executor/4::ERROR::2016-09-06 19:53:08,144::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Cannot deactivate Logical Volume: (\'General Storage Exception: ("5 [] [\\\'  Logical volume b442d48e-0398-4cad-b9bf-992a3e663573/5853cdf8-7b84-487e-ab70-827bf5b00140 is used by another device.\\\']\\\\nb442d48e-0398-4cad-b9bf-992a3e663573/[\\\'ffd27b7d-5525-4126-9f59-5a26dedad157\\\', \\\'5853cdf8-7b84-487e-ab70-827bf5b00140\\\']",)\',)', 'code': 552}}

I think the root cause here is that this https://gerrit.ovirt.org/#/c/21291/ will fail to deactivate LVs in case the Disk LV already has children (VM internal).

Comment 10 Nir Soffer 2016-09-11 15:21:38 UTC
(In reply to Germano Veit Michel from comment #0)

Germano, thanks for this detailed report.

I don't know if we can prevent systemd from scanning 
lvs inside other lvs, but we can prevent it from auto
activating lvs.

Can you check if disabling auto activation fixes this issue?

Edit /etc/lvm/lvm.conf:

    auto_activation_volume_list = []

Comment 11 Nir Soffer 2016-09-11 15:47:00 UTC
> systemd: Started LVM2 PV scan on device 253:249

Peter, can you explain why systemd is looking for lvs inside another lv, and
why it is automatically activates these lvs?

Can we configure lvm to avoid this scan?

Comment 12 Nir Soffer 2016-09-11 16:04:59 UTC
See also bug 1253640, both seems to be caused by the lvm auto-activation.

Comment 13 Germano Veit Michel 2016-09-12 06:28:51 UTC
(In reply to Nir Soffer from comment #10)
> Can you check if disabling auto activation fixes this issue?
> 
> Edit /etc/lvm/lvm.conf:
> 
>     auto_activation_volume_list = []

Hi Nir,

I just asked the customer to try it. I will keep you updated.

Cheers

Comment 14 Peter Rajnoha 2016-09-12 07:26:34 UTC
(In reply to Nir Soffer from comment #11)
> > systemd: Started LVM2 PV scan on device 253:249
> 
> Peter, can you explain why systemd is looking for lvs inside another lv, and
> why it is automatically activates these lvs?
> 

It's because the internal LV, if found, is just like any other LV. Unless you mark that somehow, LVM has no way to know whether this the LV is the one that should not be activated (...you might as well have a stack of LVs - LV on top of another LV without any VMs so from this point of view nothing is "internal" and you want to activate the LV in this case).

By default, LVM autoactivates all VGs/LVs it finds.

> Can we configure lvm to avoid this scan?

You have several ways:

  - You can set devices/global_filter to include only PVs which should be scanned and any LVs activated on the host and reject everything else (this also prevents any scans for all the other devices/LVs which contain further PVs/VGs inside.

  - You can mark LVs with tags and then set activation/auto_actiavation_volume_list to activate only LVs with certain tag. Or, without tagging, directly listing the VGs/LVs which should be autoactivated only. But this way, the VMs PVs inside are going to be scanned still, just the VGs/LVs not autoactivated.

  - You can mark individual LVs to be skipped on autoactivation (lvchange -K|--setactivationskip y). But this way, you will also prevent the autoactivation within guest system too as the flag to skip activation is stored directly in VG metadata!!! So then you'd need someone to call "vgchange/lvchagne -ay" - the direct activation (in contrast to "vgchange/lvchange -aay" - the autoactivation, which is used by boot scripts) inside the guest to activate the LV.

  - You can use new "LVM system id" (see man lvmsystemid) feature which marks VGs with system id automatically and then only the VGs created on that system are visible/accessible (again, in this case, the PVs inside are going to be scanned still because we need to get the "ID" from VG metadata to be able to do the comparison of system IDs.


If you really want to avoid scanning of internal PVs which are inside VG/LV by chance, the best is probably to use the global_filter to include only the PVs you know that are safe to access.

Comment 15 Peter Rajnoha 2016-09-12 07:32:02 UTC
(In reply to Peter Rajnoha from comment #14)
> (In reply to Nir Soffer from comment #11)
> > > systemd: Started LVM2 PV scan on device 253:249
> > 
> > Peter, can you explain why systemd is looking for lvs inside another lv, and
> > why it is automatically activates these lvs?
> > 
> 
> It's because the internal LV, if found, is just like any other LV. Unless
> you mark that somehow, LVM has no way to know whether this the LV is the one
> that should not be activated (...you might as well have a stack of LVs - LV
> on top of another LV without any VMs so from this point of view nothing is
> "internal" and you want to activate the LV in this case).
> 
> By default, LVM autoactivates all VGs/LVs it finds.

(Also, when using autoactivation, the activation is based on udev events, so if the PV appears, it's scanned for VG metadata and LVs are autoactivated. Each LV activation generates another udev event and the procedure repeates - if there's any PV found inside the LV, the autoactivation triggers for that PV. It's because the is like any other block device and as such, it can contain further PVs/VGs/LVs stacked inside... So when it comes to autoactivation, it's domino effect - the activation triggers another activation and so on unless you stop LVM from doing that by the means I described in coment #14.)

Comment 16 Nir Soffer 2016-09-12 07:51:24 UTC
(In reply to Peter Rajnoha from comment #14)
> (In reply to Nir Soffer from comment #11)
> > > systemd: Started LVM2 PV scan on device 253:249
> > 
> > Peter, can you explain why systemd is looking for lvs inside another lv, and
> > why it is automatically activates these lvs?
> > 
> 
> It's because the internal LV, if found, is just like any other LV. Unless
> you mark that somehow, LVM has no way to know whether this the LV is the one
> that should not be activated (...you might as well have a stack of LVs - LV
> on top of another LV without any VMs so from this point of view nothing is
> "internal" and you want to activate the LV in this case).
> 
> By default, LVM autoactivates all VGs/LVs it finds.
> 
> > Can we configure lvm to avoid this scan?
> 
> You have several ways:
> 
>   - You can set devices/global_filter to include only PVs which should be
> scanned and any LVs activated on the host and reject everything else (this
> also prevents any scans for all the other devices/LVs which contain further
> PVs/VGs inside.

This is an issue since we don't know what are the pvs that must be scanned
on this host.

We want to avoid scanning any pv created by vdsm, but there is no easy way
to detect these - basically anything under /dev/mapper/guid may be pv owned
by vdsm.

I don't think we can change multipath configuration / udev rules to link
devices elsewhere, since it can break other software using multipath devices.

Also we cannot use global_filter since it overrides filter used by vdsm
commands.

>   - You can mark LVs with tags and then set
> activation/auto_actiavation_volume_list to activate only LVs with certain
> tag. Or, without tagging, directly listing the VGs/LVs which should be
> autoactivated only. But this way, the VMs PVs inside are going to be scanned
> still, just the VGs/LVs not autoactivated.

We plan to disable auto activation (see comment 4), so this seems to be
the best option. Can you confirm that this should resolve this issue?

>   - You can use new "LVM system id" (see man lvmsystemid) feature which
> marks VGs with system id automatically and then only the VGs created on that
> system are visible/accessible (again, in this case, the PVs inside are going
> to be scanned still because we need to get the "ID" from VG metadata to be
> able to do the comparison of system IDs.

This will not work for shared storage, the vg/lvs created on the spm host
must be accessible on other hosts.

Comment 17 Peter Rajnoha 2016-09-12 07:59:50 UTC
(In reply to Nir Soffer from comment #16)
> We plan to disable auto activation (see comment 4), so this seems to be
> the best option. Can you confirm that this should resolve this issue?
> 

The disks (LVs inside which the PV is found) is still going to be scanned. Only activation/global_filter prevents this scan. But yes, the LVs found inside won't get activated. However, if you disable autoactivation completely, no LV will get activated at boot, not even the ones on the host, if you have any LVs there.

> >   - You can use new "LVM system id" (see man lvmsystemid) feature which
> > marks VGs with system id automatically and then only the VGs created on that
> > system are visible/accessible (again, in this case, the PVs inside are going
> > to be scanned still because we need to get the "ID" from VG metadata to be
> > able to do the comparison of system IDs.
> 
> This will not work for shared storage, the vg/lvs created on the spm host
> must be accessible on other hosts.

You can share the same ID for all the hosts where you need the VGs/LVs to be visible and accessible (see also "lvmlocal" or "file" system_id_source in man lvmsystemid).

Comment 18 Nir Soffer 2016-09-12 08:26:44 UTC
(In reply to Peter Rajnoha from comment #17)
> (In reply to Nir Soffer from comment #16)
> > We plan to disable auto activation (see comment 4), so this seems to be
> > the best option. Can you confirm that this should resolve this issue?
> > 
> 
> The disks (LVs inside which the PV is found) is still going to be scanned.
> Only activation/global_filter prevents this scan. But yes, the LVs found
> inside won't get activated. However, if you disable autoactivation
> completely, no LV will get activated at boot, not even the ones on the host,
> if you have any LVs there.

The admin can fix this by adding the needed lvs to the
auto_activation_volume_list, right?

We cannot work with auto activate everything policy, only with auto activate
only the volumes specified by the admin.

> > >   - You can use new "LVM system id" (see man lvmsystemid) feature which
> > > marks VGs with system id automatically and then only the VGs created on that
> > > system are visible/accessible (again, in this case, the PVs inside are going
> > > to be scanned still because we need to get the "ID" from VG metadata to be
> > > able to do the comparison of system IDs.
> > 
> > This will not work for shared storage, the vg/lvs created on the spm host
> > must be accessible on other hosts.
> 
> You can share the same ID for all the hosts where you need the VGs/LVs to be
> visible and accessible (see also "lvmlocal" or "file" system_id_source in
> man lvmsystemid).

Ok, this way looks good to solve bug 1202595 and . Can you confirm on that bug?

Comment 19 Peter Rajnoha 2016-09-12 08:52:24 UTC
(In reply to Nir Soffer from comment #18)
> The admin can fix this by adding the needed lvs to the
> auto_activation_volume_list, right?
> 
> We cannot work with auto activate everything policy, only with auto activate
> only the volumes specified by the admin.
> 

Sure, if that's what the configuration setting is for... (...the only issue is that it requires some manual actions/configuration from admins).

> > > >   - You can use new "LVM system id" (see man lvmsystemid) feature which
> > > > marks VGs with system id automatically and then only the VGs created on that
> > > > system are visible/accessible (again, in this case, the PVs inside are going
> > > > to be scanned still because we need to get the "ID" from VG metadata to be
> > > > able to do the comparison of system IDs.
> > > 
> > > This will not work for shared storage, the vg/lvs created on the spm host
> > > must be accessible on other hosts.
> > 
> > You can share the same ID for all the hosts where you need the VGs/LVs to be
> > visible and accessible (see also "lvmlocal" or "file" system_id_source in
> > man lvmsystemid).
> 
> Ok, this way looks good to solve bug 1202595 and . Can you confirm on that
> bug?

Yes, this should resolve the issue (see also bug #867333) as far as VG metadata is readable so we can read the ID and then decide whether the VG is allowed or not on that system.

Comment 25 Nir Soffer 2016-09-13 19:04:58 UTC
I renamed the bug to reflect the important issue in this bug. We already know that
guest created lvs are accessible via lvm, see bug 1202595.

Comment 34 Nir Soffer 2016-09-15 17:11:35 UTC
David, in this bug we recommended to disable lvm auto activation by setting:

    auto_activation_volume_list = []

Based on your comment:
https://bugzilla.redhat.com/show_bug.cgi?id=1303940#c52

Do you think we should also recommend disabling lvmetad by setting

    use_lvmetad = 0

The results are not clear yet, see comment 32.

Comment 35 David Teigland 2016-09-15 18:27:00 UTC
Sorry, my comment in the other bz was misleading.  You still want to set

auto_activation_volume_list = []

to prevent the system (lvm/systemd/udev) from automatically activating LVs, whether use_lvmetad is 0 or 1.  So in your case, you want to both disable caching by setting use_lvmetad=0 in lvm.conf, and disable autoactivation by setting auto_activation_volume_list = [ ] in lvm.conf.

Comment 47 David Teigland 2016-09-16 21:51:17 UTC
Is auto_activation_volume_list = [] set in the copy of lvm.conf used during boot (initramfs)?  Something must be calling vgchange or lvchange to activate LVs, but nothing is coming to mind at the moment.  Will have to look into that more on Monday.

Comment 48 Nir Soffer 2016-09-16 23:16:24 UTC
I could partially reproduce this issue on rhel 7.3 beta and vdsm master with
iscsi storage.

1. Setup standard lvm.conf:
   - use_lvmetad=1
   - no auto_activation_volume_list option

2. Crate preallocated disk on iscsi storage domain and attach to vm
   (4df47a96-8a1b-436e-8a3e-3a638f119b48)

3. In the guest, create pv, vg and lvs:

   pvcreate /dev/vdb
   vgcreate guest-vg /dev/vdb
   lvcreate -n guest-lv -L 10g guest-vg
   lvcreate -n guest-lv-2 -L 5g guest-vg

4. Shutdown vm

5. Put host to maintenance

6. Activate host

   pvscan --cache
   lvs

  3628a407-ef01-417f-8b5e-e88e87896477 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  4df47a96-8a1b-436e-8a3e-3a638f119b48 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-ao----  20.00g                                                    
  d502b6ad-e623-472f-bdc7-453765089f55 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  ids                                  bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-ao---- 128.00m                                                    
  inbox                                bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  leases                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a-----   2.00g                                                    
  master                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a-----   1.00g                                                    
  metadata                             bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 512.00m                                                    
  outbox                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  guest-lv                             guest-vg                             -wi-a-----  10.00g                                                    
  guest-lv-2                           guest-vg                             -wi-a-----   5.00g                                                    
  lv_home                              vg0                                  -wi-ao---- 736.00m                                                    
  lv_root                              vg0                                  -wi-ao----   7.37g                                                    
  lv_swap                              vg0                                  -wi-ao----   7.36g                                                    

   - All lvs were activated
   - the raw lvs used as guest pc is active and open
   - guest created lvs are active


On this system, disabling lvmetad fixes the issue with open lvs:

1. Disable lvmetad

    systemctl stop lvm2-lvmetad.service lvm2-lvmetad.socket
    systemctl mask lvm2-lvmetad.service lvm2-lvmetad.socket
 
2. Edit /etc/lvm/lvm.conf:

   use_lvmetad = 0
   
   Note: I did not set auto_activation_volume_list, since this host 
   won't boot with this setting. Boot fail with dependcy for /home

3. Move host to maintenance

4. Reboot host

5. Activate host

After boot, all disk lvs are inactive, and guest lvs do not show
in lvm commands:

  3628a407-ef01-417f-8b5e-e88e87896477 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi------- 128.00m                                                    
  4df47a96-8a1b-436e-8a3e-3a638f119b48 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-------  20.00g                                                    
  d502b6ad-e623-472f-bdc7-453765089f55 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi------- 128.00m                                                    
  ids                                  bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-ao---- 128.00m                                                    
  inbox                                bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  leases                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a-----   2.00g                                                    
  master                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a-----   1.00g                                                    
  metadata                             bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 512.00m                                                    
  outbox                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  lv_home                              vg0                                  -wi-ao---- 736.00m                                                    
  lv_root                              vg0                                  -wi-ao----   7.37g                                                    
  lv_swap                              vg0                                  -wi-ao----   7.36g     

6. Starting the vm using the raw disk with guest lvs

The guest lvs show when the raw lv is activated (opened by qemu)

  3628a407-ef01-417f-8b5e-e88e87896477 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi------- 128.00m                                                    
  4df47a96-8a1b-436e-8a3e-3a638f119b48 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-ao----  20.00g                                                    
  d502b6ad-e623-472f-bdc7-453765089f55 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi------- 128.00m                                                    
  ids                                  bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-ao---- 128.00m                                                    
  inbox                                bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  leases                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a-----   2.00g                                                    
  master                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a-----   1.00g                                                    
  metadata                             bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 512.00m                                                    
  outbox                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  guest-lv                             guest-vg                             -wi-------  10.00g                                                    
  guest-lv-2                           guest-vg                             -wi-------   5.00g                                                    
  lv_home                              vg0                                  -wi-ao---- 736.00m                                                    
  lv_root                              vg0                                  -wi-ao----   7.37g                                                    
  lv_swap                              vg0                                  -wi-ao----   7.36g                                                    
  
7. Shutting down the vm hide the guest lvs again

  3628a407-ef01-417f-8b5e-e88e87896477 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi------- 128.00m                                                    
  4df47a96-8a1b-436e-8a3e-3a638f119b48 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-------  20.00g                                                    
  d502b6ad-e623-472f-bdc7-453765089f55 bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi------- 128.00m                                                    
  ids                                  bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-ao---- 128.00m                                                    
  inbox                                bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  leases                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a-----   2.00g                                                    
  master                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a-----   1.00g                                                    
  metadata                             bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 512.00m                                                    
  outbox                               bb85ee2f-d674-489f-9377-3eb1f176e8fb -wi-a----- 128.00m                                                    
  lv_home                              vg0                                  -wi-ao---- 736.00m                                                    
  lv_root                              vg0                                  -wi-ao----   7.37g                                                    
  lv_swap                              vg0                                  -wi-ao----   7.36g                                                    


To hide both guest lvs and rhev lvs from the host, I tried this
filter in lvm.conf:

    filter = [ "r|^/dev/disk/by-id/dm-uuid-mpath-.*|" ] 

With this filter, lvs show only local lvs:

  lv_home vg0 -wi-ao---- 736.00m                                                    
  lv_root vg0 -wi-ao----   7.37g                                                    
  lv_swap vg0 -wi-ao----   7.36g                                                    

But when starting the vm using the raw lv with guest pv,
the guest lvs appear again:

  guest-lv   guest-vg -wi-------  10.00g                                                    
  guest-lv-2 guest-vg -wi-------   5.00g                                                    
  lv_home    vg0      -wi-ao---- 736.00m                                                    
  lv_root    vg0      -wi-ao----   7.37g                                                    
  lv_swap    vg0      -wi-ao----   7.36g                                                    

To keep the guest lvs hidden, I tried this filter:

    filter = [ "r|^/dev/disk/by-id/dm-uuid-mpath-.*|", "r|^/dev/disk/by-id/dm-uuid-LVM-.*|" ]

lvs show now only the local lvs.

This filter may not work with RHEVH, you may need to add the
internal lvs to the filter.

Vdsm is not effected by this filter since it overrides the filter 
in all lvm commands, using --config option.

Note that sos commands need to override this filter to be able to
see shared rhev lvs:

    lvs --config 'devices { filter = [ "a|.*|" ] }'

So it seems that filter is the best way now.

Garmano, see also also comment 47.

Comment 51 David Teigland 2016-09-22 16:16:24 UTC
Peter had a good explanation of the options above, so I'll just repeat some of that.

Ideally, you don't want RHEV PVs/LVs to be scanned or activated by lvm run from the host (by which I mean lvm commands not run explicitly by RHEV).  This means that the host's lvm.conf/global_filter (on root fs and initramfs) should exclude RHEV PVs.  (Instead of excluding RHEV PVs, you could also whitelist only non-RHEV PVs.  I'm not sure if whitelist or blacklist for the host would work better here.)

Without using the global_filter, the next best option as you've found may be to disable autoactivation in the host's lvm.conf (on root fs and initramfs).  This has limitations:
- It will not protect RHEV PVs/LVs from being seen by the host's lvm.
- It will not protect RHEV LVs from being activated by an unknown vgchange/lvchange -ay command run from the host that doesn't include the extra 'a' flag.
- It will protect RHEV LVs from being autoactivated by the host's own vgchange/lvchange -aay commands.

If you use one of these two methods, and RHEV LVs are still being activated by the host, outside of your own control, then those methods are not set up correctly, or there is a rogue vgchange/lvchange being run, or there's an lvm bug.

Peter also mentioned some more exotic options (e.g. system ID) which would probably take more effort to get working, but may be worth trying in a future version.  For now, global_filter or auto_activation_volume_list should be able to solve the problem of unwanted activation.

Comment 56 Nir Soffer 2016-09-24 20:19:17 UTC
This bug reveals two issues:
1. systemd and lvm are too eager to activate anything on a system - this is
   a regression in rhel 7 compared with rhel 6.
2. vdsm startup deactivation does not handle ovirt lvs with guest lvs

The root cause is 1. We will work on configuring lvm during vdsm configuration.
This seems to be very delicate, requiring special filter and regenerating 
initramfs.

For 4.0 we can improve vdsm deactivation to handle lvs which are used as guest pvs.

Workarounds:
- setting up a lvm.conf filter (see comment 48) and regenerating initramfs
- or avoiding creating pvs directly on guest devices (without creating partition
  table).

Comment 57 Marina Kalinin 2016-09-26 14:44:52 UTC
Nir / David,
Is it possible to filter by lvm tags?
If yes - maybe it will give us more flexibility?

Comment 58 David Teigland 2016-09-26 16:20:19 UTC
global_filter/filter in lvm.conf operate at the device level, and only take device path names.  tags operate at the VG level and can be used with autoactivation.

Comment 59 Marina Kalinin 2016-09-26 19:30:36 UTC
lol, I just now looked into Nir's commits. I am sorry. He is using the tags.

Comment 60 Marina Kalinin 2016-09-26 19:52:37 UTC
To add to the uses cases this bug affects, I believe direct lun attached to the guest with a VG on top of it, will have the same issue, right?

Comment 61 Germano Veit Michel 2016-09-28 02:20:53 UTC
(In reply to Nir Soffer from comment #56)
> This bug reveals two issues:
> 1. systemd and lvm are too eager to activate anything on a system - this is
>    a regression in rhel 7 compared with rhel 6.
> 2. vdsm startup deactivation does not handle ovirt lvs with guest lvs
> 
> The root cause is 1. We will work on configuring lvm during vdsm
> configuration.
> This seems to be very delicate, requiring special filter and regenerating 
> initramfs.
> 
> For 4.0 we can improve vdsm deactivation to handle lvs which are used as
> guest pvs.
> 
> Workarounds:
> - setting up a lvm.conf filter (see comment 48) and regenerating initramfs
> - or avoiding creating pvs directly on guest devices (without creating
> partition
>   table).

Thank you Nir. I'm also happy about 2, quite sure it will prevent related issues in the future!

I have created a customer facing solution explicitly for the workarounds, both on RHEL/RHV-H and RHEV-H
https://access.redhat.com/solutions/2662261

* Will do some tests regarding the required filter on RHEV-H and update the solution as well.
* I'll suggest the customer to also evaluate the workaround in a test host.

Cheers,

Comment 63 Germano Veit Michel 2016-09-28 06:17:38 UTC
(In reply to Nir Soffer from comment #56)
> Workarounds:
> - setting up a lvm.conf filter (see comment 48) and regenerating initramfs

I'm testing this in RHEV-H, to come up with a proper filter.

1. Added this to a RHEV-H host:

filter = [ "r|^/dev/disk/by-id/dm-uuid-mpath-.*|", "r|^/dev/disk/by-id/dm-uuid-LVM-.*|", "a|^/dev/disk/by-id/dm-name-HostVG-.*|", "a|^/dev/disk/by-id/dm-name-live-.*|"]

2. ovirt-node-rebuild-initramfs

3. Reboot

4. lv-guest is active right after boot, but not open.

lvs --config 'devices { filter = [ "a|.*|" ] }' | grep lv-guest
  lv-guest                             vg-guest                             -wi-a-----   4.00m 

5. VDSM did not deactivate it because it was open(!?)?

storageRefresh::DEBUG::2016-09-28 04:27:09,074::lvm::661::Storage.LVM::(bootstrap) Skipping open lv: vg=76dfe909-20a6-4627-b6c4-7e16656e89a4 lv=6aacc711-0ecf-4c68-b64d-990ae33a54e3

Just to confirm it's the Guest one being seen in the host:

vg--guest-lv--guest (253:52)
 `-76dfe909--20a6--4627--b6c4--7e16656e89a4-6aacc711--0ecf--4c68--b64d--990ae33a54e3 (253:47)
    `-360014380125989a10000400000480000 (253:7)
       |- (8:64)
       `- (8:16)

Thoughts:

A) Is this happening exclusively in RHEV-H? 
B) Should the workaround in RHEV-H also include some of the previously discussed options? Like the volume activation one?

Comment 64 Germano Veit Michel 2016-09-28 06:36:17 UTC
Just to complement with more data:

Even after applying that filter + regenerating initramfs

Disk LV boots up open.

lvs --config 'devices { filter = [ "a|.*|" ] }' | grep 6aacc
  6aacc711-0ecf-4c68-b64d-990ae33a54e3 76dfe909-20a6-4627-b6c4-7e16656e89a4 -wi-ao----   1.00g 

Guest LV is active

# lvs --config 'devices { filter = [ "a|.*|" ] }' | grep lv-guest
  lv-guest   vg-guest -wi-a----- 4.00m

vg--guest-lv--guest (253:52)
 `-76dfe909--20a6--4627--b6c4--7e16656e89a4-6aacc711--0ecf--4c68--b64d--990ae33a54e3 (253:47)
    `-360014380125989a10000400000480000 (253:7)
       |- (8:64)
       `- (8:16)

lvm.conf seems to be working as intended:
# lvs
  LV      VG        Attr       LSize  
  Config  HostVG    -wi-ao----  8.00m
  Data    HostVG    -wi-ao---- 39.92g
  Logging HostVG    -wi-ao----  2.00g
  Swap    HostVG    -wi-ao---- 17.68g
  lv_home vg_rhevh1 -wi------- 39.68g
  lv_root vg_rhevh1 -wi------- 50.00g
  lv_swap vg_rhevh1 -wi-------  9.83g

Probably initramfs was not updated properly?

Comment 65 Germano Veit Michel 2016-09-28 07:00:04 UTC
OK, got it. ovirt-node-rebuild-initramfs is not pulling the modified lvm.conf, seems to be using it's own.

# cp /run/initramfs/live/initrd0.img .
# mv initrd0.img initrd0.gz
# gunzip unitrd0.gz
# file initrd0 
initrd0: ASCII cpio archive (SVR4 with no CRC)
# cpio -i -F initrd0
# find ./ -name lvm.conf
./etc/lvm/lvm.conf
# cat etc/lvm/lvm.conf              
global {
locking_type = 4
use_lvmetad = 0
}

Explains my results... Any ideas on how to update this in a supported way that customers can use? Or the only option is to do it manually (mount, dracut, umount)?

Comment 66 David Teigland 2016-09-28 14:47:29 UTC
You always want to use the standard lvm.conf as the starting point and modify fields in that.

Comment 73 Fabian Deutsch 2016-09-30 09:39:08 UTC
Reply to comment 69: My idea would be to allow passing some arguments to dracut, i.e. --lvmconf in this case.

We can work on such a solution once we know that the filtering is working correct. Btw I do not see a technical reason why the filters should not work in RHEV-H 3.6's initrd.

Comment 74 Zdenek Kabelac 2016-09-30 10:27:50 UTC
From discussion with  Nir Soffer  - here are couple points
for better functionality of this vdsm system:

Every host in the system should have lvm.conf 'filter'&'global_filter' setting  set in a way - it will NOT see a  'SharedVG' mpath device.
(Such change needs to be reflected in initramdisk - so regeneration is needed)

This is IMHO the most 'complex' step - since lvm2 does not support some 'filter' chaining - the filter has to be properly configured on every host.

I'd always advise  'white-list' logic - so if the host knows it's using only  'sda' & 'sdb'  as a PV -   only those 2 device should be 'accepted' and all other devices 'rejected'.  But I've already seen way more complex filters - so this part of advice is not 'trivial' to be easily automated.

It's always possible to check via look at  'lvs -vvvv' output whether devices are rejected accordingly.

To validate which settings are in-use by command - see 'man lvmconfig'.

Once the  'host' is set to never ever see  SharedVG mpath device -  it's mostly done. Since easier part is now to ensure  every executed 'vdsm' command comes with special --config  option which  DOES make only SharedVG mpath visible and reject every other device  -   and it should also go with  'locking_type=4'  to ensure  host is not able to accidentally  modify anything on a VG (even it would be some internal lvm2 bug issue)

This should lead to a system -  where  'individual' lvm2 commands executed on a host in such system - DO NEVER influence state of 'SharedVG' - as well as they will never try to auto-active LVs, never try to fix invalid metadata and so on....

Also 'vdsm' will clearly provide his FULL control - which command across whole system may be working with sharedVG metadata.

I'd like to emphasize -  while  'vdsm' is running i.e. activation command on any host  - it should NOT try to modify  VG metadata anywhere else - especially if such VG consists of multiple PV - there is possibly to hit the 'race' where   read-only  metadata user could see partially updated metadata.
So 'update' of VG metadata  require exclusive access.


And a final comment - IMHO such configured host system then could possibly use 'lvmetad' locally for locally available devices - since there shall be no interference.

Just vdsm commands needs to go with --config  lvmetad=0

Comment 75 Nir Soffer 2016-09-30 10:40:42 UTC
(In reply to Zdenek Kabelac from comment #74)
> Every host in the system should have lvm.conf 'filter'&'global_filter'
> setting  set in a way - it will NOT see a  'SharedVG' mpath device.
> (Such change needs to be reflected in initramdisk - so regeneration is
> needed)

Unfortunately we cannot use global_filter with current vdsm, sine it will
override vdsm filter, and vdsm will not be able to use shared storage.

We will consider switching to global_filter in future release. The reason we are
using filter is to allow an admin to reject certain devices from our filter.

Comment 76 Nir Soffer 2016-09-30 11:24:52 UTC
(In reply to Zdenek Kabelac from comment #74)
> And a final comment - IMHO such configured host system then could possibly
> use 'lvmetad' locally for locally available devices - since there shall be
> no interference.
> 
> Just vdsm commands needs to go with --config  lvmetad=0

This requires modifying global_filter, this is not compatible with current vdsm
(3.5, 3.6, 4.0).

Comment 77 Nir Soffer 2016-09-30 11:27:37 UTC
Fabian, maybe we should open a separate bug (RFE?) for node? I think it will be 
easier to get good filtering on node baked into node, before we can get a general
purpose solution for rhel or other systems.

Comment 78 Zdenek Kabelac 2016-09-30 11:43:18 UTC
Yes - if you seek solution without mods on vdsm side - then just skip 'global_filter' & 'locking_type=4' advise.

One has to just make sure that any host  is NOT masking mpath device needed by vdsm in its local lvm.conf file.

And since it's not possible to exclude vdsm mpath devices via global_filter user as well MAY NOT use lvmetad locally.

Comment 84 Raz Tamir 2016-11-12 14:08:01 UTC
Hi Nir,
Can we consider steps 1 - 6 from comment #48 as a steps to reproduce this bug?
If not, can you please provide it?

Comment 85 Nir Soffer 2016-11-16 16:22:38 UTC
The old patches were trying to cleanup after lvm during vdsm bootstrap.

The new patch <https://gerrit.ovirt.org/66893> is handling the root cause by
adding a global filter rejecting ovirt lvs.

The configuration is generic and should work on any host, assuming that only
ovirt uses uuids for vg names.

After testing this, we will work on installing this file during vdsm configuration.

Germano, can we test this configuration in some the relevant cases?

Comment 86 Nir Soffer 2016-11-16 19:53:48 UTC
(In reply to Raz Tamir from comment #84)
> Can we consider steps 1 - 6 from comment #48 as a steps to reproduce this
> bug?
Yes

Comment 87 Germano Veit Michel 2016-11-17 01:42:37 UTC
(In reply to Nir Soffer from comment #85)
> The old patches were trying to cleanup after lvm during vdsm bootstrap.
> 
> The new patch <https://gerrit.ovirt.org/66893> is handling the root cause by
> adding a global filter rejecting ovirt lvs.
> 
> The configuration is generic and should work on any host, assuming that only
> ovirt uses uuids for vg names.
> 
> After testing this, we will work on installing this file during vdsm
> configuration.
> 
> Germano, can we test this configuration in some the relevant cases?

Hi Nir,

Absolutely. 

We currently have a Sev4 case open, I can check with that customer is he wants to help testing. We can also use our own reproducer from comment #54 (re-installing with RHEL). Or even better, do both.

But first, I have two questions:

1) You want me to cherry-pick both new and old patches, not just the new right?
(all the gerrits attached to the bz into latest stable vdsm + any dependency)

Not sure if we should go for latest master, especially if we are asking for a customer to help testing.

2) Does that filter in the new patch needs to go into initrd as well?

Thanks,
Germano

Comment 88 Nir Soffer 2016-11-17 09:08:31 UTC
(In reply to Germano Veit Michel from comment #87)
> (In reply to Nir Soffer from comment #85)
> 1) You want me to cherry-pick both new and old patches, not just the new
> right?

No, just the new configuration. This is trivial to deploy on a customer machine
and compatible with any version of RHV.

> 2) Does that filter in the new patch needs to go into initrd as well?

It should, so guest lvs on RHV raw lvs are never active on the host.

If we find that this configuration is a good solution for this issue we will 
integrate this in vdsm-tool configure later.

Comment 89 Germano Veit Michel 2016-11-21 05:15:26 UTC
Hi Nir,

I added the global filter to both initrd's and etc's lvm.conf and the raw disks LVs are not activated in Host upon reboot. So it looks good.

I am not sure if this will work for Direct LUNs though (Roman's bug). As AFAIK they don't follow the regex you specified in the global filter. Still, it looks like a solid step forward.

Below is only the test data.

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160711.0.el7ev)

initrd:
[root@rhevh-2 ~]# dir=`mktemp -d` && cd $dir
[root@rhevh-2 tmp.NShI7bl4lg]# cp /run/initramfs/live/initrd0.img .
[root@rhevh-2 tmp.NShI7bl4lg]# mv initrd0.img initrd0.gz
[root@rhevh-2 tmp.NShI7bl4lg]# gunzip initrd0.gz
[root@rhevh-2 tmp.NShI7bl4lg]# cpio -i -F initrd0
236815 blocks
[root@rhevh-2 tmp.NShI7bl4lg]# cat etc/lvm/lvm.conf
global {
locking_type = 4
use_lvmetad = 0
}
devices {
global_filter = [ "r|^/dev/mapper/[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-.+|" ]
}

etc:
cat /etc/lvm/lvm.conf | grep global_filter | grep -v '#'
	global_filter = [ "r|^/dev/mapper/[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-.+|" ]

# lvs | awk -F' ' '{print $1,$3}' | egrep '\-wi\-a'
ids -wi-ao----
inbox -wi-a-----
leases -wi-a-----
master -wi-ao----
metadata -wi-a-----
outbox -wi-a-----
ids -wi-a-----
inbox -wi-a-----
leases -wi-a-----
master -wi-a-----
metadata -wi-a-----
outbox -wi-a-----
Config -wi-ao----
Data -wi-ao----
Logging -wi-ao----
Swap -wi-ao----

# dmsetup ls --tree
HostVG-Logging (253:69)
 `-2a802d0e800d00000p4 (253:4)
    `-2a802d0e800d00000 (253:0)
       `- (8:0)
603f851b--7388--49f1--a8cc--095557ae0a20-ids (253:18)
 `-360014380125989a100004000004c0000 (253:8)
    |- (8:80)
    `- (8:32)
HostVG-Swap (253:67)
 `-2a802d0e800d00000p4 (253:4)
    `-2a802d0e800d00000 (253:0)
       `- (8:0)
76dfe909--20a6--4627--b6c4--7e16656e89a4-inbox (253:28)
 `-360014380125989a10000400000480000 (253:7)
    |- (8:64)
    `- (8:16)
603f851b--7388--49f1--a8cc--095557ae0a20-master (253:20)
 `-360014380125989a100004000004c0000 (253:8)
    |- (8:80)
    `- (8:32)
HostVG-Data (253:70)
 `-2a802d0e800d00000p4 (253:4)
    `-2a802d0e800d00000 (253:0)
       `- (8:0)
603f851b--7388--49f1--a8cc--095557ae0a20-outbox (253:16)
 `-360014380125989a100004000004c0000 (253:8)
    |- (8:80)
    `- (8:32)
603f851b--7388--49f1--a8cc--095557ae0a20-metadata (253:15)
 `-360014380125989a100004000004c0000 (253:8)
    |- (8:80)
    `- (8:32)
603f851b--7388--49f1--a8cc--095557ae0a20-inbox (253:19)
 `-360014380125989a100004000004c0000 (253:8)
    |- (8:80)
    `- (8:32)
live-base (253:6)
 `- (7:1)
360014380125989a10000400000500000p2 (253:11)
 `-360014380125989a10000400000500000 (253:9)
    |- (8:96)
    `- (8:48)
76dfe909--20a6--4627--b6c4--7e16656e89a4-leases (253:26)
 `-360014380125989a10000400000480000 (253:7)
    |- (8:64)
    `- (8:16)
360014380125989a10000400000500000p1 (253:10)
 `-360014380125989a10000400000500000 (253:9)
    |- (8:96)
    `- (8:48)
76dfe909--20a6--4627--b6c4--7e16656e89a4-ids (253:27)
 `-360014380125989a10000400000480000 (253:7)
    |- (8:64)
    `- (8:16)
76dfe909--20a6--4627--b6c4--7e16656e89a4-metadata (253:24)
 `-360014380125989a10000400000480000 (253:7)
    |- (8:64)
    `- (8:16)
603f851b--7388--49f1--a8cc--095557ae0a20-leases (253:17)
 `-360014380125989a100004000004c0000 (253:8)
    |- (8:80)
    `- (8:32)
HostVG-Config (253:68)
 `-2a802d0e800d00000p4 (253:4)
    `-2a802d0e800d00000 (253:0)
       `- (8:0)
live-rw (253:5)
 |- (7:2)
 `- (7:1)
2a802d0e800d00000p3 (253:3)
 `-2a802d0e800d00000 (253:0)
    `- (8:0)
76dfe909--20a6--4627--b6c4--7e16656e89a4-master (253:29)
 `-360014380125989a10000400000480000 (253:7)
    |- (8:64)
    `- (8:16)
2a802d0e800d00000p2 (253:2)
 `-2a802d0e800d00000 (253:0)
    `- (8:0)
76dfe909--20a6--4627--b6c4--7e16656e89a4-outbox (253:25)
 `-360014380125989a10000400000480000 (253:7)
    |- (8:64)
    `- (8:16)
2a802d0e800d00000p1 (253:1)
 `-2a802d0e800d00000 (253:0)
    `- (8:0)

Comment 91 Germano Veit Michel 2016-12-21 06:40:22 UTC
Nir, we have a problem.

In a customer environment we deployed that latest filter from comment #85 and #89. I have checked both lvm.conf and initrd and the filter is there.

Problem: we still have Guest LVs activated. The following are all VM internal. Same story, RAW disk, VM PV directly on top of the Image LV.

MESSAGE=1 logical volume(s) in volume group "vg_cache_pulp" now active
MESSAGE=1 logical volume(s) in volume group "vg_pulp" now active
MESSAGE=1 logical volume(s) in volume group "vg_mongodb" now active
MESSAGE=1 logical volume(s) in volume group "vg_apps1" now active

cat etc/lvm/lvm.conf | grep global_filter | grep -v '#'
        global_filter = [ "r|^/dev/mapper/[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-.+|" ]

The lvm.conf other settings are RHEV-H default.

And this is in the initrd (extracted manually to confirm).

global {
locking_type = 4
use_lvmetad = 0
}
devices {
global_filter =  [ "r|^/dev/mapper/[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-.+|" ]
}

The filter does match (with egrep) the underlying device of vg_pulp-lv_pulp..

vg_pulp-lv_pulp (253:289)
 `-b442d48e--0398--4cad--b9bf--992a3e663573-0f902453--5caf--473f--8df7--f3c4a4c...
    `-36000144000000010706222888cc3684d (253:104)

Ideas? Is the initrd correct?

Comment 93 Germano Veit Michel 2016-12-21 06:43:28 UTC
By the way, it was on:

Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160711.0.el7ev)

Comment 94 Raz Tamir 2016-12-26 14:27:46 UTC
*** Bug 1326828 has been marked as a duplicate of this bug. ***

Comment 95 Raz Tamir 2016-12-26 14:27:54 UTC
*** Bug 1374549 has been marked as a duplicate of this bug. ***

Comment 96 Raz Tamir 2016-12-26 14:28:44 UTC
*** Bug 1398918 has been marked as a duplicate of this bug. ***

Comment 98 Nir Soffer 2016-12-28 19:39:54 UTC
(In reply to Germano Veit Michel from comment #91)
> In a customer environment we deployed that latest filter from comment #85
> and #89. I have checked both lvm.conf and initrd and the filter is there.

What is the difference between this environment and the one form comment 89
that seems to work?

Comment 99 Germano Veit Michel 2016-12-29 23:21:24 UTC
(In reply to Nir Soffer from comment #98)
> What is the difference between this environment and the one form comment 89
> that seems to work?

I can't see any big differences. Both are late 3.6 7.2 RHV-H with just the filter added. 

This one that it did not work is a real production environment, with many more disks though, possibly the boot sequence regarding the FC devices is slightly different (could timing be related?)

> locking_type = 4

I will fix this. You think it might be worth trying again?

Comment 103 Zdenek Kabelac 2017-01-03 10:21:44 UTC
So please provide  version of lvm2  in use here.

Please attach to this BZ lvm.conf.

Also please  attach debug trace from such system for this command:

pvscan -vvvv --aay --cache /dev/mapper/your_problematic_device 


(make sure LVs from this device are NOT active before running this command
(i.e.  deactive them with 'dmsetup remove xxxxxxx')

Comment 104 Germano Veit Michel 2017-01-09 03:51:07 UTC
Hi Zdenek,

(In reply to Zdenek Kabelac from comment #103)
> So please provide  version of lvm2  in use here.
lvm2-2.02.130-5.el7_2.5.x86_64
 
> Please attach to this BZ lvm.conf.
Attaching now.

> Also please  attach debug trace from such system for this command:
> 
> pvscan -vvvv --aay --cache /dev/mapper/your_problematic_device 
> 

I cannot ask the customer to run this, as it may activate everything, unless you can confirm nothing will be activated.

They just lost another VM over the holidays due to a RHV LSM over a disk affected by this BZ. Any other command that would help? 

If there isn't another option then I will have to ask them for a maintenance window in one of the hosts to perform this.

Thanks

Comment 106 Jonathan Earl Brassow 2017-01-13 16:02:49 UTC
Why is this bug in POST if it is unresolved?

Do we have an in-house reproducer for this issue so we can get the debug trace requested in comment 103?

Comment 107 Tal Nisan 2017-01-15 09:33:27 UTC
*** Bug 1412900 has been marked as a duplicate of this bug. ***

Comment 109 Zdenek Kabelac 2017-01-15 16:16:15 UTC
(In reply to Germano Veit Michel from comment #104)
> Hi Zdenek,
> 
> (In reply to Zdenek Kabelac from comment #103)
> > So please provide  version of lvm2  in use here.
> lvm2-2.02.130-5.el7_2.5.x86_64
>  
> > Please attach to this BZ lvm.conf.
> Attaching now.
> 
> > Also please  attach debug trace from such system for this command:
> > 
> > pvscan -vvvv --aay --cache /dev/mapper/your_problematic_device 
> > 
> 
> I cannot ask the customer to run this, as it may activate everything, unless
> you can confirm nothing will be activated.

We really cannot processed unless we see the exact trace of problematic behavior.

Try to capture full lvmdump.

And possibly system logs from the moment customer is noticing some unwanted activating.

> If there isn't another option then I will have to ask them for a maintenance
> window in one of the hosts to perform this.

Yep - we need to get the info - we can't guess..

Comment 110 Germano Veit Michel 2017-01-16 01:45:06 UTC
Hi Zdenek,

Thanks for the help. Two things:

1. We have a maintenance window to troubleshoot this with the customer. They have separated a Host to troubleshoot this and we created a test VM (with PV directly on top of Disk LV), all production VMs have been moved to PVs on top of partitions on top of Disk LV.

2. In our labs, I've got a similar system to the customers and I managed to confirm it's not working on FC (but it seems fine on iSCSI).

It looks to me the filter is fine, but there is still something wrong at boot time which scans activates these LVs. They seem to be activated on boot and then filtered. And it doesn't happen anymore if I deactivate them and scan again. This matches my iscsi/FC observation I guess, since iscsi storage attachment is at a much later stage.

So, this one is coming up active, and it shouldn't:

(lvs):
LV                                   VG                                   Attr       LSize                                                
baa40da9-d2e7-4b56-9e4d-7654bb854ec1 603f851b-7388-49f1-a8cc-095557ae0a20 -wi-ao----   1.00g

myvg is a VG on a PV directly on top of baa40da9, no partitions. This VG has 3 LVs, (lvol0, 1 and 2).

(dmsetup ls --tree):
myvg-lvol2 (253:81)
 `-603f851b--7388--49f1--a8cc--095557ae0a20-baa40da9--d2e7--4b56--9e4d--7654bb854ec1 (253:26)
    `-360014380125989a100004000004c0000 (253:8)
       |- (8:80)
       `- (8:32)
myvg-lvol1 (253:80)
 `-603f851b--7388--49f1--a8cc--095557ae0a20-baa40da9--d2e7--4b56--9e4d--7654bb854ec1 (253:26)
    `-360014380125989a100004000004c0000 (253:8)
       |- (8:80)
       `- (8:32)
myvg-lvol0 (253:79)
 `-603f851b--7388--49f1--a8cc--095557ae0a20-baa40da9--d2e7--4b56--9e4d--7654bb854ec1 (253:26)
    `-360014380125989a100004000004c0000 (253:8)
       |- (8:80)
       `- (8:32)

As expected, vdsm fails to deactivate it on boot, because of those 3 LVs:

# lvchange -an /dev/603f851b-7388-49f1-a8cc-095557ae0a20/baa40da9-d2e7-4b56-9e4d-7654bb854ec1 
  Logical volume 603f851b-7388-49f1-a8cc-095557ae0a20/baa40da9-d2e7-4b56-9e4d-7654bb854ec1 is used by another device.

/dev/myvg/lvol0 -> ../dm-79
/dev/myvg/lvol1 -> ../dm-80
/dev/myvg/lvol2 -> ../dm-81

Interestingly I don't see these 3 LVs on 'lvs'. Because the filter is actually working?

The command you requested always returns empty output (because of lvmetad?):

I've grabbed the following for you:
- etc/lvm.conf
- initrd lvm.conf
- lvm2-2.02.130-5.el7_2.5.x86_64
- kernel-3.10.0-327.22.2.el7.x86_64
- systemd-219-19.el7_2.11.x86_64
- Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160711.0.el7ev)
- pvscan -vvvv 
- pvscan -vvvv -aay --cache  (empty)
- pvscan -vvvv -aay --cache /dev/mapper/360014380125989a100004000004c0000 (empty)
- journalctl --all --this-boot --no-pager -o verbose (pvscan on 253:26?)

I also changed locking_type to 1 as per comment #97 and #99, just to confirm it's not related. Same results.

I'm still wondering what is difference between this and comment #89, I even used the same system and the outputs of the commands are clearly different from what I pasted at that time.

Any idea what is wrong? What else do you need to troubleshoot this?

Comment 112 Zdenek Kabelac 2017-01-16 10:27:34 UTC
So from logs in comment  111 -  there is  disabled  lvmetad - in which case the auto-activation is handled via lvm2-activation-early.service which calls:

'vgchange -aay --ignoreskippedcluster'

global_filter|filter  does CONTROL only  visibility of block device for lvm2 command.

Auto activation is DEFAULT behavior for any LV you have.
If users want to disable it completely - as been already said here in comment 10 - in lvm.conf use:

activation { auto_activation_volume_list = [] }

I'm really lost here what is the problem being solved:

1. Are we solving problem that LV is activating volumes from devices it should not see ?  (in this case we need trace from  'vgchange -aay -vvvv')


2. Or there is just misunderstanding how to control auto activation ?
(if it's unwanted it should be disabled by config option).

Comment 113 Germano Veit Michel 2017-01-17 00:48:53 UTC
(In reply to Zdenek Kabelac from comment #112)
> 1. Are we solving problem that LV is activating volumes from devices it
> should not see ?  (in this case we need trace from  'vgchange -aay -vvvv')
Yes, but on boot only.

Let me try to summarize and give it current status:

With current configuration (just the filter), after removing (dmsetup) deactivating (lvchange) all the unwanted volumes, vgchange -aay -vvvv don't seem to activate anything we don't want it to. I suppose this is because our filters are working fine here. This is good.

Our problem seems to be restricted to boot time. So are you saying lvm2-activation-early.service doesn't care about our filter?

(dmsetup ls --tree after boot):
myvg-lvol0 (253:79)
 `-603f851b--7388--49f1--a8cc--095557ae0a20-baa40da9.... (253:26)
    `-360014380125989a100004000004c0000 (253:8)
       |- (8:80)
       `- (8:32)

After boot all LVs are active and vdsm is supposed to deactivate them. But vdsm fails to deactivate a few ones[1]. In the case above, myvg-lvol0 prevents 603f851b from being deactivated by vdsm on startup. We don't want that myvg-lvol0 to be found or activated.

[1] ones that have a PV/VG directly on top of them (no partitions), which were scanner and had internal LVs activated.

As you found out, we actually tried auto_activation_volume_list = [] in a previous version of the solution, but AFAIK it's not present in the latest fix, that's why I did not include it.

I have now added auto_activation_volume_list = [] to both initrd and lvm.conf and it seems to be working! No unwanted LVs are active and it did not find that myvg-lvol0. Once Nir confirms this might be the fix, I will forward it to one of our customers to give it another round of testing.

@Nir, don't we also need auto_activation_volume_list = [] in https://gerrit.ovirt.org/66893? See Zdenek comment #112 above.

Comment 114 Zdenek Kabelac 2017-01-18 09:44:20 UTC
I do think there are couple 'terms' mixed together and the bad results are taken out of it.

So from 'comment 113'  - let me explain:

lvm2 filters do serve only the purpose to 'hide'  devices from being SEEN by lvm2 command -  so the volumes which cannot be seen cannot be used for i.e. activation or listing or....

Auto-activation is a feature which is  turned ON  by default for LVs.
If user  does not like/does not want it - lvm2 must be configured to not do it (it cannot guess it).

What I can see here is likely the customer does not want auto-activation - so he must configure lvm.conf to disable it.


Now the interesting bit is - out of where 'vgchange -aay' got into  ramdisk.
AFAIK this command is executed AFTER  ramdisk switches to /rootfs.
Standard RHEL dracut boot process does ONLY activate  rootLV and shall not activate anything else - unless instrumented on grub config line.

So since here we are being told the  initramdisk lvm.conf  must have been also configured to NOT make any auto activation - it's quite weird. Since such thing should NOT really be needed - nothing should be using auto activation from ramdisk - unless there has been some  nonstandard 'tweaking'  of ramdisk boot sequence!

In such case - please attach 'ramdisk' in-use and the history how it's been created...

Comment 115 Nir Soffer 2017-01-18 09:57:38 UTC
(In reply to Zdenek Kabelac from comment #114)
> I do think there are couple 'terms' mixed together and the bad results are
> taken out of it.
> 
> So from 'comment 113'  - let me explain:
> 
> lvm2 filters do serve only the purpose to 'hide'  devices from being SEEN by
> lvm2 command -  so the volumes which cannot be seen cannot be used for i.e.
> activation or listing or....
> 
> Auto-activation is a feature which is  turned ON  by default for LVs.
> If user  does not like/does not want it - lvm2 must be configured to not do
> it (it cannot guess it).

Zdenek, are you saying that having a filter hiding a device, will *not* prevent
activation of the device and other devices inside this device? This does not
make sense.

Can you point me to the documentation specifying this behavior?

> What I can see here is likely the customer does not want auto-activation -
> so he must configure lvm.conf to disable it.

But we cannot auto_activation_list, since we vdsm cannot guess what are
the host devices, it knows only what are ovirt devices.

auto_activation_list does not support excluding, only including.
If it will support exclusion, e.g. (r"@RHT_STORAGE_DOMAIN") we
can use it to prevent auto activation.
 
> Now the interesting bit is - out of where 'vgchange -aay' got into  ramdisk.
> AFAIK this command is executed AFTER  ramdisk switches to /rootfs.
> Standard RHEL dracut boot process does ONLY activate  rootLV and shall not
> activate anything else - unless instrumented on grub config line.
> 
> So since here we are being told the  initramdisk lvm.conf  must have been
> also configured to NOT make any auto activation - it's quite weird. Since
> such thing should NOT really be needed - nothing should be using auto
> activation from ramdisk - unless there has been some  nonstandard 'tweaking'
> of ramdisk boot sequence!
> 
> In such case - please attach 'ramdisk' in-use and the history how it's been
> created...

Comment 116 Nir Soffer 2017-01-18 10:19:47 UTC
(In reply to Germano Veit Michel from comment #113)
> @Nir, don't we also need auto_activation_volume_list = [] in
> https://gerrit.ovirt.org/66893? See Zdenek comment #112 above.

We cannot, this will prevent activation of lvs needed by the host. When I tested
this, vg0-lv_home was not activated after boot.

This can be configured with the host lvs (or using tags) on user machine manually
as a temporary workaround until we find a better solution. For example:

    auto_activation_volume_list = ["vg0"]

This should probably update the support article about configuring hypervisors
with FC storage.

Comment 117 Zdenek Kabelac 2017-01-18 10:23:13 UTC
(In reply to Nir Soffer from comment #115)
> (In reply to Zdenek Kabelac from comment #114)
> > I do think there are couple 'terms' mixed together and the bad results are
> > taken out of it.
> > 
> > So from 'comment 113'  - let me explain:
> > 
> > lvm2 filters do serve only the purpose to 'hide'  devices from being SEEN by
> > lvm2 command -  so the volumes which cannot be seen cannot be used for i.e.
> > activation or listing or....
> > 
> > Auto-activation is a feature which is  turned ON  by default for LVs.
> > If user  does not like/does not want it - lvm2 must be configured to not do
> > it (it cannot guess it).
> 
> Zdenek, are you saying that having a filter hiding a device, will *not*
> prevent
> activation of the device and other devices inside this device? This does not
> make sense.


You are misreading my comment.

Hiding device will OF COURSE avoid auto activation of ANY LV on such device.
Command will simply not see such VG so cannot activate LV on it.

But the auto activated devices  presented in multiple comments in this BZ were clearly activated from devices (PVs)  which have  passed through filters.

So either filters are wrong - or user has 'devices' which are not excluded from filtering and thus are candidates for auto activation.

> 
> > What I can see here is likely the customer does not want auto-activation -
> > so he must configure lvm.conf to disable it.
> 
> But we cannot auto_activation_list, since we vdsm cannot guess what are
> the host devices, it knows only what are ovirt devices.

Your  'vdsm' commands simply have to be running  with any auto activation disabled (assuming  this logic is unwanted for you vdsm master server) +  filtering out all users  devices.

While  users local guest commands must filter our all  vdsm devices.

Comment 118 Zdenek Kabelac 2017-01-18 10:35:21 UTC
Here could be an idea - 

lvm2 fully supports  'separate' configuration with envvar - 
so you could your own  /etc/lvm directory set via LVM_SYSTEM_DIR
(see  man lvm.8)

So all commands running via 'vdsm' remotely may 'enjoy' your special prepared configuration which must  'white-list' only your storage devices
and reject everything else.

While  the guest/vdsm-slave (well not sure what terminology is used here) is continuing using his OWN /etc/lvm config.  The ONLY mandatory condition he needs to ensure is -  he will reject/exclude/not touch/not see/....  ANY vdsm device.
So admin must 'update' his local lvm.conf (unavoidable).

So when such user runs lvm2 command on his box, he will NEVER see vdsm LVs.

When 'vdsm' runs its commands on user's box (With its own $LVM_SYSTEM_DIR) - it will NEVER see/touch/interact with any user local disks.

Is this making it more clear?

Comment 119 Nir Soffer 2017-01-18 10:56:45 UTC
(In reply to Zdenek Kabelac from comment #117)
> (In reply to Nir Soffer from comment #115)
> > (In reply to Zdenek Kabelac from comment #114)
> > > I do think there are couple 'terms' mixed together and the bad results are
> > > taken out of it.
> > > 
> > > So from 'comment 113'  - let me explain:
> > > 
> > > lvm2 filters do serve only the purpose to 'hide'  devices from being SEEN by
> > > lvm2 command -  so the volumes which cannot be seen cannot be used for i.e.
> > > activation or listing or....
> > > 
> > > Auto-activation is a feature which is  turned ON  by default for LVs.
> > > If user  does not like/does not want it - lvm2 must be configured to not do
> > > it (it cannot guess it).
> > 
> > Zdenek, are you saying that having a filter hiding a device, will *not*
> > prevent
> > activation of the device and other devices inside this device? This does not
> > make sense.
> 
> 
> You are misreading my comment.
> 
> Hiding device will OF COURSE avoid auto activation of ANY LV on such device.
> Command will simply not see such VG so cannot activate LV on it.
> 
> But the auto activated devices  presented in multiple comments in this BZ
> were clearly activated from devices (PVs)  which have  passed through
> filters.
> 
> So either filters are wrong - or user has 'devices' which are not excluded
> from filtering and thus are candidates for auto activation.

OK, thanks for clarifying this, we will debug the filter.

> > > > > What I can see here is likely the customer does not want auto-activation -
> > > so he must configure lvm.conf to disable it.
> > 
> > But we cannot auto_activation_list, since we vdsm cannot guess what are
> > the host devices, it knows only what are ovirt devices.
> 
> Your  'vdsm' commands simply have to be running  with any auto activation
> disabled (assuming  this logic is unwanted for you vdsm master server) + 
> filtering out all users  devices.
> 
> While  users local guest commands must filter our all  vdsm devices.

We don't have any issue in vdsm, we are using --config devices/filter including
only the shared storage relevant for the operation, and rejecting anything else.

(In reply to Zdenek Kabelac from comment #118)
> Here could be an idea - 
> 
> lvm2 fully supports  'separate' configuration with envvar - 
> so you could your own  /etc/lvm directory set via LVM_SYSTEM_DIR
> (see  man lvm.8)
> 
> So all commands running via 'vdsm' remotely may 'enjoy' your special
> prepared configuration which must  'white-list' only your storage devices
> and reject everything else.
> 
> While  the guest/vdsm-slave (well not sure what terminology is used here) is
> continuing using his OWN /etc/lvm config.  The ONLY mandatory condition he
> needs to ensure is -  he will reject/exclude/not touch/not see/....  ANY
> vdsm device.
> So admin must 'update' his local lvm.conf (unavoidable).

This is the issue, how to configure the system automatically.

To add a host to RHV system, you click "add host", fill in the host address
and root password, and click "ok". The system does the rest for you.

We don't want to require manual configuration of each host in the cluster.

We don't ask lvm to guess, only provide the necessary configuration options that
will allow vdsm to automatically configure the host in safe way.

> So when such user runs lvm2 command on his box, he will NEVER see vdsm LVs.
> 
> When 'vdsm' runs its commands on user's box (With its own $LVM_SYSTEM_DIR) -
> it will NEVER see/touch/interact with any user local disks.
> 
> Is this making it more clear?

We are using --config for this, maybe using our own lvm directory is a better
idea, will keep the vdsm commands more clear. 

Here is example vdsm command (taken from vdsm.log):

    lvchange --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/360014057ce1a1afffd744dc8c34643d7|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --refresh 40825394-bb03-4b66-8a82-e6ddbb789ec3/leases

Comment 120 Nir Soffer 2017-01-18 11:04:07 UTC
Germano, please check my reply in comment 116.

Comment 121 Nir Soffer 2017-01-18 11:29:56 UTC
Germano, please also provide the info requested by Zdenek in comment 114.

Comment 122 Nir Soffer 2017-01-18 11:41:14 UTC
Updating bug name, this is not about auto activation of ovirt lvs, but about auto
activation of guest lvs inside ovirt raw lvs.

We want to solve also the activation of ovirt lvs, the current solution we have,
deactivating them in vdsm during startup is a workaround, but this issue is not
in the scope of this bug.

Comment 123 Zdenek Kabelac 2017-01-18 12:08:31 UTC
So let's make clear further more:

Auto activation is the feature which is running from udev rules using 'pvscan -aay....' command when 'new' device appears.

You will NOT get auto activation from lvm2 command by some 'magical' autonomous daemon. 

So if your guest system has devices auto-activated from your  'vdsm' active volumes - then such guest system is not filtering out properly vdsm devices.

i.e. you activate vdsm LV - and such device is scanned by udev and pvscan can see it via  /dev/dm-XXX device.


This can be rather very easily 'verified' - when 'guest' runs a simple command: 'pvs -a'  - it MUST NOT report any of your 'vdsm' PV volumes  and it also MUST NOT report any of your activate volume/LV from vdsm system.

If you can see some  /dev/XXX/YYY  listed which belongs to 'vdsm'
the filter is set wrong.

And here I'm always 'advocating' for 'white-list' filter logic - which is IMHO the most human understandable.

Means - you list explicitly devices you want to see on guest's lvm.conf (where using filters like a|/dev/mapper/*|  is really BAD idea)  and then you reject everything else (r|.*| being the last rule after  'a' rules first)

NB: creating correctly working reject/(aka black-list) filter is way more  complicated as you would have to preserve some 'strict' naming convention rules within vdsm.

Comment 124 Germano Veit Michel 2017-01-19 05:28:14 UTC
Hi Zdenek and Nir!

Right, I think I have digested most of your feedback.

After reading your comments I decided to go for 9 different config tests on the system I was previously using to run these tests (RHEV-H). I ran them (several variations of initrd and root/lvm.conf) and got some very confusing results, which wouldn't make sense. Then I decided to repeat them and to my surprise many configs gave different results when compared to the first round!

I have no idea what causes this, maybe due to the way etc/lvm/lvm.conf is "written" in RHEV-H? It's actually mounted there (this is a Read-Only RHEL based appliance):

# mount | grep lvm.conf
/dev/mapper/HostVG-Config on /etc/lvm/lvm.conf type ext4 (rw,noatime,seclabel,data=ordered)

This was on:

- RHEV-H 7.2 (20160711.0.el7ev)
- kernel-3.10.0-327.22.2.el7.x86_64
- lvm2-2.02.130-5.el7_2.5.x86_64
- systemd-219-19.el7_2.11.x86_64

My conclusion on RHEV-H.
- There is some weird behavior or maybe a race condition on RHEV-H. Maybe sometimes lvm.conf is mounted before vgchange -aay, sometimes after? Sometimes the filter works and sometimes not? Most of the times it doesn't. I'm sorry but I'm deeply confused on what is happening on RHEV-H. I can't get consistent results from it. Most of the times it seems the filter doesn't work on boot.

Then I gave up on that RHEV-H and I moved to RHEL. I got consistent results and also in line with what Zdenek is saying. And everything is working just fine just with Nir's global_filter. So there is indeed something different on RHEV-H.

- RHEL 7.2
- kernel-3.10.0-327.el7.x86_64
- lvm2-2.02.130-5.el7_2.5.x86_64
- systemd-219-19.el7.x86_64

Test 10------------------------------------------------------------------------
lvm.conf:
  auto_activation_volume_list = [ "rhel_rhevh-1" ]
  global_filter =  [ "r|^/dev/mapper/[a-f0-9][a-f0-9].....
initrd:
  auto_activation_volume_list = [ "rhel_rhevh-1" ]
  global_filter =  [ "r|^/dev/mapper/[a-f0-9][a-f0-9].....

Result = GOOD

Test 11------------------------------------------------------------------------
lvm.conf:
  global_filter =  [ "r|^/dev/mapper/[a-f0-9][a-f0-9].....
initrd:
  global_filter =  [ "r|^/dev/mapper/[a-f0-9][a-f0-9].....

Result = GOOD

Test 12------------------------------------------------------------------------
lvm.conf:
  global_filter =  [ "r|^/dev/mapper/[a-f0-9][a-f0-9].....
initrd:
  default

Result = GOOD

Test 13------------------------------------------------------------------------
lvm.conf:
  auto_activation_volume_list = [ "rhel_rhevh-1" ]
  global_filter =  [ "r|^/dev/mapper/[a-f0-9][a-f0-9].....
initrd:
  default

Result = GOOD.

Test 14------------------------------------------------------------------------
lvm.conf:
  auto_activation_volume_list = [ "rhel_rhevh-1" ]
initrd:
  default

Result = GOOD.

Conclusion and notes on RHEL:
- 14 was just for fun :), doesn't sound like a good idea as the ovirt LVs are not filtered.
- Consistent results after rebooting many times (RHEL)
- The filter seems to be enough to prevent activation of Guest LVs on RAW ovirt LVs, which is the aim of this bug. So I think it's indeed solved - on RHEL.

And Zdenek is correct, auto_activation_volume_list on initrd makes no difference.

It looks like we are all good on RHEL. But on RHEV-H things are not that simple. 
Hopefully RHVH 4.0 behaves the same way as RHEL.

TODO:
- Find a solution for legacy RHEV-H
    * maybe inject Nir's filter on root/lvm.conf in the base image?
- Check behavior on RHVH 4

Comment 125 Germano Veit Michel 2017-01-19 05:33:56 UTC
@Fabian

1. Is there any easy way to modify RHEV-H (legacy) appliance to inject a custom lvm.conf? I want to run a test without the persist mechanism. 

2. I also found we have two lvm.conf in the squashfs. /etc/lvm.conf and /etc/lvm/lvm.conf, not sure if this is correct.

3. Do you have any idea at what time (during boot) the HostVG-Config files are mounted? I couldn't find it from the journal logs.

Comment 127 Nir Soffer 2017-01-19 11:33:22 UTC
Zdenek, base on comment 124, do we really need to include the lvmlocal.conf
in initrd?

We are using these settings:

global {
    use_lvmetad = 0 
}

devices {
    global_filter = [ "r|^/dev/mapper/[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9]--[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-.+|" ]
}

Comment 128 Zdenek Kabelac 2017-01-19 12:51:47 UTC
Hi Nir


It doesn't really matter where you place the filter - it gets all merged.

The important thing here is -  the setting is REPLACING previous setting in this configuration merging (overriding).

So when user has set some 'global_filter'  in his  lvm.conf - and then you 'just override' such filter in   lvmlocal.conf   - then  you ensure your matching device gets filter out BUT all previously preset filters by user in his lvm.conf are simply gone/lost/vanished.

There is no such thing as 'combing' setting' or applying filter in chain.

It's always - the latest setting wins.

The purpose of 'lvmlocal.conf' is different and mainly targeted for cluster where you wand to set local difference but the 1 single admin has still full control over all nodes and there is not some  dedicated 'local life' on cluster nodes.


So I'd here rather avoid using lvmlocal.conf - and rather always properly configure lvm.conf.


The next important issue which might deserve explicit sentence -  there is NOT any 'transitivity' applied on filters.

So if you 'filter-out'  your  multipath device for local usage - but  'vdsm' then activates some LV from this multipath device (with --config on cmdline) then such device becomes visible local -  and since your  filter is excluding  PURELY  a /dev/mapper/some-special-number-device   then all other DM devices will be locally process  (so auto activate normally applies to them)

In general you are stacking VG on top of another VG - there is NO support in lvm2 to handle this - we  are not stopping users to use such configuration - but we are not 'advertising' such stacking either - as it has many complex issues which are ATM out-of-scope for single lvm2 commands.
So if skilled admin can handle rough edges - fine - but it needs very careful handle otherwise there are unavoidable deadlocks.

Comment 130 Zdenek Kabelac 2017-01-19 13:27:42 UTC
Technically I've forget to explicitly answer the question whether there is need of lvmlocal.conf inside  initramdisk.


If the lvm command executed inside ramdisk shall not see such device - and such device is already present in ramdisk  (assuming it's multipath started in ramdisk)  - then yes  - needs to be there.

But as said in comment 128 - it's likely better to 'combine' filter settings in a single file  lvm.conf  which is then copied into ramdisk.


As a useful hint lvm2 provides command 'lvmconfig' (man lvmconfig)

So at anytime you could 'query' what will be some lvm2 setting set to.

I.e. in dracut you can use rd.break=pre-mount and check:

'lvm lvmconfig  devices/global_filter'

Comment 131 Nir Soffer 2017-01-19 13:47:42 UTC
(In reply to Zdenek Kabelac from comment #128)
> So I'd here rather avoid using lvmlocal.conf - and rather always properly
> configure lvm.conf.

But configuring lvm.conf means that changes to lvm.conf during upgrades
will never be applied, and the system administrator will have merge
the conf changes manually using lvm.conf.rpmnew files.

Also this means we have to modify existing lvm.conf. There is no way to
do that in a clean way with a proper comments. I tried to use augtool
to edit lvm.conf, but it seems impossible to have your edit in the
write place in the file.

So this leave the option to replace the entire file with a new file as we
do for /etc/multipath.conf. This means that vdsm will have follow every change
in lvm.conf, and include a new file each time lvm change this file. I don't
want this dependency in vdsm.

Using lvmlocal.conf, we can override in a clean way only the changes that
matter to us, and we can document properly each override.
See https://github.com/oVirt/vdsm/blob/master/static/usr/share/vdsm/lvmlocal.conf

So I don't plan to touch lvm.conf, and I expect lvm to use lvmlocal.conf as
documented in lvm.conf(5).

> The next important issue which might deserve explicit sentence -  there is
> NOT any 'transitivity' applied on filters.
>
> So if you 'filter-out'  your  multipath device for local usage - but  'vdsm'
> then activates some LV from this multipath device (with --config on cmdline)
> then such device becomes visible local -  and since your  filter is
> excluding  PURELY  a /dev/mapper/some-special-number-device   then all other
> DM devices will be locally process  (so auto activate normally applies to
> them)

We don't filter out multipath devices, we filter out ovirt-lvs, see
See https://github.com/oVirt/vdsm/blob/master/static/usr/share/vdsm/lvmlocal.conf

> In general you are stacking VG on top of another VG - there is NO support in
> lvm2 to handle this - we  are not stopping users to use such configuration -
> but we are not 'advertising' such stacking either - as it has many complex
> issues which are ATM out-of-scope for single lvm2 commands.
> So if skilled admin can handle rough edges - fine - but it needs very
> careful handle otherwise there are unavoidable deadlocks.

We are not stacking vg on top of another vg - it is the guest vm. This may
happen on raw volumes when the guest vm is using the raw volume
as a pv.

The flow is:

1. run a vm with a raw block volume (/dev/vgname/lvname)
2. in the guest admin uses the volume e.g. /dev/vdb as a pv,
    adding it a a vg and creating lvs
3. on the host, lvm scan the vm volume, find the new pv/vg/lvs
    and activate them.

Comment 132 Zdenek Kabelac 2017-01-19 14:10:50 UTC
There needs to be 'someone'  to combine filters. Otherwise when you just basically 'replace'  user's filter setting with yours - then previously working user commands starts to fail as filter from lvm.conf is not applied and is overruled by lvmlocal.conf  setting.


So effectively what you do with lvmlocal.conf  could be equally well provided by 'sed' scripting where you replace one setting in lvm.conf with your new setting.


As said - there is no cascade of filters applied from different config files - it's always one single value for setting which is applied.

---

Now to explain stacking issue - when you  'pass'  /dev/vgname/lvname to guest VM running on this vdsm guest system - the content of this LV is still FULLY visible on guest.

Nothing stops you can to do i.e. 'dd' on such device while it's in use by guest VM  (and cause some major discrepancy and data instability issue for VM for obvious reason...)

With the very same logic when 'guestVM' will create a VG & LV inside on it's /dev/vdb  device  - you could easily see such  'stacked' VG on your vdsm guest machine.

That's why it is important on your vdsm guest to precisely filter out such device. Lvm2 itself is not guessing disk hierarchy and not estimating which devices it should see or not see.

So back to your filter in comment 127 - it will filter out likely your uniquely named multipath device - but it will accept everything else (so very other i.e. /dev/dm-XXX  except your multipath will PASS)

So when you run on vdsm guest  lvm2 command - it will not read data out of your multipath device - but there is nothing  stopping lvm2 from reading data out of  /dev/vgname/lvname running on top if this device.

Probably repeating myself here again - but lvm2 is not doing any 'device stack' in-depth analysis - so it has no idea  that  /dev/vgname/lvname has backend on multipath device which you filtered out  by the explicit reject filter.

So with such filter settings -  if on the guest  inside  guestVM someone makes a VG + LV -  such LV could be then seen and auto/activated on guest -  I'm almost 99.999% you do not want this (unless guestVM is killed and you want to inspect content)

(Hopefully I still remember properly your description  of 'vdms' master system and satellite  guest systems)

Comment 133 Nir Soffer 2017-01-19 19:11:47 UTC
Created attachment 1242551 [details]
Script for generating lvm filter

Comment 134 Nir Soffer 2017-01-19 19:19:59 UTC
Germano, can you test the attached script (attachment 1242551 [details]) for generating lv
filter automatically?

Example usage - on a hypervisor:
    
    # ./gen-lvm-filter
    filter = [ "a|^/dev/vda2$|", "r|.*|" ]

This filter replaces the blacklist filter suggested earlier. Using a whitelist
we prevent:
- auto activation of lvs on shared storage
- auto activation of guest lvs (since lv cannot access ovirt lvs)
- accidental access of shared storage on a hypervisor

Using this should also improve boot time of hypervisors with a lot of devices,
storage domains and volumes.

See also https://gerrit.ovirt.org/70899 if you have ideas how to imporve this.

The generated filter may not be good enough, we need to test it in the field 
before we use it as part of vdsm configuration.

Comment 135 Germano Veit Michel 2017-01-20 02:44:11 UTC
Hi Nir,

Sounds like the best idea so far. 

Not sure if you want this to work on 4.0 node, I suppose you do as now the user can have some sort of customization over the lvm layout. I added some comments on gerrit about it.

I'm having trouble to run this on 3.6 Hosts. The import from vdsm is failing on 'commands'. I copied it over and then it started failing o compat/CPopen. Our reproducers are currently on 3.6 and the customers who are willing to help are also on 3.6. Can we have a version (or make sure the same code) runs on 3.6 too?

I ran it on a 4.0 host attached to iSCSI and it seems to work fine, but it's a simpler case then those we have on 3.6

Comment 136 Germano Veit Michel 2017-01-20 02:51:03 UTC
@Fabian please see comment #125.
@Nir please see comment #135.

Comment 137 Fabian Deutsch 2017-01-20 08:10:39 UTC
(In reply to Germano Veit Michel from comment #125)
> @Fabian
> 
> 1. Is there any easy way to modify RHEV-H (legacy) appliance to inject a
> custom lvm.conf? I want to run a test without the persist mechanism. 

Not easy, but possible.

The initrd is stored in a writable partition so you can modify or regenerate it.

> 2. I also found we have two lvm.conf in the squashfs. /etc/lvm.conf and
> /etc/lvm/lvm.conf, not sure if this is correct.

That's a good question. I am not sure why there are two. needs to be investigated.

> 3. Do you have any idea at what time (during boot) the HostVG-Config files
> are mounted? I couldn't find it from the journal logs.

Pretty early - What are you looking for?

For all questions I'd suggets to reach out to dougsland@ in an email and CC sbonazzo@ and myself.

Comment 138 Nir Soffer 2017-01-20 12:44:59 UTC
(In reply to Germano Veit Michel from comment #135)
Germano, thanks for testing this!

Latest version should work with any rhev version on rhel 7.

Comment 139 Nir Soffer 2017-01-20 12:46:24 UTC
Created attachment 1242784 [details]
Script for generating lvm filter

This version remove the unneeded dependencies on vdsm code that limit the usage to 4.1.

Comment 140 Germano Veit Michel 2017-01-23 06:23:07 UTC
Thanks Nir!

I added two more details on gerrit.

I fixed those with a "dumb patch" just to be able to run on our reproducer, it it gives the correct filter for that host, which is great:

# ./gen-lvm-filter 
filter = [ "a|^/dev/mapper/2a802d0e800d00000p4$|", "r|.*|" ]

# pvs
  PV                                                     VG                                   Fmt  Attr PSize    PFree   
  /dev/mapper/2a802d0e800d00000p4                        HostVG                               lvm2 a--    60.00g  404.00m
  /dev/mapper/360014380125989a10000400000480000          76dfe909-20a6-4627-b6c4-7e16656e89a4 lvm2 a--    99.62g    5.38g
  /dev/mapper/360014380125989a100004000004c0000          603f851b-7388-49f1-a8cc-095557ae0a20 lvm2 a--    99.62g   79.50g

I think that 2a802d0e800d00000p2 is a quite specific problem (EFI) and the live-rw only affects vintage RHEV-H. But nothing prevents someone from hitting similar issues on RHEL with custom layouts so I guess we need to fix them anyway. Once we have this fixed I'll forward to a customer who is willing to help.

Comment 142 Nir Soffer 2017-01-23 12:50:44 UTC
Created attachment 1243580 [details]
Script for generating lvm filter

Comment 143 Nir Soffer 2017-01-23 13:59:22 UTC
Created attachment 1243605 [details]
Script for generating lvm filter

Comment 144 Nir Soffer 2017-01-23 14:04:15 UTC
(In reply to Germano Veit Michel from comment #140)
Please test next version.

I simplified the way devices are found - we don't look now at mounted filesystems,
but instead we find all vgs which are not ovirt vgs (do not have RHAT_storge_domain
tag), and we include the devices included by these vgs in the filter, except ovirt
lvs (used as pv on a guest).

I think this should be pretty safe, limiting the chance of breaking esoteric user
setup.

After this filter is applied, if the user want to add a new device, the device must
be added manually to the filter.

Comment 145 Germano Veit Michel 2017-01-24 04:33:32 UTC
Nir, thanks again. I agree the new logic is safer, but it looks like we still have problems. 

1) The regex is not working anymore. Please check my comment on gerrit for more details. 

I am also attaching here the output of the vgs command in an affected host to help you develop this. This host is our reproducer and its got 2 ovirt LVs as PVs.

2) Not sure how we will handle Direct LUNs, but currently it seems to be a problem (not sure if you want to handle this later though). In the attached vgs you will see /dev/mapper/360014380125989a10000400000500000, that's the "Direct LUN". In fact its not really a direct LUN on RHEV, this is a blade system which sees the other blade LUN. But from our perspective it could well be a direct lun for RHV that is presented via FC. We need to find out it's not in used and filter it out too I guess.

Hopefully this helps.

Comment 146 Germano Veit Michel 2017-01-24 04:34:25 UTC
Created attachment 1243821 [details]
vgs -o vg_name,pv_name --noheading --select 'vg_tags != {RHAT_storage_domain}'

output of command for debugging/developing

Comment 148 Otheus 2017-01-27 11:54:49 UTC
> After boot all LVs are active and vdsm is supposed to deactivate them. But
> vdsm fails to deactivate a few ones[1]. In the case above, myvg-lvol0
> prevents 603f851b from being deactivated by vdsm on startup. We don't want
> that myvg-lvol0 to be found or activated.
> 
> [1] ones that have a PV/VG directly on top of them (no partitions), which
> were scanner and had internal LVs activated.

As it is, the analysis of this problem is incomplete. Why should vdsm fail to deactivate VGs on raw disks, as opposed to those on parititoned disks? To wit, are you certain that putting VGs on a partition is an actual work-around, as opposed to a happy accident?

Comment 149 Germano Veit Michel 2017-01-29 23:07:35 UTC
(In reply to Otheus from comment #148)
> As it is, the analysis of this problem is incomplete. Why should vdsm fail
> to deactivate VGs on raw disks, as opposed to those on parititoned disks?
> To wit, are you certain that putting VGs on a partition is an actual
> work-around, as opposed to a happy accident?

First of all vdsm doesn't deactivate VGs, it deactivates some ovirt LVs. The partitioned disks prevent auto activation of Guests LVs internal to the ovirt LVs which are raw disks because the internal VGs are not found. This allows vdsm to deactivate the ovirt LVs as intended.

Comment 150 Otheus 2017-01-30 11:38:04 UTC
(In reply to Germano Veit Michel from comment #149)
> ... The
> partitioned disks prevent auto activation of Guests LVs internal to the
> ovirt LVs which are raw disks because the internal VGs are not found.

This smells like danger. I admit there's a lot I don't understand here, but this I understand the least: how is it "internal VGs" (you mean, internal to a Guest VM, right?) are NOT found on partitioned disks that are (at LVM2-scan time) visible to the (host) system? 

> This allows vdsm to deactivate the ovirt LVs as intended.

Sorry, but B does not logically follow from A. If somehow a partitioned disk prevents auto-activation of an LV, then vdsm cannot deactivate it -- it was never activated. However, if vdsm is deactivating LVs of VGs that came from partitioned disks, as opposed to VGs that came from non-partitioned disks, I ask: why/how the hell does it do one but not the other? 

Regardless of the previous question, I think the first is the most pertinent.

Comment 151 Germano Veit Michel 2017-01-31 07:13:44 UTC
(In reply to Otheus from comment #150)
> This smells like danger. I admit there's a lot I don't understand here, but
> this I understand the least: how is it "internal VGs" (you mean, internal to
> a Guest VM, right?) are NOT found on partitioned disks that are (at
> LVM2-scan time) visible to the (host) system? 

From what I understand it simply finds a partition table signature and skips the device.

> Sorry, but B does not logically follow from A. If somehow a partitioned disk
> prevents auto-activation of an LV, then vdsm cannot deactivate it -- it was
> never activated. However, if vdsm is deactivating LVs of VGs that came from
> partitioned disks, as opposed to VGs that came from non-partitioned disks, I
> ask: why/how the hell does it do one but not the other? 

I think you did not understand the bug. Maybe because most of the comments are private. 

The partitioned Guest disk prevents the host from discovering AND activating the internal guest LVs. Regardless of the Guest LVM, the ovirt LVs are always activated on block storage on connection/boot, and vdsm deactivates the ovirt LVs when it starts. Now, if a guest LV was scanned and is active vdsm will always fail to deactivate the ovirt LV because it's open. This is our problem as it leads to numerous other issues.

I suggest you to read our customer portal article [1], especially the diagnostic steps which shows what holds what from being deactivated.

[1] https://access.redhat.com/solutions/2662261

Comment 152 Nir Soffer 2017-01-31 07:34:03 UTC
(In reply to Otheus from comment #148)
> > After boot all LVs are active and vdsm is supposed to deactivate them. But
> > vdsm fails to deactivate a few ones[1]. In the case above, myvg-lvol0
> > prevents 603f851b from being deactivated by vdsm on startup. We don't want
> > that myvg-lvol0 to be found or activated.
> > 
> > [1] ones that have a PV/VG directly on top of them (no partitions), which
> > were scanner and had internal LVs activated.
> 
> As it is, the analysis of this problem is incomplete. Why should vdsm fail
> to deactivate VGs on raw disks, as opposed to those on parititoned disks? To
> wit, are you certain that putting VGs on a partition is an actual
> work-around, as opposed to a happy accident?

Hi Otheus,

When a guest is using ovirt raw volume, the guest see the same device seen by
the host. If the guest create a pv on the lv directly, both the guest and the host
are seeing a device with a pv header.

It looks like this (filtered to show the interesting items):

# lvs -o vg_name,lv_name,devices,attr
  VG                                   LV                                   Devices                                                                             Attr      
  7df95b16-1bd3-4c23-bbbe-b21d403bdcd8 e288d725-f620-457e-b9e9-ea8a4edf89c4 /dev/mapper/360014052c81462a280847e8a6e3af8cd(239)                                  -wi-ao----
  guest-vg-vol                         guest-lv-1                           /dev/7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/e288d725-f620-457e-b9e9-ea8a4edf89c4(0)   -wi-------
  guest-vg-vol                         guest-lv-2                           /dev/7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/e288d725-f620-457e-b9e9-ea8a4edf89c4(512) -wi-------

(Note that the guest lvs are not active, this system run with lvmetad disabled)

When ovirt lv is activated (e.g, when starting a vm), udev rules trigger a pvscan
of the new device (/dev/vgname/lvname). If the device is used as a pv (e.g. guest
created a pv from the device), lvm will activate the lvs inside the ovirt raw lvs.

The active guest lvs are "holding" the ovirt raw lvs:

# realpath /dev/7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/e288d725-f620-457e-b9e9-ea8a4edf89c4
/dev/dm-38

# ls /sys/block/dm-38/holders/
dm-39  dm-40

# realpath /dev/guest-vg-vol/guest-lv-*
/dev/dm-39
/dev/dm-40

When you stop the vm, deactivation of the raw ovirt volume will fail (from vdsm log):

CannotDeactivateLogicalVolume: Cannot deactivate Logical Volume: ('General Storage Exception: ("5 [] [\'  Logical volume 7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/e288d725-f620-457e-b9e9-ea8a4edf89c4 is used by another device.\']\\n7df95b16-1bd3-4c23-bbbe-b21d403bdcd8/[\'e288d725-f620-457e-b9e9-ea8a4edf89c4\']",)',)

To deactivate it, you must deactivate first the lvs "holding it":

# vgchange -an guest-vg-vol
  0 logical volume(s) in volume group "guest-vg-vol" now active

For iSCSI storage, the fix to this issue is to disable lvmetad service, which is 
doing event based auto activation. Guest lvs will still be seen on the host when
running lvm commands, but lvm will not try to activate the lvs.

For FC storage, activation is done during boot. The fix is to have a lvm filter
blacklisting ovirt lvs, or better, whitelisting the devices needed by the host.

A whitelist filter will also solve the issue with guest lvs on luns not used as
ovirt lvs. These luns may be used by vms directly or unused luns that were just
added to the system.

Comment 153 Ryan Barry 2017-02-14 14:11:07 UTC
*** Bug 1421424 has been marked as a duplicate of this bug. ***

Comment 154 Yaniv Kaul 2017-02-14 14:13:19 UTC
Ryan - are you sure it's a dup? Don't we need to do any changes in RHVH (initrd?) to ensure there's no auto-activation? Is the work on VDSM enough?

Comment 155 Ryan Barry 2017-02-14 14:26:58 UTC
I'm positive this is a dup -- confirmed with Sahina as well.

The bug against RHVH is adding the host to the engine, which has spurious output from lvmetad. Installation itself is ok, so we don't need any changes to the initrd.

Comment 156 Nir Soffer 2017-02-27 14:59:04 UTC
In this bug we track only the issue of dynamic activation of guest logical volumes
on RHV raw volumes when using iSCSI storage.

This issue is caused by lvmetad, which is not compatible with RHV shared storage
and causes also other trouble. This service is disabled in 4.1.1.

The issue of activation of guest logical volumes during boot on when using FC 
storage will be track in another bug (we have several lvm bugs related to this).

Comment 158 Allon Mureinik 2017-03-30 11:31:14 UTC
Nir, can you please add some doctext explain what was done here? Thanks!

Comment 159 Natalie Gavrielov 2017-04-03 14:26:52 UTC
Nir,
I saw in comment 48 a way to partially reproduce this issue, should I use this scenario for verification procedure, or is there something else? and what is the expected result after the fix?
another thing - should it be tested on both RHEL 7.3 and RHV-H? or one of them is enough?

Comment 160 Nir Soffer 2017-04-03 22:03:21 UTC
(In reply to Natalie Gavrielov from comment #159)
> Nir,
> I saw in comment 48 a way to partially reproduce this issue, should I use
> this scenario for verification procedure, or is there something else? 

Yes, this is the best way to reproduce the issue of guest lvs visible on a host.

> and what is the expected result after the fix?

After the fix, guests lvs will not be visible on the host when the vm disk
lv is not active.

With iSCSI storage, the vm disk lv should be active only when the vm is running,
or when performing storage operations on this lv (e.g. copy disk to another storage
domain).

With FC storage, all lvs are activated during boot, and vdsm deactivate all unused
lvs during vdsm startup. After a host becomes "UP" in engine, all unused lvs
should be in active.

Note that special lvs used by vdsm (e.g. inbox, outbox) are always active.

When starting a vm, the vm lvs will be activated. At this point, you will be
able to see guest when using "lvs" command. But the guest lvs should not be
active. Also, since the guest lvs are not active, when stopping the vm, the vm
lvs should be deactivated. Previously if guest lv was active, the vm lv would
fail to deactivate.

> another thing - should it be tested on both RHEL 7.3 and RHV-H? or one of
> them is enough?

You should test with both so make sure we son't have any surprises.

Comment 161 Natalie Gavrielov 2017-04-09 12:38:51 UTC
Scenario used: described in comment 48, (the  first one)
Environment: Red Hat Enterprise Linux Server release 7.3 (Maipo)

Step 6, output:
----------------------------------------------------------------------------
[root@storage-ge9-vdsm2 images]# pvscan --cache
[root@storage-ge9-vdsm2 images]# lvs
  LV                                   VG                                   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  2c09d589-01a9-4a86-9342-95fb2b6f57e7 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------  11.00g   <-- This one is attached to the vm                                                 
  3ab6a9ae-63b3-4624-8b08-1cf59e8c1698 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------   1.00g                                                    
  568226c0-8343-48f8-a74b-dbdf7d8a4105 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------   6.00g                                                    
  5c291905-f617-46ce-b167-2351aeeebf1d 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------   1.00g                                                    
  cc23e1a4-0df0-4d97-962b-2b39503e2ee9 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------  10.00g                                                    
  d8ddae5e-bf6c-4eef-b0e9-9ac570ed927e 4742316d-4091-4aad-99c9-6521c94ac3fd -wi------- 128.00m                                                    
  db636fb0-caaa-4764-bfe0-c202dae20328 4742316d-4091-4aad-99c9-6521c94ac3fd -wi------- 128.00m                                                    
  ids                                  4742316d-4091-4aad-99c9-6521c94ac3fd -wi-ao---- 128.00m                                                    
  inbox                                4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a----- 128.00m                                                    
  leases                               4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a-----   2.00g                                                    
  master                               4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a-----   1.00g                                                    
  metadata                             4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a----- 512.00m                                                    
  outbox                               4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a----- 128.00m                                                    
  xleases                              4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a-----   1.00g                                                    
  568226c0-8343-48f8-a74b-dbdf7d8a4105 577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-------   6.00g                                                    
  79f3acdf-66ad-4836-87c2-37c21d7871b8 577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi------- 128.00m                                                    
  aa0a43d7-4652-4203-a3eb-86199e6c7b77 577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi------- 128.00m                                                    
  ids                                  577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-ao---- 128.00m                                                    
  inbox                                577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a----- 128.00m                                                    
  leases                               577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a-----   2.00g                                                    
  master                               577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a-----   1.00g                                                    
  metadata                             577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a----- 512.00m                                                    
  outbox                               577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a----- 128.00m                                                    
  xleases                              577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a-----   1.00g                                                    
  568226c0-8343-48f8-a74b-dbdf7d8a4105 800983a3-51f8-4977-ab08-b8d391c9da77 -wi-------   6.00g                                                    
  a0b0fd34-f61f-4563-8260-7bc160f186b1 800983a3-51f8-4977-ab08-b8d391c9da77 -wi------- 128.00m                                                    
  aabd877e-c506-4e95-897e-6d472eeeb529 800983a3-51f8-4977-ab08-b8d391c9da77 -wi------- 128.00m                                                    
  ids                                  800983a3-51f8-4977-ab08-b8d391c9da77 -wi-ao---- 128.00m                                                    
  inbox                                800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a----- 128.00m                                                    
  leases                               800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a-----   2.00g                                                    
  master                               800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a-----   1.00g                                                    
  metadata                             800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a----- 512.00m                                                    
  outbox                               800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a----- 128.00m                                                    
  xleases                              800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a-----   1.00g                                                    
  root                                 VolGroup01                           -wi-ao----  17.31g                                                    
  guest-lv                             guest-vg                             -wi-a-----  10.00g                                                    
  guest-lv-2                           guest-vg                             -wi-a-----   5.00g                                                    
  guest-lv-3                           guest-vg-2                           -wi-a-----  10.00g                                                    
  guest-lv-4                           guest-vg-2                           -wi-a-----   5.00g                                                    
------------------------------------------------------------------------------

1. 2c09d589-01a9-4a86-9342-95fb2b6f57e7 is the volume that's attached to the vm.
   When the vm is not running it shows: -wi------- and when the vm is running: -wi-ao----

2. The lvs I created: guest-lv, guest-lv-2, guest-lv-3, guest-lv-4
   show: -wi-a----- (regardless of the vm state) when running lvs on the same host I created the lvs on.
and show -wi------- when running lvs command from another host (regardless of the vm state).

In comment 160 it says:

> After the fix, guests lvs will not be visible on the host when the vm disk
lv is not active.

I don't really understand why should I expect any lvs to not be visible?

and also:

> When starting a vm, the vm lvs will be activated. At this point, you will be
able to see guest when using "lvs" command. But the guest lvs should not be
active. 

When creating a lv it's automatically activated.. so I see it as active regardless of the vm state (-wi-a-----).


Anything I'm missing here?

Comment 162 Nir Soffer 2017-04-12 12:18:01 UTC
(In reply to Natalie Gavrielov from comment #161)
> Scenario used: described in comment 48, (the  first one)

This first scenario is how to reproduce this *without* the fix.

Which version of vdsm did you test?

Can you show the output of:
systemctl status lvm2-lvmetad.service


> Step 6, output:
> ----------------------------------------------------------------------------
> [root@storage-ge9-vdsm2 images]# pvscan --cache
> [root@storage-ge9-vdsm2 images]# lvs
>   LV                                   VG                                  
> Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
>   guest-lv                             guest-vg                            
> -wi-a-----  10.00g                                                    
>   guest-lv-2                           guest-vg                            
> -wi-a-----   5.00g                                                    
>   guest-lv-3                           guest-vg-2                          
> -wi-a-----  10.00g                                                    
>   guest-lv-4                           guest-vg-2                          
> -wi-a-----   5.00g                                                    

Can you share the output of:
lvs -o vg_name,lv_name,attr,devices guest-vg guest-vg-2

These lvs are active, this is what we expect when lvmetad is active.

> > When starting a vm, the vm lvs will be activated. At this point, you will be
> able to see guest when using "lvs" command. But the guest lvs should not be
> active. 
> 
> When creating a lv it's automatically activated.. so I see it as active
> regardless of the vm state (-wi-a-----).

If you create the lv inside the guest, it should not be activated 
on the host. Maybe you created the lvs on the host?

It will be best if I can check the host myself.

Comment 163 Natalie Gavrielov 2017-04-12 13:46:54 UTC
Scenario used:

1. Create a preallocated disk on iscsi storage domain, and attach it to a vm with an OS (this disk will not be running the OS).

2. Start vm, connect to it and create a pv,vg and lv on the disk attached from step 1:
root@localhost ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   12G  0 disk 
sr0     11:0    1 1024M  0 rom  
vda    253:0    0   10G  0 disk 
├─vda1 253:1    0  200M  0 part /boot
├─vda2 253:2    0    2G  0 part [SWAP]
└─vda3 253:3    0  7.8G  0 part /

[root@localhost ~]# pvcreate /dev/sda
  Physical volume "/dev/sda" successfully created.

[root@localhost ~]# vgcreate bug-verification-guest-vg /dev/sda
  Volume group "bug-verification-guest-vg" successfully created

[root@localhost ~]# lvcreate -n bug-verification-guest-lv-1 -L 4g bug-verification-guest-vg
  Logical volume "bug-verification-guest-lv-1" created.

[root@localhost ~]# lvcreate -n bug-verification-guest-lv-2 -L 6g bug-verification-guest-vg
  Logical volume "bug-verification-guest-lv-2" created.

3. Shutdown vm.

4. Put host to maintenance.

5. Activate host, connect to it and run:

[root@storage-ge9-vdsm1 ~]# pvscan --cache
[root@storage-ge9-vdsm1 ~]# lvs

-------------------------------------------------------------------------------
  LV                                   VG                                   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  2c09d589-01a9-4a86-9342-95fb2b6f57e7 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------  11.00g                                                    
  3ab6a9ae-63b3-4624-8b08-1cf59e8c1698 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------   1.00g                                                    
  568226c0-8343-48f8-a74b-dbdf7d8a4105 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------   6.00g                                                    
  5c291905-f617-46ce-b167-2351aeeebf1d 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------   1.00g                                                    
  660cd2c3-88ee-48ac-ae13-74ca1b485052 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------  12.00g                                                    
  cc23e1a4-0df0-4d97-962b-2b39503e2ee9 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------  10.00g                                                    
  d8ddae5e-bf6c-4eef-b0e9-9ac570ed927e 4742316d-4091-4aad-99c9-6521c94ac3fd -wi------- 128.00m                                                    
  db636fb0-caaa-4764-bfe0-c202dae20328 4742316d-4091-4aad-99c9-6521c94ac3fd -wi------- 128.00m                                                    
  ids                                  4742316d-4091-4aad-99c9-6521c94ac3fd -wi-ao---- 128.00m                                                    
  inbox                                4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a----- 128.00m                                                    
  leases                               4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a-----   2.00g                                                    
  master                               4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a-----   1.00g                                                    
  metadata                             4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a----- 512.00m                                                    
  outbox                               4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a----- 128.00m                                                    
  xleases                              4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a-----   1.00g                                                    
  568226c0-8343-48f8-a74b-dbdf7d8a4105 577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-------   6.00g                                                    
  79f3acdf-66ad-4836-87c2-37c21d7871b8 577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi------- 128.00m                                                    
  aa0a43d7-4652-4203-a3eb-86199e6c7b77 577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi------- 128.00m                                                    
  ids                                  577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-ao---- 128.00m                                                    
  inbox                                577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a----- 128.00m                                                    
  leases                               577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a-----   2.00g                                                    
  master                               577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a-----   1.00g                                                    
  metadata                             577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a----- 512.00m                                                    
  outbox                               577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a----- 128.00m                                                    
  xleases                              577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a-----   1.00g                                                    
  568226c0-8343-48f8-a74b-dbdf7d8a4105 800983a3-51f8-4977-ab08-b8d391c9da77 -wi-------   6.00g                                                    
  a0b0fd34-f61f-4563-8260-7bc160f186b1 800983a3-51f8-4977-ab08-b8d391c9da77 -wi------- 128.00m                                                    
  aabd877e-c506-4e95-897e-6d472eeeb529 800983a3-51f8-4977-ab08-b8d391c9da77 -wi------- 128.00m                                                    
  ids                                  800983a3-51f8-4977-ab08-b8d391c9da77 -wi-ao---- 128.00m                                                    
  inbox                                800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a----- 128.00m                                                    
  leases                               800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a-----   2.00g                                                    
  master                               800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a-----   1.00g                                                    
  metadata                             800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a----- 512.00m                                                    
  outbox                               800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a----- 128.00m                                                    
  xleases                              800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a-----   1.00g                                                    
  root                                 VolGroup01                         -wi-ao----  17.31g                                                    

-------------------------------------------------------------------------------
* The 2 lvs created, bug-verification-guest-lv-1 and bug-verification-guest-lv-2 are not displayed in the list
** The preallocated disk created in step 1, appears as inactive (-wi-------).
660cd2c3-88ee-48ac-ae13-74ca1b485052 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------  12.00g 

6. Start the vm, and run the following commands on the host:

[root@storage-ge9-vdsm1 ~]# pvscan --cache
[root@storage-ge9-vdsm1 ~]# lvs

-------------------------------------------------------------------------------
  LV                                   VG                                   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  2c09d589-01a9-4a86-9342-95fb2b6f57e7 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------  11.00g                                                    
  3ab6a9ae-63b3-4624-8b08-1cf59e8c1698 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------   1.00g                                                    
  568226c0-8343-48f8-a74b-dbdf7d8a4105 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------   6.00g                                                    
  5c291905-f617-46ce-b167-2351aeeebf1d 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------   1.00g                                                    
  660cd2c3-88ee-48ac-ae13-74ca1b485052 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-ao----  12.00g                                                    
  cc23e1a4-0df0-4d97-962b-2b39503e2ee9 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-------  10.00g                                                    
  d8ddae5e-bf6c-4eef-b0e9-9ac570ed927e 4742316d-4091-4aad-99c9-6521c94ac3fd -wi------- 128.00m                                                    
  db636fb0-caaa-4764-bfe0-c202dae20328 4742316d-4091-4aad-99c9-6521c94ac3fd -wi------- 128.00m                                                    
  ids                                  4742316d-4091-4aad-99c9-6521c94ac3fd -wi-ao---- 128.00m                                                    
  inbox                                4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a----- 128.00m                                                    
  leases                               4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a-----   2.00g                                                    
  master                               4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a-----   1.00g                                                    
  metadata                             4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a----- 512.00m                                                    
  outbox                               4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a----- 128.00m                                                    
  xleases                              4742316d-4091-4aad-99c9-6521c94ac3fd -wi-a-----   1.00g                                                    
  568226c0-8343-48f8-a74b-dbdf7d8a4105 577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-------   6.00g                                                    
  79f3acdf-66ad-4836-87c2-37c21d7871b8 577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi------- 128.00m                                                    
  aa0a43d7-4652-4203-a3eb-86199e6c7b77 577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi------- 128.00m                                                    
  ids                                  577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-ao---- 128.00m                                                    
  inbox                                577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a----- 128.00m                                                    
  leases                               577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a-----   2.00g                                                    
  master                               577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a-----   1.00g                                                    
  metadata                             577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a----- 512.00m                                                    
  outbox                               577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a----- 128.00m                                                    
  xleases                              577b4f73-fa4d-4ead-b614-30c3ec3e9fdc -wi-a-----   1.00g                                                    
  568226c0-8343-48f8-a74b-dbdf7d8a4105 800983a3-51f8-4977-ab08-b8d391c9da77 -wi-------   6.00g                                                    
  a0b0fd34-f61f-4563-8260-7bc160f186b1 800983a3-51f8-4977-ab08-b8d391c9da77 -wi------- 128.00m                                                    
  aabd877e-c506-4e95-897e-6d472eeeb529 800983a3-51f8-4977-ab08-b8d391c9da77 -wi------- 128.00m                                                    
  ids                                  800983a3-51f8-4977-ab08-b8d391c9da77 -wi-ao---- 128.00m                                                    
  inbox                                800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a----- 128.00m                                                    
  leases                               800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a-----   2.00g                                                    
  master                               800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a-----   1.00g                                                    
  metadata                             800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a----- 512.00m                                                    
  outbox                               800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a----- 128.00m                                                    
  xleases                              800983a3-51f8-4977-ab08-b8d391c9da77 -wi-a-----   1.00g                                                    
  root                                 VolGroup01                           -wi-ao----  17.31g                                                    
  bug-verification-guest-lv-1          bug-verification-guest-vg            -wi-------   4.00g                                                    
  bug-verification-guest-lv-2          bug-verification-guest-vg            -wi-------   6.00g  
-------------------------------------------------------------------------------

* The 2 lvs created, bug-verification-guest-lv-1 and bug-verification-guest-lv-2 are displayed, and are not shown as active: -wi-------.  
** The preallocated disk created in step 1, should appear as active and open (with -wi-ao----).
660cd2c3-88ee-48ac-ae13-74ca1b485052 4742316d-4091-4aad-99c9-6521c94ac3fd -wi-ao----  12.00g

Environment: Red Hat Enterprise Linux Server release 7.3 (Maipo)
Builds:
vdsm-4.19.10.1-1.el7ev.x86_64
rhevm-4.1.1.7-0.1.el7.noarch

Moving on to testing this on RHV-H

Comment 164 Nir Soffer 2017-04-12 14:44:11 UTC
(In reply to Nir Soffer from comment #162)
> (In reply to Natalie Gavrielov from comment #161)
Natalie, this looks fine so far.

Comment 165 Natalie Gavrielov 2017-04-13 15:07:37 UTC
Scenario performed using RHV-H


1. [root@camel-vdsb ~]# cat /etc/lvm/lvm.conf | grep "use_lvmetad = 1"
	use_lvmetad = 1
2. Create a preallocated disk on iscsi storage domain, and attach it to a vm with an OS (this disk will not be running the OS).

3. Start vm, connect to it and create a pv,vg and lv on the disk attached from step 1. (will attach snapshot of the issued commands)

4. Shutdown vm.

5. Put host to maintenance.

6. Activate host, connect to it and run:

[root@camel-vdsb ~]# pvscan --cache
[root@camel-vdsb ~]# lvs

-----------------------------------------------------------------------------
 LV                                   VG                                   Attr       LSize   Pool   Origin                Data%  Meta%  Move Log Cpy%Sync Convert
  6a2c2fcd-bcc3-457a-b7dd-059ca14422e6 5b723c72-2428-4230-a77d-2580637fa263 -wi-------   8.00g                                                                     
  8ea07646-035b-4987-bed3-30a230686ced 5b723c72-2428-4230-a77d-2580637fa263 -wi------- 128.00m                                                                     
  c227148a-3487-42cf-901a-709ef0cdd5c1 5b723c72-2428-4230-a77d-2580637fa263 -wi------- 128.00m                                                                     
  ids                                  5b723c72-2428-4230-a77d-2580637fa263 -wi-ao---- 128.00m                                                                     
  inbox                                5b723c72-2428-4230-a77d-2580637fa263 -wi-a----- 128.00m                                                                     
  leases                               5b723c72-2428-4230-a77d-2580637fa263 -wi-a-----   2.00g                                                                     
  master                               5b723c72-2428-4230-a77d-2580637fa263 -wi-a-----   1.00g                                                                     
  metadata                             5b723c72-2428-4230-a77d-2580637fa263 -wi-a----- 512.00m                                                                     
  outbox                               5b723c72-2428-4230-a77d-2580637fa263 -wi-a----- 128.00m                                                                     
  xleases                              5b723c72-2428-4230-a77d-2580637fa263 -wi-a-----   1.00g                                                                     
  200ae2b3-11af-460a-bbe1-49967dd6d0cf c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  4d423b3f-caf2-4690-ace7-40a99967c53a c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  9dcb10bb-72bd-4c9c-89e3-4a3ed1a8e733 c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi-------   7.00g                                                                     
  ids                                  c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  inbox                                c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  leases                               c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi-------   2.00g                                                                     
  master                               c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi-------   1.00g                                                                     
  metadata                             c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 512.00m                                                                     
  outbox                               c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  xleases                              c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi-------   1.00g                                                                     
  pool00                               rhvh_camel-vdsb                      twi-aotz-- 106.05g                              4.80   0.33                            
  rhvh-4.1-0.20170403.0                rhvh_camel-vdsb                      Vwi---tz-k  91.05g pool00 root                                                         
  rhvh-4.1-0.20170403.0+1              rhvh_camel-vdsb                      Vwi-aotz--  91.05g pool00 rhvh-4.1-0.20170403.0 3.91                                   
  root                                 rhvh_camel-vdsb                      Vwi-a-tz--  91.05g pool00                       3.88                                   
  swap                                 rhvh_camel-vdsb                      -wi-ao----  13.68g                                                                     
  var                                  rhvh_camel-vdsb                      Vwi-aotz--  15.00g pool00                       3.72 
-----------------------------------------------------------------------------
*  The 2 lvs created, bug-ver-guest-lv-1 and bug-ver-guest-lv-2 are not 
   displayed in the list
** The preallocated disk created in step 1, appears as inactive (-wi-------).
   6a2c2fcd-bcc3-457a-b7dd-059ca14422e6 5b723c72-2428-4230-a77d-2580637fa263 
   -wi-------   8.00g   

6. Start the vm, and run the following commands on the host:
[root@camel-vdsb ~]# pvscan --cache
[root@camel-vdsb ~]# lvs
-----------------------------------------------------------------------------
  LV                                   VG                                   Attr       LSize   Pool   Origin                Data%  Meta%  Move Log Cpy%Sync Convert
  6a2c2fcd-bcc3-457a-b7dd-059ca14422e6 5b723c72-2428-4230-a77d-2580637fa263 -wi-ao----   8.00g                                                                     
  8ea07646-035b-4987-bed3-30a230686ced 5b723c72-2428-4230-a77d-2580637fa263 -wi------- 128.00m                                                                     
  c227148a-3487-42cf-901a-709ef0cdd5c1 5b723c72-2428-4230-a77d-2580637fa263 -wi------- 128.00m                                                                     
  ids                                  5b723c72-2428-4230-a77d-2580637fa263 -wi-ao---- 128.00m                                                                     
  inbox                                5b723c72-2428-4230-a77d-2580637fa263 -wi-a----- 128.00m                                                                     
  leases                               5b723c72-2428-4230-a77d-2580637fa263 -wi-a-----   2.00g                                                                     
  master                               5b723c72-2428-4230-a77d-2580637fa263 -wi-a-----   1.00g                                                                     
  metadata                             5b723c72-2428-4230-a77d-2580637fa263 -wi-a----- 512.00m                                                                     
  outbox                               5b723c72-2428-4230-a77d-2580637fa263 -wi-a----- 128.00m                                                                     
  xleases                              5b723c72-2428-4230-a77d-2580637fa263 -wi-a-----   1.00g                                                                     
  bug-ver-guest-lv-1                   bug-ver-guest-vg                     -wi-------   3.00g                                                                     
  bug-ver-guest-lv-2                   bug-ver-guest-vg                     -wi-------   2.00g                                                                     
  200ae2b3-11af-460a-bbe1-49967dd6d0cf c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  4d423b3f-caf2-4690-ace7-40a99967c53a c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  9dcb10bb-72bd-4c9c-89e3-4a3ed1a8e733 c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi-------   7.00g                                                                     
  ids                                  c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  inbox                                c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  leases                               c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi-------   2.00g                                                                     
  master                               c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi-------   1.00g                                                                     
  metadata                             c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 512.00m                                                                     
  outbox                               c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi------- 128.00m                                                                     
  xleases                              c874fd8d-6d69-4882-9ac9-61eff12d7a9b -wi-------   1.00g                                                                     
  pool00                               rhvh_camel-vdsb                      twi-aotz-- 106.05g                              4.80   0.33                            
  rhvh-4.1-0.20170403.0                rhvh_camel-vdsb                      Vwi---tz-k  91.05g pool00 root                                                         
  rhvh-4.1-0.20170403.0+1              rhvh_camel-vdsb                      Vwi-aotz--  91.05g pool00 rhvh-4.1-0.20170403.0 3.91                                   
  root                                 rhvh_camel-vdsb                      Vwi-a-tz--  91.05g pool00                       3.88                                   
  swap                                 rhvh_camel-vdsb                      -wi-ao----  13.68g                                                                     
  var                                  rhvh_camel-vdsb                      Vwi-aotz--  15.00g pool00                       3.72 
-----------------------------------------------------------------------------
*  The two lv's now displayed, bug-ver-guest-lv-1 and bug-ver-guest-lv-2, and 
   are not shown as active: -wi-------.  
** The preallocated disk appears as active and open:
   6a2c2fcd-bcc3-457a-b7dd-059ca14422e6 5b723c72-2428-4230-a77d-2580637fa263 
   -wi-ao----   8.00g 

Environment:
hosted engine with 2 hosts
Red Hat Virtualization Host 4.1 (el7.3)
redhat-release-virtualization-host-content-4.1-0.14.el7.x86_64
redhat-virtualization-host-image-update-placeholder-4.1-0.14.el7.noarch
redhat-release-virtualization-host-4.1-0.14.el7.x86_64

vdsm-4.19.10.1-1.el7ev.x86_64

rhevm-4.1.1.8-0.1.el7.noarch

Comment 166 Natalie Gavrielov 2017-04-13 15:12:26 UTC
Created attachment 1271442 [details]
command issued for creating pv,vg and lv in verification process, step 3 comment 165

Comment 167 David Teigland 2017-04-13 16:46:03 UTC
The command to activate LVs is "pvscan --cache -aay", not "pvscan --cache".

Comment 168 Nir Soffer 2017-04-14 15:25:14 UTC
(In reply to David Teigland from comment #167)
> The command to activate LVs is "pvscan --cache -aay", not "pvscan --cache".

Thanks David, but we were not trying to activate lvs. pvscan --cache was needed
to update lvmetad  with changes made by vdsm using
--config "global {use_lvmetad = 0}". It is not needed now that we disable lvmetad.


Note You need to log in before you can comment on or make changes to this bug.