Bug 1305327 - [RFE] - LVM commands called with autoback [NEEDINFO]
[RFE] - LVM commands called with autoback
Status: NEW
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.6.2
Unspecified Unspecified
medium Severity high
: ovirt-4.3.0
: ---
Assigned To: Nir Soffer
Raz Tamir
: FutureFeature
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-07 02:52 EST by Pavel Zhukov
Modified: 2017-10-29 09:36 EDT (History)
20 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
ykaul: needinfo? (nsoffer)
ratamir: testing_plan_complete-


Attachments (Terms of Use)

  None (edit)
Description Pavel Zhukov 2016-02-07 02:52:05 EST
Description of problem:
Most of lvm commands are called from within vdsm code with disabled autoback. This leads missed LV or corrupted VM images if VG metadata was restored as result of recovery procedure. 

Version-Release number of selected component (if applicable):
vdsm-4.17.18-0.el7ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. Create new disk
2. Check if LV of the disk is under /etc/lvm

Actual results:
LV is not in metadata backup and will not be restored with vgcfgrestore

Additional info:
adcb6c26-2652-4cef-82d5-804e711e8824::DEBUG::2016-02-07 02:41:06,521::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvcreate --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/36001405ba299a3fb8524452890938e29|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --autobackup n --contiguous n --size 2048m --addtag OVIRT_VOL_INITIALIZING --name a45899e9-016d-4790-b6b3-5336f4b0b7af db10eec1-fd6b-46e6-8bdf-467657e58b91 (cwd None)
Comment 2 Nir Soffer 2016-02-07 12:10:42 EST
There is no bug here, we do not expect users restoring our vgs. 

I don't think we can use host based metadata backup, as there is no way to ensure that this backup is correct.
Comment 7 Marina 2016-02-08 13:16:23 EST
I went back to 3.0 code and I see it has no change - it has been that way always.
I can see we use no backup option for all lv operations. Thinking back, it was always a concern to restore from too old of an lvm backup, if customer has introduced any changes to his environment since then.

Actually, I must say, this is a reasonable behavior from vdsm - there maybe hundreds of lvs created (if not more) during a single day. Take, for example, customers that are using big pools. Creating lvm backup with each such change would be non-feasible and would create a lot of overhead - time, i/o and disk space.
Thus, I'll reduce the severity of the bug back to medium.

What is interesting to me is how we do get lvs information restored now, when using vgcfgrestore. When does the archive get updated with lv information? 
And maybe we can find a mid term solution, when vdsm would issue lvm backup, let's say every 100[configurable] lvs? 
Pavel, would it help in your case?
Comment 9 Pavel Zhukov 2016-02-08 16:44:31 EST
(In reply to Marina from comment #7)
> I went back to 3.0 code and I see it has no change - it has been that way
> always.
> I can see we use no backup option for all lv operations. Thinking back, it
> was always a concern to restore from too old of an lvm backup, if customer
> has introduced any changes to his environment since then.
Right. That's why we want to have most recent backup if possible. 
> 
> Actually, I must say, this is a reasonable behavior from vdsm - there maybe
> hundreds of lvs created (if not more) during a single day. Take, for
> example, customers that are using big pools. Creating lvm backup with each
> such change would be non-feasible and would create a lot of overhead - time,
> i/o and disk space.
Not sure about overhead. LVM writes metadata to many places in disk (for backup purposes) every single change. One more place (file) should not bring visible impact especially taking into account it's another block device (not SD). Disk space is not an issue here. It's configurable and 50 latest backups are kept in current implementation. But LVM developers can say more about.
> Thus, I'll reduce the severity of the bug back to medium.
> 
> What is interesting to me is how we do get lvs information restored now,
> when using vgcfgrestore. When does the archive get updated with lv
> information? 
vgchange (addtag/deltag) is called without nobackup as well as vgs from within _reloadvgs
Examples:
description = "Created *after* executing 'vgchange ///  <== addtag/deltag
OR
description = "Created *after* executing 'vgdisplay 
OR
description = "Created *after* executing '/sbin/vgs /// <== _reloadVGs

So in current implementation backup is created *only* if vgchange/reloadvgs called. I don't know the reason (probably because LVM_NOBACKUP was forgotten here. But thanks to this we have backups (adding/removing/attaching/detaching SD, master reconstructing and vdsm restart).
So if user added let's say 100 VMs and for whatever reason LVM metadata was corrupted before storage layout changed all 100 VMs disks will be lost. If qcow image was extended old size will be restored from backup (lvresize). 
> And maybe we can find a mid term solution, when vdsm would issue lvm backup,
> let's say every 100[configurable] lvs? 
If user has 50 LVs they will loose all. Not good... And logic to determinate if backup should be written or not may take more resources than writing itself (see explanation above).
> Pavel, would it help in your case?
Comment 10 Marina 2016-02-26 14:58:13 EST
Allon,
What do you think? 
Should we move it into RFE?

I believe this bug definitely deserves at least a discussion, if we can and when we can to back up the lvs lvm information, since it is definitely beneficial for our customers in case they overwrite their storage by mistake(a pretty common mistake).
Comment 11 Nir Soffer 2016-02-26 15:46:20 EST
Having lvm backups sounds useful, but we cannot use host based backups, since
you will different backups on different hosts (spm moved from one host to
another).

This will be worse in 4.x, when there is no spm - lvs would be created on 
any host, so every host would have different backup.

If we want this, we will have to copy the backups to engine, or to other place
that can store these backups.

I don't expect a performance issue for creating these backups, since we don't 
created lot of lvs, but I did not measure this.
Comment 12 Marina 2016-02-26 16:51:30 EST
Good points.
We are planning to open RFE to enable centralized logging to RHEV-M, so this can go there as well.

thank you!
Comment 16 Yaniv Lavi 2016-05-09 07:06:27 EDT
oVirt 4.0 Alpha has been released, moving to oVirt 4.0 Beta target.
Comment 23 Yaniv Kaul 2016-11-21 05:43:03 EST
Duplicate (sort of) of bug 1380698 ?
Comment 26 Nir Soffer 2016-12-28 11:30:27 EST
(In reply to Yaniv Kaul from comment #23)
> Duplicate (sort of) of bug 1380698 ?

Not really, that bug is just a minor cleanup, avoid using backup when the command
does not support it.

This bug is about having backups, the main issue is where to keep the backups.
With single host, local backups are fine, when working with shared storage, 
local backups are meaningless. You may have different backups on every host,
depennding where the spm was running when lvm command generated a backup.

This can be RFE for 4.2.
Comment 27 Yaniv Lavi 2016-12-28 11:46:10 EST
(In reply to Nir Soffer from comment #26)
> (In reply to Yaniv Kaul from comment #23)
> > Duplicate (sort of) of bug 1380698 ?
> 
> Not really, that bug is just a minor cleanup, avoid using backup when the
> command
> does not support it.
> 
> This bug is about having backups, the main issue is where to keep the
> backups.
> With single host, local backups are fine, when working with shared storage, 
> local backups are meaningless. You may have different backups on every host,
> depennding where the spm was running when lvm command generated a backup.
> 
> This can be RFE for 4.2.

We can have local backups and recommend to use the one from SPM. It at least can help CEE with the restore process, even if it won't be complete.
Comment 28 Nir Soffer 2016-12-28 11:58:25 EST
(In reply to Yaniv Dary from comment #27)
> We can have local backups and recommend to use the one from SPM. It at least
> can help CEE with the restore process, even if it won't be complete.

But the spm may change any time, the system can pick any host as the spm.

For example this flow:

1. SPM running on host A
2. Change storage, backup...
3. SPM switch to host B
4. disaster
5. CEE uses backup from host B instead of host A

I think we need a way to send the backups to external system that does not
depend on block storage this backup describes.
Comment 29 Yaniv Lavi 2016-12-28 12:06:37 EST
(In reply to Nir Soffer from comment #28)
> (In reply to Yaniv Dary from comment #27)
> > We can have local backups and recommend to use the one from SPM. It at least
> > can help CEE with the restore process, even if it won't be complete.
> 
> But the spm may change any time, the system can pick any host as the spm.
> 
> For example this flow:
> 
> 1. SPM running on host A
> 2. Change storage, backup...
> 3. SPM switch to host B
> 4. disaster
> 5. CEE uses backup from host B instead of host A
> 
> I think we need a way to send the backups to external system that does not
> depend on block storage this backup describes.

I was suggesting a local back on each if the hosts and then using the current SPM backup for the initial restore process. The backup doesn't need to be perfect, but allow CEE to start the restore process. We can find longer term solution to an external system.
Comment 30 Yaniv Kaul 2017-06-06 14:55:46 EDT
Does it make sense to externalize it? Have an Ansible script that runs (for example) every hour, goes to the current SPM and saves a backup on the local host? Sounds like a good enough solution to me.
Comment 31 Allon Mureinik 2017-06-06 17:27:02 EDT
(In reply to Yaniv Kaul from comment #30)
> Does it make sense to externalize it? Have an Ansible script that runs (for
> example) every hour, goes to the current SPM and saves a backup on the local
> host? Sounds like a good enough solution to me.

That sounds like a way to automate a really bad idea (see, e.g., comment 28). I wouldn't go that way.

Note You need to log in before you can comment on or make changes to this bug.