Bug 1589508

(In reply to Yaniv Lavi from comment #0)
> Description of problem:
> RHV, OSP and CNV use raw block devices in VMs and container pods. We would
> like to be able to capture changed blocks on these block devices for
> incremental backup purposes.
> 
> This includes:
> - Starting/disabling tracking changed blocks to bitmaps.
> - Layering bitmaps (starting a new one and merging to a old one).
> 
> This would enable smart backup of these devices.

We already offer the "era" DM target (dm-era) in RHEL7.

In the rhel7 kernel tree, see: Documentation/device-mapper/era.txt

Also, the device-mapper-persistent-data rpm provides the era tools,
the era_invalidate utility is what would be used to read a snapshot of
the era metadata to see what, if any, blocks have changed since the
start of a particular era checkpoint.

lvm2 doesn't currently support adding a dm-era layer to existing LVs but I checked with Joe Thornber and he is open to adding it if there is potential for more consumers of dm-era (and given this RFE, it seems likely to be the case).

Thanks! 
This blog was very useful for me to understand this option:

https://blog.cloudandheat.com/index.php/en/2017/04/24/block-level-data-tracking-using-device-mappers-dm-era-e-g-for-replication/

(In reply to Mike Snitzer from comment #4)
>
> lvm2 doesn't currently support adding a dm-era layer to existing LVs but I
> checked with Joe Thornber and he is open to adding it if there is potential
> for more consumers of dm-era (and given this RFE, it seems likely to be the
> case).

Is LVM a specific use case where this is a problem? 
Does this mean that every device that is in use would not be able to add this capability after provisioning and use?

(In reply to Yaniv Lavi from comment #6)
> (In reply to Mike Snitzer from comment #4)
> >
> > lvm2 doesn't currently support adding a dm-era layer to existing LVs but I
> > checked with Joe Thornber and he is open to adding it if there is potential
> > for more consumers of dm-era (and given this RFE, it seems likely to be the
> > case).
> 
> Is LVM a specific use case where this is a problem? 
> Does this mean that every device that is in use would not be able to add
> this capability after provisioning and use?

I was just saying that lvm2 doesn't yet have support for dm-era.  As in, lvm2 doesn't have the ability to make using dm-era more approachable.

Anyone could manually create a dm-era device using the dmsetup utility, etc.  It would just take a user to embrace that more low-level dm tool and dm interface to load an "era" device.

Knowing whether the expected users will be using lvm2 devices anyway would help to inform PM and the lvm2 developers about if it makes sense to prioritize adding dm-era support.

But I'd encourage any layered product to try dm-era as is.  See if it meets the various requirements.  If not, report back with why not, etc.

(In reply to Mike Snitzer from comment #7)
> 
> I was just saying that lvm2 doesn't yet have support for dm-era.  As in,
> lvm2 doesn't have the ability to make using dm-era more approachable.
> 
> Anyone could manually create a dm-era device using the dmsetup utility, etc.
> It would just take a user to embrace that more low-level dm tool and dm
> interface to load an "era" device.
> 
> Knowing whether the expected users will be using lvm2 devices anyway would
> help to inform PM and the lvm2 developers about if it makes sense to
> prioritize adding dm-era support.
> 
> But I'd encourage any layered product to try dm-era as is.  See if it meets
> the various requirements.  If not, report back with why not, etc.

In the use case of VMs and container the LVM would be created by the user on the device, but it would be filtered on the host level (only used in the context of the VM or container). Would era work for this use case?

Not the 'era' related comment - but worth to note that when thin-provisioning in use - users can use lot of snapshots and can see a difference between their mappings.

See  'man thin_delta' - maybe there is some use-case ?

(In reply to Yaniv Lavi from comment #8)
> (In reply to Mike Snitzer from comment #7)
> > 
> > I was just saying that lvm2 doesn't yet have support for dm-era.  As in,
> > lvm2 doesn't have the ability to make using dm-era more approachable.
> > 
> > Anyone could manually create a dm-era device using the dmsetup utility, etc.
> > It would just take a user to embrace that more low-level dm tool and dm
> > interface to load an "era" device.
> > 
> > Knowing whether the expected users will be using lvm2 devices anyway would
> > help to inform PM and the lvm2 developers about if it makes sense to
> > prioritize adding dm-era support.
> > 
> > But I'd encourage any layered product to try dm-era as is.  See if it meets
> > the various requirements.  If not, report back with why not, etc.
> 
> In the use case of VMs and container the LVM would be created by the user on
> the device, but it would be filtered on the host level (only used in the
> context of the VM or container). Would era work for this use case?

dm-era requires a separate metadata device to store the bitmap, etc.  But I don't see why it couldn't be used on the host as a layer that is managed by the Virt or Container  infrastructure (at the "host" level).

To my understanding the era device is not persistent at reboot. Also we have cases where a VM would be moved from one host to another, this would need ability to migrate the bitmap as well.

This RFE should cover also solving these use cases as well.

(In reply to Yaniv Lavi from comment #11)
> To my understanding the era device is not persistent at reboot. Also we have
> cases where a VM would be moved from one host to another, this would need
> ability to migrate the bitmap as well.
> 
> This RFE should cover also solving these use cases as well.

dm-era is persistent on disk across reboots, it just needs something to reactivate it.  Adding dm-era support to lvm2 would provide the persistence.

So really what needs to be determined by the RHEV stake-holders: does dm-era work for your needs (using a stop-gap script to setup and reactivate the dm-era device as needed)?  If so, _then_ lvm2 development of dm-era support can be prioritized.

As for the need to migrate hosts, this isn't a new requirement for RHEV storage technologies.  It would need to be active/passive though, and would imply shared storage (which afaik is something RHEV already requires: be it via FC, iSCSI, etc).

The main use case for dm-era was for tracking changes on a block device so that we can rollback snapshots on external devices (ie. the snapshots are managed by a third party device).

I'm sure you could use it for incremental back ups.  But I think thinp's snapshots combined with it's ability to take a 'metadata snapshot' which allows userland to examine live metadata is really the way to go.  I'm very keen to write a tool based around this, but it's not going to appear any time soon.