Bug 1930261

BZ1928284 covers the VDO side of the conversion process.

So upstream now supports these conversion:

Disk/MD->VDO->FS
Disk/MD->LVM->VDO->FS

preparing mandatory patches:

https://listman.redhat.com/archives/lvm-devel/2021-June/msg00082.html
https://listman.redhat.com/archives/lvm-devel/2021-June/msg00083.html
https://listman.redhat.com/archives/lvm-devel/2021-June/msg00084.html

main patch with vdoimport script:

https://listman.redhat.com/archives/lvm-devel/2021-July/msg00008.html

----

Simple outline of testing:

# Create some VDO volume by vdo manager
$ vdo create --name VDONAME --device=/dev/xxxx --vdoLogicalSize=SSSS

# Convert by vdoimport script from lvm2 package
# (requires  vdo manager with vdo convert support)
$ vdoimport  --name vgname/lvname  /dev/xxxx

And missed man page in main patch:

https://listman.redhat.com/archives/lvm-devel/2021-July/msg00009.html

This doesn't appear to be working in the latest lvm build:


[root@hayes-01 ~]# vdo create --name V2 --vdoLogicalSize 15G --device /dev/sdc1
Creating VDO V2
      The VDO volume can address 928 GB in 464 data slabs, each 2 GB.
      It can grow to address at most 16 TB of physical storage in 8192 slabs.
      If a larger maximum size might be needed, use bigger slabs.
Starting VDO V2
Starting compression on VDO V2
VDO instance 1 volume is ready at /dev/mapper/V2

[root@hayes-01 ~]# vdoimport  --name testvg/testlv  /dev/sdc1
-bash: /usr/sbin/vdoimport: Permission denied

[root@hayes-01 ~]# which vdoimport
/usr/bin/which: no vdoimport in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)


kernel-4.18.0-323.el8    BUILT: Wed Jul 14 12:12:22 CDT 2021
lvm2-2.03.12-5.el8    BUILT: Tue Jul 13 11:50:03 CDT 2021
lvm2-libs-2.03.12-5.el8    BUILT: Tue Jul 13 11:50:03 CDT 2021

vdo-6.2.5.65-14.el8    BUILT: Thu Jul 22 12:56:43 CDT 2021
kmod-kvdo-6.2.5.65-79.el8    BUILT: Thu Jul 22 12:59:43 CDT 2021

Looking into this a bit, it looks like the permissions are incorrect on vdoimport and therefore you're not finding it.

When I install lvm2-2.03.12-5.el8, I can find /usr/sbin/vdoimport, but with only 0444 permissions.  Fixing that makes things work for me.

[root@localhost ~]# rpm -qa kmod-kvdo vdo lvm2 kernel
vdo-6.2.5.41-14.el8.x86_64
lvm2-2.03.12-5.el8.x86_64
kmod-kvdo-6.2.5.41-79.el8.x86_64
kernel-4.18.0-324.el8.x86_64
[root@localhost ~]# ls -l /usr/sbin/vdoimport
-r--r--r--. 1 root root 11067 Jul 13 12:49 /usr/sbin/vdoimport
[root@localhost ~]# chmod +x /usr/sbin/vdoimport 
[root@localhost ~]# vdo create --name vdo0 --device /dev/loop0
Creating VDO vdo0
      Logical blocks defaulted to 3139552 blocks.
      The VDO volume can address 12 GB in 6 data slabs, each 2 GB.
      It can grow to address at most 16 TB of physical storage in 8192 slabs.
      If a larger maximum size might be needed, use bigger slabs.
Starting VDO vdo0
Starting compression on VDO vdo0
VDO instance 0 volume is ready at /dev/mapper/vdo0
[root@localhost ~]# vdoimport /dev/loop0
Device does not exist.
Command failed.
  Volume group "vdovg" not found
  Cannot process volume group vdovg
Stopping VDO vdo0
Converting VDO vdo0
      Opening /dev/loop0 exclusively
      Loading the VDO superblock and volume geometry
      Checking the VDO state
      Converting the UDS index
      Converting the VDO
      Conversion completed for '/dev/loop0': VDO is now offset by 2097152 bytes
  Physical volume "/dev/loop0" successfully created.
  Volume group "vdovg" successfully created
  WARNING: Logical volume vdovg/vdolvol_vpool not zeroed.
  Logical volume "vdolvol_vpool" created.
  WARNING: Converting logical volume vdovg/vdolvol_vpool to VDO pool volume WITHOUT formating.
  WARNING: Using invalid VDO pool data MAY DESTROY YOUR DATA!
Do you really want to convert vdovg/vdolvol_vpool? [y/n]: y
  Logical volume "vdolvol" created.
  Converted vdovg/vdolvol_vpool to VDO pool volume and created virtual vdovg/vdolvol VDO volume.
[root@localhost ~]# lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                           7:0    0   15G  0 loop 
└─vdovg-vdolvol_vpool_vdata   252:0    0   15G  0 lvm  
  └─vdovg-vdolvol_vpool-vpool 252:1    0   12G  0 lvm  
    └─vdovg-vdolvol           252:2    0   12G  0 lvm  
sr0                            11:0    1  470K  0 rom  
vda                           253:0    0   20G  0 disk 
└─vda1                        253:1    0   20G  0 part /

Yeah, i tried manually changing the perms as well, but still ended of with VG create errors. I'm using the syntax provided in comment #8.


[root@hayes-01 ~]# vdo create --name V3 --vdoLogicalSize 15G --device /dev/sdd1
Creating VDO V3
      The VDO volume can address 928 GB in 464 data slabs, each 2 GB.
      It can grow to address at most 16 TB of physical storage in 8192 slabs.
      If a larger maximum size might be needed, use bigger slabs.
Starting VDO V3
Starting compression on VDO V3
VDO instance 2 volume is ready at /dev/mapper/V3

[root@hayes-01 ~]#  vdoimport  --name bar/foo  /dev/sdd1
Device does not exist.
Command failed.
  Volume group "bar" not found
  Cannot process volume group bar
Device does not exist.
Command failed.
/usr/sbin/vdoimport: line 263: vdo_logicalSize: unbound variable

[root@hayes-01 ~]#  vdoimport  --name bar/foo  /dev/mapper/V3
  Volume group "bar" not found
  Cannot process volume group bar
Device does not exist.
Command failed.
/usr/sbin/vdoimport: line 263: vdo_logicalSize: unbound variable


[root@hayes-01 ~]#  vdoimport  --name foo  /dev/mapper/V3
  Volume group "foo" not found
  Cannot process volume group foo
Device does not exist.
Command failed.
/usr/sbin/vdoimport: line 263: vdo_logicalSize: unbound variable

Yep - permission for 'vdoimport' script seems to be missing  executable bit set -  make instal/packaging error.

But your 2nd. errors seems to be caused by some missing handling of system setup.

Note  'vdoimport  --name foo  /dev/mapper/V3'  is not valid case we require (ATM) to pass  backend device.
But I think it wouldn't be hard to also add syntax sugar and recognize also running VDO volume and add support to prompt for stop&convert.

Noramlly user should 'stop' VDO volume before the conversion (as it can't happen online) - so before the conversion runs this command
(somehow I missed to emphesize this important detail in my Comment 8)

# vdo stop  --name  VDONAME

But it's clear that shown error messages do not look very helpful for some novice users.


So in case the 'stopped'  VDO volume does NOT fix the issue for you - can you please try to attach the result of your command execution when you simply edit  /usr/sbin/vdoimport  script file and you replace its 1st. line:

#!/bin/bash

with

#!/bin/bash -x

this should provide detailed bash logging.

Actually stopping should be automated.

So not really sure - what's wrong yet - so please provide the extended log.

Andy can you try with a real device? If I use a loop back device it works for me as well.

[...]
+ lvm lvconvert --config 'allocation {
        vdo_use_compression = 1
        vdo_use_deduplication = 1
        vdo_use_metadata_hints=1
        vdo_minimum_io_size = 4096
        vdo_block_map_cache_size_mb = 128
        vdo_block_map_period = 16380
        vdo_check_point_frequency = 0
        vdo_use_sparse_index = 0
        vdo_index_memory_size_mb = 256
        vdo_slab_size_mb = 128
        vdo_ack_threads = 1
        vdo_bio_threads = 4
        vdo_bio_rotation = 64
        vdo_cpu_threads = 2
        vdo_hash_zone_threads = 1
        vdo_logical_threads = 1
        vdo_physical_threads = 1
        vdo_write_policy = auto
        vdo_max_discard = 4096
        vdo_pool_header_size = 0
}' -Zn -V 6278744k -n vdolvol --type vdo-pool vdovg/vdolvol_vpool
  WARNING: Converting logical volume vdovg/vdolvol_vpool to VDO pool volume WITHOUT formating.
  WARNING: Using invalid VDO pool data MAY DESTROY YOUR DATA!
Do you really want to convert vdovg/vdolvol_vpool? [y/n]: y
  Logical volume "vdolvol" created.
  Converted vdovg/vdolvol_vpool to VDO pool volume and created virtual vdovg/vdolvol VDO volume.
+ rm -fr /tmp/vdoimport_287815410

So there is missing code for detecting 'aliased' names.

As /etc/vdoconf.yml can store any kind of device alias for i.e.  /dev/sdc1 it can use 
something like  /dev/disk/by-id/scsi-36d094660575ece002291b9e01bad8691-part1

So 'vdoimport' need to be able to match different aliases to be pointing to a single major:minor device


Also 'vdoimport' should hide some 'error' logs from command that are expected to fail.

As a workaround user needs to use the same device path as is stored  in  /etc/vdoconf.yml to pass through the conversion process.

Pushed: https://listman.redhat.com/archives/lvm-devel/2021-August/msg00044.html
to solve the most of those reports - some still need some thinking thought.

this patches should be also included:

to fix conversion of large (>2T) VDO LV
https://listman.redhat.com/archives/lvm-devel/2021-August/msg00046.html

to support lvcreate -ky
https://listman.redhat.com/archives/lvm-devel/2021-August/msg00045.html

to skip zeroing of VDO LV
https://listman.redhat.com/archives/lvm-devel/2021-August/msg00043.html

Several more patches further improving importing experience:

Fixes reporting about missing devices:

https://listman.redhat.com/archives/lvm-devel/2021-September/msg00004.html



Fixed prompting before conversion begins:

https://listman.redhat.com/archives/lvm-devel/2021-September/msg00005.html


Added support for  'async-unsafe'  vdo writePolicy:

https://listman.redhat.com/archives/lvm-devel/2021-September/msg00002.html


Corrected maxDiscardSize translated units used by lvm2:

https://listman.redhat.com/archives/lvm-devel/2021-September/msg00003.html

Scratch build: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=39647501
Repo: http://brew-task-repos.usersys.redhat.com/repos/scratch/mcsontos/lvm2/2.03.12/9.el8/

On the top of scratch build few minor patches were added:


Mandatory fix for missing error handling for non-dm devices leaked from patchset:
(and already used by Corey's scratch build testing)

https://listman.redhat.com/archives/lvm-devel/2021-September/msg00012.html



Man page improved to better clarify naming selection:

https://listman.redhat.com/archives/lvm-devel/2021-September/msg00014.html



Script returns failure  (!=0) when answering NO on conversion prompt:

https://listman.redhat.com/archives/lvm-devel/2021-September/msg00016.html

 Naming the vg can reasonably be seen as "up for grabs" as it has no counterpart in the current vdo namespace.  For the created lv it would seem preferable that, to the extent possible (i.e., minimal changes based on incompatible characters), the vdo name be preserved as it likely has some meaning to the user.

Current usage of lvm_import_vdo creates an lv (e.g., vdolvol) which has no relation to the originating vdo's name.

In a simulated system failure [exit(1) from vdo manager at the end of the convert operation] lvm_import_vdo produces output that implies success although it's exit status is non-zero and it has not created the expected vg/lv entities:

[root@pr85b-1 vdomgmnt]# vgs
  VG   #PV #LV #SN Attr   VSize    VFree
  rhel   1   3   0 wz--n- <255.00g    0 

[root@pr85b-1 vdomgmnt]# lvs
  LV   VG   Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home rhel -wi-ao---- <182.95g                                                    
  root rhel -wi-ao----   70.00g                                                    
  swap rhel -wi-ao----    2.05g                                                    

[root@pr85b-1 vdomgmnt]# vdo create --name test0 --device /dev/sdb
Creating VDO test0
      Logical blocks defaulted to 3139552 blocks.
      The VDO volume can address 12 GB in 6 data slabs, each 2 GB.
      It can grow to address at most 16 TB of physical storage in 8192 slabs.
      If a larger maximum size might be needed, use bigger slabs.
Starting VDO test0
Starting compression on VDO test0
VDO instance 8 volume is ready at /dev/mapper/test0

[root@pr85b-1 vdomgmnt]# vgs
  VG   #PV #LV #SN Attr   VSize    VFree
  rhel   1   3   0 wz--n- <255.00g    0 

[root@pr85b-1 vdomgmnt]# lvs
  LV   VG   Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home rhel -wi-ao---- <182.95g                                                    
  root rhel -wi-ao----   70.00g                                                    
  swap rhel -wi-ao----    2.05g                                                    

[root@pr85b-1 vdomgmnt]# lvm_import_vdo /dev/sdb
Convert VDO device "/dev/sdb" to VDO LV "vdovg/vdolvol"? [y|N]: Yes
Stopping VDO test0
Converting VDO test0
      Opening /dev/disk/by-id/ata-pr85b-1-0_SSD_6W6V79YY2W34N991V05H exclusively
      Loading the VDO superblock and volume geometry
      Checking the VDO state
      Converting the UDS index
      Converting the VDO
      Conversion completed for '/dev/disk/by-id/ata-pr85b-1-0_SSD_6W6V79YY2W34N991V05H': VDO is now offset by 2097152 bytes

[root@pr85b-1 vdomgmnt]# echo $?
1

[root@pr85b-1 vdomgmnt]# vgs
  VG   #PV #LV #SN Attr   VSize    VFree
  rhel   1   3   0 wz--n- <255.00g    0 

[root@pr85b-1 vdomgmnt]# lvs
  LV   VG   Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home rhel -wi-ao---- <182.95g                                                    
  root rhel -wi-ao----   70.00g                                                    
  swap rhel -wi-ao----    2.05g                                                    
[root@pr85b-1 vdomgmnt]#

Additionally if the simulated failure occurs after 'vdo convert' has removed the vdo being converted from the vdo config file attempting to convert the vdo device again will fail in one of two ways depending on whether the vdo device was the last one configured (which cause vdo to remove the config file).

If the config file is missing the failure is:
[root@pr85b-1 vdomgmnt]# lvm_import_vdo /dev/sdb
vdo: ERROR - Configuration file /etc/vdoconf.yml does not exist.

If the config file is present (due to the existence of another vdo device):
[root@pr85b-1 vdomgmnt]# lvm_import_vdo /dev/sdb
lvm_import_vdo: Can't find matching device in vdo configuration file.

In either case the result is that the vdo instance will require some manual intervention in order to effect its importation.

(In reply to Joe Shimkus from comment #35)
>  Naming the vg can reasonably be seen as "up for grabs" as it has no
> counterpart in the current vdo namespace.  For the created lv it would seem
> preferable that, to the extent possible (i.e., minimal changes based on
> incompatible characters), the vdo name be preserved as it likely has some
> meaning to the user.
> 
> Current usage of lvm_import_vdo creates an lv (e.g., vdolvol) which has no
> relation to the originating vdo's name.

The naming issue is already documented in bug https://bugzilla.redhat.com/show_bug.cgi?id=2002790.

For the rest of the issues, please file separate bugs so that this feature can move forward. I'll add a test case for the simulated vdo cmd failure, but that is not a requirement for this feature initially in 8.5. Thanks.

All of the lvm bugs mentioned in comment #23, with the exception of (https://bugzilla.redhat.com/show_bug.cgi?id=1989650) are now fixed in the latest scratch build. Since the --dry-run argument was not mentioned as a requirement of this feature originally in rhel8.5, I'm marking this verified:Tested with the latest scratch build.

lvm2-2.03.12-9.el8    BUILT: Tue Sep 14 09:53:56 CDT 2021
lvm2-libs-2.03.12-9.el8    BUILT: Tue Sep 14 09:53:56 CDT 2021

All current regression scenarios passed with the latest lvm2 and kernel. Marking VERIFIED.

kernel-4.18.0-345.el8    BUILT: Thu Sep 23 18:34:50 CDT 2021
lvm2-2.03.12-10.el8    BUILT: Mon Sep 20 03:30:20 CDT 2021
lvm2-libs-2.03.12-10.el8    BUILT: Mon Sep 20 03:30:20 CDT 2021


attempt_non_existence_device_conversion
attempt_existent_non_vdo_device_conversion
basic_vdo_create_and_convert_to_lvm
basic_already_stopped_vdo_create_and_convert_to_lvm
xfs_io_vdo_create_and_convert_to_lvm
ext3_io_vdo_create_and_convert_to_lvm
ext4_io_vdo_create_and_convert_to_lvm
gfs2_io_vdo_create_and_convert_to_lvm
vdo_create_and_convert_to_lvm_full_vdoconf_path
vdo_convert_to_lvm_with_given_vg_and_lv_name
attempt_no_flag_answer_to_lvm_import_vdo_convert
multiple_vdo_create_and_convert_with_named_vg_operations
attempt_in_use_vdovg_name_lvm_import_vdo_convert
attempt_in_use_lv_name_lvm_import_vdo_convert
multi_device_lvm_raid_lvm_import_vdo_convert
vdo_convert_dry_run
vdo_convert_dry_run_failure_attempts
multi_device_md_raid_lvm_import_vdo_convert
crypt_device_lvm_import_vdo_convert
large_vdo_logical_size_convert_to_lvm
small_vdo_logical_size_convert_to_lvm

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4431