Bug 1510558

Summary: vdoformat tries to recreate UDS index instead of creating a new one
Product: Red Hat Enterprise Linux 7 Reporter: Jakub Krysl <jkrysl>
Component: kmod-kvdoAssignee: Sweet Tea Dorminy <sweettea>
Status: CLOSED ERRATA QA Contact: Jakub Krysl <jkrysl>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.5CC: awalsh, cmarthal, limershe, sweettea
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 6.1.0.44 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 16:25:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1512127    
Bug Blocks:    

Description Jakub Krysl 2017-11-07 16:35:27 UTC
Description of problem:
When creating vdo on device where VDO already existed, the vdoformat recreates UDS index instead of creating a new one. This happens even when the superblock is zeroed beforehand.
This leads to ignoring any options user specified for UDS index. If the superblock is zeroed, there is and error in console. Example with --indexMem:

[ 3277.915812] kvdo6:dmsetup: starting device 'vdo' device instantiation 6 (ti=ffffc90000f91040) write policy sync
[ 3277.967932] kvdo6:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported
[ 3278.006901] kvdo6:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[ 3278.197801] kvdo6:dmsetup: Setting UDS index target state to online
[ 3278.226072] kvdo6:dmsetup: uds: kvdo6:dedupeQ: Initializing index session for index: dev=/dev/sda3 offset=4096 size=22512553984
uds: kvdo6:dedupeQ: Loading index from saved state and rebuilding if necessary.
[ 3278.226425] uds: kvdo6:dedupeQ: bad region table magic number: UDS Error: Corrupt saved component (1030)
[ 3278.226428] uds: kvdo6:dedupeQ: cannot read superblock header: UDS Error: Corrupt saved component (1030)
[ 3278.226439] uds: kvdo6:dedupeQ: Failed to initialize grid: UDS Error: Corrupt saved component (1030)
uds: kvdo6:dedupeQ: Closed index session (0, ffff88006d46c6c0)
uds: kvdo6:dedupeQ: Initializing index session for index: dev=/dev/sda3 offset=4096 size=22512553984
uds: kvdo6:dedupeQ: Creating new index.
uds: kvdo6:dedupeQ: Using 6 indexing zones for concurrency.

[ 3278.546773] device 'vdo' started
[ 3278.562429] kvdo6:dmsetup: resuming device 'vdo'
[ 3278.583204] kvdo6:dmsetup: device 'vdo' resumed
[ 3278.634172] kvdo6:packerQ: compression is enabled
[ 3278.659443] uds: kvdo6:dedupeQ: Created index session (9)
[ 3278.683717] uds: kvdo6:dedupeQ: Opened block context (7)

The indexMem size is correctly reflected in vdo status and in /sys/uds/index, but in device superblock is the original size on address 0x00001000, because the UDS index is rebuilt.

Also if the UDS index is not former correctly (giving --indexMem value larger than available space), this rebuilding leads to more errors.

Version-Release number of selected component (if applicable):
vdo-6.1.0.34-8
kmod-kvdo-6.1.0.34-7

How reproducible:
100%

Steps to Reproduce:
1.create vdo1
2.remove vdo1
3.dd if=/dev/zero of=DEVICE bs=4092 count=2
Step 3 is here just to produce error in console
4.create vdo2 with different --indexMem size
5.check index memory size

Actual results:
real index memory size is not changed

Expected results:
real index memory size reflects user specified size

Additional info:

Comment 2 Bryan Gurney 2017-11-07 17:01:47 UTC
One of the programs that runs as part of a "vdo create" command is "vdoformat".  However, if this command is executed as displayed in the "vdo create ... -verbose" output, it looks like this:

# vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 /dev/sdc

However, this command fails with this error message:

vdoformat: Cannot format device already containing a valid VDO!
If you are sure you want to format this device again, use the
--force option.

So instead of creating a new index, the rest of the commands in the "vdo create" sequence continue to run.  If an "--indexMem" value that isn't appropriate for the system is used, it ends up "succeeding", because it starts the original default-sized index (using 256 MB of memory, and 2.5 GB of disk), and not the intended index size (I was trying --indexMem=10, on a system with 2 GB of memory, and a 40 GB virtual disk; the system should have had neither the disk space for the index, nor the memory to run it.)

However, if you zero the VDO superblock (on this system, on block 679128, or 0xa5cd8), the "vdo create" ends up failing with a vdoformat failure (which I expect to see, since this system doesn't have enough disk space for the index):

[root@rhel75test-20171023 ~]# dd if=/dev/zero of=/dev/sdc bs=4096 seek=679128 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00295328 s, 1.4 MB/s
[root@rhel75test-20171023 ~]# hexdump -C -n 4096 -s 2781708288 /dev/sdc
a5cd8000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
a5cd9000
[root@rhel75test-20171023 ~]# vdo create --name=vdo2 --device=/dev/sdc --indexMem=10 --verbose
Creating VDO vdo2
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=10 /dev/sdc
vdo: ERROR - vdoformat: formatVDO failed on '/dev/sdc': VDO Status: Exceeds maximum number of slabs supported

Comment 4 Bryan Gurney 2017-11-13 17:05:31 UTC
*** Bug 1512618 has been marked as a duplicate of this bug. ***

Comment 6 Jakub Krysl 2017-11-14 13:17:42 UTC
Could not verify with kmod-kvdo-6.1.0.46-8.el7 and vdo-6.1.0.46-9, as the vdo refuses to create new vdo altogether:

# vdo create --device /dev/sda3 --name vdo
Creating VDO vdo
Starting VDO vdo
Starting compression on VDO vdo
VDO instance 2 volume is ready at /dev/mapper/vdo
# vdo remove --all
Removing VDO vdo
Stopping VDO vdo
# vdo create --device /dev/sda3 --name vdo
Creating VDO vdo
vdo: ERROR - vdoformat: Cannot format device already containing a valid VDO!

kmod-kvdo-6.1.0.46-8.el7  with previous vdo version vdo-6.1.0.34-8 does not work:
[ 5180.857041] kvdo6:dmsetup: release version 0 requires an upgrade: kvdo: Unsupported component version (2058)

So adding dependency BZ 1512127, which needs to be fixed before I can verify this.

Comment 7 Jakub Krysl 2017-12-04 13:37:40 UTC
BZ 1512127 is verified in vdo-6.1.0.72-12.

As of this BZ, with kmod-kvdo-6.1.0.72-10:
[ 1498.607669] kvdo1:dmsetup: starting device 'vdo' device instantiation 1 (ti=ffffb2dd81031040) write policy sync
[ 1498.657060] kvdo1:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[ 1498.805539] kvdo1:dmsetup: uds: kvdo1:dedupeQ: loading or rebuilding index: dev=/dev/mapper/mpatha offset=4096 size=2781704192
[ 1498.805903] uds: kvdo1:dedupeQ: Failed loading or rebuilding index: UDS Error: No index found (1061)
[ 1498.805910] kvdo1:dedupeQ: Error opening index dev=/dev/mapper/mpatha offset=4096 size=2781704192: UDS Error: No index found (1061)
uds: kvdo1:dedupeQ: creating index: dev=/dev/mapper/mpatha offset=4096 size=2781704192
uds: kvdo1:dedupeQ: Using 6 indexing zones for concurrency.

[ 1499.018973] Setting UDS index target state to online
[ 1499.042948] kvdo1:dmsetup: device 'vdo' started
[ 1499.064259] kvdo1:dmsetup: resuming device 'vdo'
[ 1499.085649] kvdo1:dmsetup: device 'vdo' resumed
[ 1499.124396] kvdo1:packerQ: compression is enabled

kvdo tried to load the index, but there is no index to load. So it creates a new index as it should. As of vdo-6.1.0.72-12 there should never be an index present, as vdo removes the superblock at the end of every removal.

Comment 10 errata-xmlrpc 2018-04-10 16:25:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0900