RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1520972 - vdoformat does not clean up metadata properly after fail
Summary: vdoformat does not clean up metadata properly after fail
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: vdo
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: corwin
QA Contact: Jakub Krysl
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-05 14:45 UTC by Jakub Krysl
Modified: 2019-03-06 00:40 UTC (History)
8 users (show)

Fixed In Version: 6.1.0.123
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 15:48:32 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:0871 0 None None None 2018-04-10 15:49:01 UTC

Description Jakub Krysl 2017-12-05 14:45:19 UTC 
Description of problem:
When creating VDO, vdoformat is called to set up the volume. I found a situation where it fails and leaves new metadata on device, which prevents new vdo to be created there. The approach BZ 1512127 took to fix this does not prevent this behaviour, as vdo remove does not know it should clean this device.
There might be more ways to hit this, but the one I found is using --vdoSlabSize 128M on 5.5T disk to hit slab count limit (8192):


# hexdump -C /dev/sdc -n 8192
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000

# vdo create --name vdo --device /dev/sdc --verbose --vdoSlabSize 128M
Creating VDO vdo
    pvcreate -qq --test /dev/sdc
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --slab-bits=15 /dev/sdc
vdo: ERROR - vdoformat: formatVDO failed on '/dev/sdc': VDO Status: Exceeds maximum number of slabs supported

# hexdump -C /dev/sdc -n 8192
00000000  64 6d 76 64 6f 30 30 31  05 00 00 00 04 00 00 00  |dmvdo001........|
00000010  00 00 00 00 5d 00 00 00  00 00 00 00 09 01 02 00  |....]...........|
00000020  59 a2 aa fe 9c 5f 05 00  00 b4 8f db 69 0d 46 a1  |Y...._......i.F.|
00000030  be a2 a6 6c 66 c8 ee 8f  00 00 00 00 01 00 00 00  |...lf...........|
00000040  00 00 00 00 01 00 00 00  d8 5c 0a 00 00 00 00 00  |.........\......|
00000050  00 ff ff ff 00 00 00 00  00 fe 96 a1 43 00 00 00  |............C...|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000

# vdo create --name vdo --device /dev/sdc --verbose
Creating VDO vdo
    pvcreate -qq --test /dev/sdc
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 /dev/sdc
vdo: ERROR - vdoformat: Cannot format device already containing a valid VDO!


vdo remove cannot fix this:
# vdo remove --all
# vdo remove --all --force
# vdo remove --device /dev/sdc --all
# vdo remove --device /dev/sdc --all --force
# vdo create --name vdo --device /dev/sdc --verbose
Creating VDO vdo
    pvcreate -qq --test /dev/sdc
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 /dev/sdc
vdo: ERROR - vdoformat: Cannot format device already containing a valid VDO!


vdo create --force can:
# vdo create --name vdo --device /dev/sdc --verbose --force
Creating VDO vdo
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --force /dev/sdc
    vdodumpconfig /dev/sdc
Starting VDO vdo
    dmsetup status vdo
    modprobe kvdo
    vdodumpconfig /dev/sdc
    dmsetup create vdo --uuid VDO-708f6f14-3b68-4e07-9131-c1549c450661 --table '0 11721045168 dedupe /dev/sdc 4096 disabled 0 32768 16380 on sync vdo ack=1,bio=4,bioRotationInterval=64,cpu=2,hash=1,logical=1,physical=1'
    dmsetup status vdo
Starting compression on VDO vdo
    dmsetup message vdo 0 compression on
    dmsetup status vdo
VDO instance 25 volume is ready at /dev/mapper/vdo


But to user there is no hint to use vdo create --force not I believe it is the best approach, as this situation can be avoided with cleanup in case vdoformat fails. (checking the superblock for metadata and erasing it)

Version-Release number of selected component (if applicable):
vdo-6.1.0.72-12

How reproducible:
100%

Steps to Reproduce:
1. vdo create --name vdo --device /dev/sdc --verbose --vdoSlabSize 128M on large disk to exceed 8192 slabs.

Actual results:
some metadata created on the device

Expected results:
no metadata created on the device

Additional info:
Comment 2 bjohnsto 2017-12-05 18:34:16 UTC 
I think this is a vdo manager issue.
Comment 3 corwin 2017-12-06 18:02:05 UTC 
This can be fixed in vdoformat.
Comment 5 Jakub Krysl 2017-12-14 11:13:53 UTC 
Tested with vdo-6.1.0.98-13:

# vdo create --name vdo --device /dev/sdc --verbose --vdoSlabSize 128M
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate -qq --test /dev/sdc
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --slab-bits=15 /dev/sdc
vdo: ERROR - vdoformat: formatVDO failed on '/dev/sdc': VDO Status: Exceeds maximum number of slabs supported

# hexdump -C /dev/sdc -n 8192
00000000  64 6d 76 64 6f 30 30 31  05 00 00 00 04 00 00 00  |dmvdo001........|
00000010  00 00 00 00 5d 00 00 00  00 00 00 00 09 01 02 00  |....]...........|
00000020  51 04 22 23 4f 60 05 00  31 15 6f a5 0a e7 46 46  |Q."#O`..1.o...FF|
00000030  91 79 b2 b7 4a e6 82 de  00 00 00 00 01 00 00 00  |.y..J...........|
00000040  00 00 00 00 01 00 00 00  d8 5c 0a 00 00 00 00 00  |.........\......|
00000050  00 ff ff ff 00 00 00 00  00 91 39 37 8e 00 00 00  |..........97....|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000

# vdo create --name vdo --device /dev/sdc --verbose
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate -qq --test /dev/sdc
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 /dev/sdc
vdo: ERROR - vdoformat: Cannot format device already containing a valid VDO!


This bug is still reproducible, giving back.
Comment 6 Radka Brychtova 2018-01-08 15:47:55 UTC 
Hi, I just hit in similar problem when I tested the blkid command for vdo device, and I never used vdo before. So i just add scsi device and tried to create vdo device. This command failed with message "Out of space"
And when I tried test blkid command - there was the vdo type set even if the command vdo failed before.

# modprobe scsi_debug dev_size_mb=1000
# blkid /dev/sda
<nothing>
#vdo create --name=vdo1 --device=/dev/sda --vdoLogicalSize=1M
Creating VDO vdo1
vdo: ERROR - vdoformat: formatVDO failed on '/dev/sda': VDO Status: Out of space
<return code 2>
# blkid /dev/sda
/dev/sda: UUID="801b85ef-bc0f-40f4-ae82-3a1cbe0f715c" TYPE="vdo" 

I assume it should NOT set the type if the vdo command failed.

#rpm -q util-linux vdo
util-linux-2.23.2-49.el7.x86_64
vdo-6.1.0.98-13.x86_64
Comment 8 corwin 2018-02-06 21:42:53 UTC 
This was fixed in version 6.1.0.123 and should have been in the January 19th push.
The fix is to zero the geometry block at the start of format, and then only write a valid geometry block after a valid super block has been written.

The steps to reproduce listed above should now show that the problem does not recur.
Comment 10 Jakub Krysl 2018-02-12 15:22:23 UTC 
Now the superblock is deleted even if the vdoFormat fails, so there is no metadata remaining:
# vdo create --name vdo --device /dev/sdc --verbose --vdoSlabSize 128M
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate -qq --test /dev/sdc
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --slab-bits=15 /dev/sdc
vdo: ERROR - vdoformat: formatVDO failed on '/dev/sdc': VDO Status: Exceeds maximum number of slabs supported

# hexdump -C /dev/sdc -n 4K
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

As there is checking before running vdoFormat to be sure there is nothing on the device already (using pvcreate --test), this should not be an issue as long as user does not use --force in which case he should know what he is doing.
Regression testing found no issues with this new approach.
Comment 13 errata-xmlrpc 2018-04-10 15:48:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0871
Note You need to log in before you can comment on or make changes to this bug.