RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1507999 - Interruption vdo create before superblock is created hangs the vdo without enough info to remove it
Summary: Interruption vdo create before superblock is created hangs the vdo without en...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: vdo
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Joe Shimkus
QA Contact: Jakub Krysl
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-31 16:25 UTC by Jakub Krysl
Modified: 2019-03-06 01:13 UTC (History)
5 users (show)

Fixed In Version: 6.1.0.40
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 15:46:34 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1508920 0 unspecified CLOSED vdo manager 'create/remove' command documentation 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHEA-2018:0871 0 None None None 2018-04-10 15:47:21 UTC

Internal Links: 1508920

Description Jakub Krysl 2017-10-31 16:25:40 UTC
Description of problem:
When interrupting (CTRL+C) "vdo create" at the very begginning (to ~1 sec from start), it cannot be removed by "vdo remove" nor by following guidelines given by that command. Information about the vdo has to be manually removed from /etc/vdoconf.yml.

Version-Release number of selected component (if applicable):
6.1.0.0.6

How reproducible:
100%

Steps to Reproduce:
1. vdo create --device /dev/sdc --name vdo
2. CTRL + C
3. vdo remove --all

Actual results:
# vdo remove --all
Removing VDO vdo
Stopping VDO vdo
Steps to clean up VDO vdo:
    umount -f /dev/mapper/vdo
    udevadm settle
    dmsetup remove vdo
vdo: ERROR - VDO volume vdo previous operation is incomplete

Expected results:
# vdo remove --all
Removing VDO vdo
Stopping VDO vdo
Steps to clean up VDO vdo:
    umount -f /dev/mapper/vdo
    udevadm settle
    dmsetup remove vdo
    remove said vdo from /etc/vdoconf.yml
vdo: ERROR - VDO volume vdo previous operation is incomplete

Additional info:

Comment 2 Bryan Gurney 2017-10-31 18:04:39 UTC
The title of "before superblock is created" may be a misinterpretation.  I was able to reproduce this; here's the output of an interrupted "vdo create" command with verbose output:

$ sudo vdo create --name=vdo1 --device=/dev/md0 --vdoLogicalSize=4T --verboseCreating VDO vdo1
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --logical-size=4T /dev/md0
    vdodumpconfig /dev/md0
Starting VDO vdo1
    dmsetup status vdo1
    modprobe kvdo
    dmsetup create vdo1 --table '0 8589934592 dedupe /dev/md0 4096 disabled 0 32768 16380 on sync vdo1 ack=1,bio=4,bioRotationInterval=64,cpu=2,hash=1,logical=1,physical=1'
^X^CTraceback (most recent call last):
  File "/usr/bin/vdo", line 781, in <module>
    main()
  File "/usr/bin/vdo", line 764, in main
    operation.run(options)
  File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOOperation.py", line 197, in run
    self.execute(args)
  File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOOperation.py", line 49, in wrap
    return lock(False, func, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOOperation.py", line 44, in lock
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/utils/Transaction.py", line 196, in wrap
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOOperation.py", line 304, in execute
    vdo.create()
  File "/usr/lib/python2.7/site-packages/utils/Transaction.py", line 196, in wrap
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOService.py", line 256, in create
    self._constructServiceStart()
  File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOService.py", line 1137, in _constructServiceStart
    self.start()
  File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOService.py", line 485, in start
    "--table", self._generateDeviceMapperTable()])
  File "/usr/lib/python2.7/site-packages/utils/Command.py", line 278, in runCommand
    return Command(commandList, kwargs.pop('environment', None)).run(**kwargs)
  File "/usr/lib/python2.7/site-packages/utils/Command.py", line 152, in run
    output = self._execute(stdin)
  File "/usr/lib/python2.7/site-packages/utils/Command.py", line 235, in _execute
    stdoutdata, stderrdata = p.communicate(stdin)
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt

At this point, if I run "dmsetup table", I see an empty table entry:

$ sudo dmsetup table
vdo1: 

Yes, that output is correct: a device called "vdo1" with no parameters.  There isn't a "/dev/mapper" symlink, but there is a 253:0 device:

$ ls -l /dev/dm-*
brw-rw---- 1 root disk 253, 0 Oct 31 13:56 /dev/dm-0

This can be removed with "sudo dmsetup remove vdo1", but after this, the configuration for this VDO volume needs to be removed from /etc/vdoconf.yml to fully clean up, and allow a new VDO volume to be created.

Comment 3 Bryan Gurney 2017-10-31 18:47:41 UTC
The "--force" switch" of the "vdo remove" command should also be able to cleanly remove the partially removed device:

$ sudo vdo remove --name=vdo1 --force --verbose
Removing VDO vdo1
Stopping VDO vdo1
    dmsetup status vdo1
    mount
    udevadm settle
    dmsetup remove vdo1
[bgurney@parkst-08 ~]$ ls /etc/vdoconf.yml
ls: cannot access /etc/vdoconf.yml: No such file or directory

Comment 4 Joe Shimkus 2017-11-09 17:17:37 UTC
Documentation from command updated.
Bug 1508920 logged for modification to official documentation.

Comment 6 Jakub Krysl 2017-11-14 15:24:38 UTC
Tested with vdo-6.1.0.46-9:

# vdo remove --all --verbose
Removing VDO vdo
Stopping VDO vdo
    mount
A previous operation failed.
Recovery from the failure either failed or was interrupted.
Add '--force' to 'remove' to perform the following cleanup.
Steps to clean up VDO vdo:
    umount -f /dev/mapper/vdo
    udevadm settle
    dmsetup remove vdo
vdo: ERROR - VDO volume vdo previous operation (create) is incomplete


The manual steps are still missing removal of stuck VDO from /etc/vdoconf.yml, so just following them is not enough now:

# umount -f /dev/mapper/vdo
umount: /dev/mapper/vdo: mountpoint not found
# udevadm settle
# msetup remove vdo
-bash: msetup: command not found
# vdo remove --all
Removing VDO vdo
Stopping VDO vdo
A previous operation failed.
Recovery from the failure either failed or was interrupted.
Add '--force' to 'remove' to perform the following cleanup.
Steps to clean up VDO vdo:
    umount -f /dev/mapper/vdo
    udevadm settle
    dmsetup remove vdo
vdo: ERROR - VDO volume vdo previous operation (create) is incomplete

Jow, can you please rephrase "removal of stuck VDO from /etc/vdoconf.yml" and add it there?

Comment 7 Joe Shimkus 2017-11-14 16:47:02 UTC
Performing the remove via vdo manager will remove the entry from the vdoconf.yml.
If that is not happening that would be a separate problem.

Comment 8 Joe Shimkus 2017-11-14 17:02:53 UTC
Please note that the output of the remove in this situation is advisory to the administrator regarding any changes that could impact the "physical" configuration of the system.  It is not intended to indicate manual steps that the admin should perform, but those that vdo manager will execute when run with 'remove --force'.

Comment 9 Jakub Krysl 2017-11-16 11:30:36 UTC
There still might be some confusion, because 'remove --force' does not only do the listed steps, but also removes the entry from /etc/vdoconf.cfg. As the file definitely should not be altered manually, it is probably better not to mention that step in this list.

Setting to verified as the --force option works correctly.

Comment 12 errata-xmlrpc 2018-04-10 15:46:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0871


Note You need to log in before you can comment on or make changes to this bug.