Bug 1507999
| Summary: | Interruption vdo create before superblock is created hangs the vdo without enough info to remove it | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Jakub Krysl <jkrysl> |
| Component: | vdo | Assignee: | Joe Shimkus <jshimkus> |
| Status: | CLOSED ERRATA | QA Contact: | Jakub Krysl <jkrysl> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.5 | CC: | awalsh, bgurney, bjohnsto, jshimkus, limershe |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | 6.1.0.40 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-04-10 15:46:34 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The title of "before superblock is created" may be a misinterpretation. I was able to reproduce this; here's the output of an interrupted "vdo create" command with verbose output:
$ sudo vdo create --name=vdo1 --device=/dev/md0 --vdoLogicalSize=4T --verboseCreating VDO vdo1
modprobe kvdo
vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --logical-size=4T /dev/md0
vdodumpconfig /dev/md0
Starting VDO vdo1
dmsetup status vdo1
modprobe kvdo
dmsetup create vdo1 --table '0 8589934592 dedupe /dev/md0 4096 disabled 0 32768 16380 on sync vdo1 ack=1,bio=4,bioRotationInterval=64,cpu=2,hash=1,logical=1,physical=1'
^X^CTraceback (most recent call last):
File "/usr/bin/vdo", line 781, in <module>
main()
File "/usr/bin/vdo", line 764, in main
operation.run(options)
File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOOperation.py", line 197, in run
self.execute(args)
File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOOperation.py", line 49, in wrap
return lock(False, func, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOOperation.py", line 44, in lock
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/utils/Transaction.py", line 196, in wrap
result = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOOperation.py", line 304, in execute
vdo.create()
File "/usr/lib/python2.7/site-packages/utils/Transaction.py", line 196, in wrap
result = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOService.py", line 256, in create
self._constructServiceStart()
File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOService.py", line 1137, in _constructServiceStart
self.start()
File "/usr/lib/python2.7/site-packages/vdomgmnt/VDOService.py", line 485, in start
"--table", self._generateDeviceMapperTable()])
File "/usr/lib/python2.7/site-packages/utils/Command.py", line 278, in runCommand
return Command(commandList, kwargs.pop('environment', None)).run(**kwargs)
File "/usr/lib/python2.7/site-packages/utils/Command.py", line 152, in run
output = self._execute(stdin)
File "/usr/lib/python2.7/site-packages/utils/Command.py", line 235, in _execute
stdoutdata, stderrdata = p.communicate(stdin)
File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
return self._communicate(input)
File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
stdout, stderr = self._communicate_with_poll(input)
File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
ready = poller.poll()
KeyboardInterrupt
At this point, if I run "dmsetup table", I see an empty table entry:
$ sudo dmsetup table
vdo1:
Yes, that output is correct: a device called "vdo1" with no parameters. There isn't a "/dev/mapper" symlink, but there is a 253:0 device:
$ ls -l /dev/dm-*
brw-rw---- 1 root disk 253, 0 Oct 31 13:56 /dev/dm-0
This can be removed with "sudo dmsetup remove vdo1", but after this, the configuration for this VDO volume needs to be removed from /etc/vdoconf.yml to fully clean up, and allow a new VDO volume to be created.
The "--force" switch" of the "vdo remove" command should also be able to cleanly remove the partially removed device:
$ sudo vdo remove --name=vdo1 --force --verbose
Removing VDO vdo1
Stopping VDO vdo1
dmsetup status vdo1
mount
udevadm settle
dmsetup remove vdo1
[bgurney@parkst-08 ~]$ ls /etc/vdoconf.yml
ls: cannot access /etc/vdoconf.yml: No such file or directory
Documentation from command updated. Bug 1508920 logged for modification to official documentation. Tested with vdo-6.1.0.46-9:
# vdo remove --all --verbose
Removing VDO vdo
Stopping VDO vdo
mount
A previous operation failed.
Recovery from the failure either failed or was interrupted.
Add '--force' to 'remove' to perform the following cleanup.
Steps to clean up VDO vdo:
umount -f /dev/mapper/vdo
udevadm settle
dmsetup remove vdo
vdo: ERROR - VDO volume vdo previous operation (create) is incomplete
The manual steps are still missing removal of stuck VDO from /etc/vdoconf.yml, so just following them is not enough now:
# umount -f /dev/mapper/vdo
umount: /dev/mapper/vdo: mountpoint not found
# udevadm settle
# msetup remove vdo
-bash: msetup: command not found
# vdo remove --all
Removing VDO vdo
Stopping VDO vdo
A previous operation failed.
Recovery from the failure either failed or was interrupted.
Add '--force' to 'remove' to perform the following cleanup.
Steps to clean up VDO vdo:
umount -f /dev/mapper/vdo
udevadm settle
dmsetup remove vdo
vdo: ERROR - VDO volume vdo previous operation (create) is incomplete
Jow, can you please rephrase "removal of stuck VDO from /etc/vdoconf.yml" and add it there?
Performing the remove via vdo manager will remove the entry from the vdoconf.yml. If that is not happening that would be a separate problem. Please note that the output of the remove in this situation is advisory to the administrator regarding any changes that could impact the "physical" configuration of the system. It is not intended to indicate manual steps that the admin should perform, but those that vdo manager will execute when run with 'remove --force'. There still might be some confusion, because 'remove --force' does not only do the listed steps, but also removes the entry from /etc/vdoconf.cfg. As the file definitely should not be altered manually, it is probably better not to mention that step in this list. Setting to verified as the --force option works correctly. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:0871 |
Description of problem: When interrupting (CTRL+C) "vdo create" at the very begginning (to ~1 sec from start), it cannot be removed by "vdo remove" nor by following guidelines given by that command. Information about the vdo has to be manually removed from /etc/vdoconf.yml. Version-Release number of selected component (if applicable): 6.1.0.0.6 How reproducible: 100% Steps to Reproduce: 1. vdo create --device /dev/sdc --name vdo 2. CTRL + C 3. vdo remove --all Actual results: # vdo remove --all Removing VDO vdo Stopping VDO vdo Steps to clean up VDO vdo: umount -f /dev/mapper/vdo udevadm settle dmsetup remove vdo vdo: ERROR - VDO volume vdo previous operation is incomplete Expected results: # vdo remove --all Removing VDO vdo Stopping VDO vdo Steps to clean up VDO vdo: umount -f /dev/mapper/vdo udevadm settle dmsetup remove vdo remove said vdo from /etc/vdoconf.yml vdo: ERROR - VDO volume vdo previous operation is incomplete Additional info: