RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1620572 - vdodumpconfig: Failed to make FileLayer from '/dev/disk/by-id/*' with No such file or directory
Summary: vdodumpconfig: Failed to make FileLayer from '/dev/disk/by-id/*' with No such...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: vdo
Version: 7.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Thomas Jaskiewicz
QA Contact: Jakub Krysl
URL:
Whiteboard:
Depends On:
Blocks: 1627859
TreeView+ depends on / blocked
 
Reported: 2018-08-23 08:06 UTC by Jakub Krysl
Modified: 2019-08-06 13:08 UTC (History)
8 users (show)

Fixed In Version: 6.1.2.17
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1627859 (view as bug list)
Environment:
Last Closed: 2019-08-06 13:08:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2233 0 None None None 2019-08-06 13:08:14 UTC

Description Jakub Krysl 2018-08-23 08:06:17 UTC 
Description of problem:
When creating VDO over device with PV, VDO does not see the PV and overwrites it. Than it fails to dump config with $SUBJECT error.

# pvcreate /dev/sda
  Physical volume "/dev/sda" successfully created.

# hexdump -C /dev/sda -n 4K
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00000210  4f 00 30 11 20 00 00 00  4c 56 4d 32 20 30 30 31  |O.0. ...LVM2 001|
00000220  75 4f 4c 42 54 43 31 75  41 6a 37 75 61 57 4b 50  |uOLBTC1uAj7uaWKP|
00000230  76 6d 4b 33 6a 59 6a 4a  36 67 46 58 41 55 56 38  |vmK3jYjJ6gFXAUV8|
00000240  00 00 00 80 18 09 00 00  00 00 10 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 0f 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  02 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

# vdo create --name vdo --device /dev/sda --verbose
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate -qq --test /dev/sda
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 /dev/disk/by-id/lvm-pv-uuid-uOLBTC-1uAj-7uaW-KPvm-K3jY-jJ6g-FXAUV8
    vdodumpconfig /dev/disk/by-id/lvm-pv-uuid-uOLBTC-1uAj-7uaW-KPvm-K3jY-jJ6g-FXAUV8
Removing VDO vdo
Stopping VDO vdo
    dmsetup status vdo
vdo: ERROR - vdodumpconfig: Failed to make FileLayer from '/dev/disk/by-id/lvm-pv-uuid-uOLBTC-1uAj-7uaW-KPvm-K3jY-jJ6g-FXAUV8' with No such file or directory

# hexdump -C /dev/sda -n 4K
00000000  64 6d 76 64 6f 30 30 31  05 00 00 00 04 00 00 00  |dmvdo001........|
00000010  00 00 00 00 5d 00 00 00  00 00 00 00 09 01 02 00  |....]...........|
00000020  2a ae 5c 8a 15 74 05 00  b0 a2 b2 36 62 dc 4b 9f  |*.\..t.....6b.K.|
00000030  81 90 00 eb 29 ad e8 1e  00 00 00 00 01 00 00 00  |....)...........|
00000040  00 00 00 00 01 00 00 00  d8 5c 0a 00 00 00 00 00  |.........\......|
00000050  00 ff ff ff 00 00 00 00  00 17 80 b6 c1 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

Version-Release number of selected component (if applicable):
vdo-6.1.1.120-3.el7.x86_64 

How reproducible:
100%

Steps to Reproduce:
1. pvcreate /dev/sda
2. vdo create --name vdo --device /dev/sda

Actual results:
VDO is not created and existing PV is overwritten

Expected results:
VDO is created

Additional info:
Comment 2 Jakub Krysl 2018-08-23 08:10:22 UTC 
Lvm handles pvcreate by running pvcreate on existing PV creates the PV again. Because of this behaviour 'pvcreate -qq --test' passes and the whole creation continues. But changing dumpconfig to /dev/disk/by-id' points to nonexistent device now, in 7.5 this was not an issue:

# pvcreate /dev/sda
   Physical volume "/dev/sda" successfully created.

# hexdump -C /dev/sda -n 4k
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00000210  82 fc 41 16 20 00 00 00  4c 56 4d 32 20 30 30 31  |..A. ...LVM2 001|
00000220  43 4c 62 57 49 71 52 30  66 53 73 39 6b 63 59 39  |CLbWIqR0fSs9kcY9|
00000230  78 30 6f 6c 58 37 69 6c  63 5a 49 56 4c 65 77 43  |x0olX7ilcZIVLewC|
00000240  00 00 00 80 02 00 00 00  00 00 10 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 0f 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  02 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

# vdo create --name vdo --device /dev/sda --verbose
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate -qq --test /dev/sda
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 /dev/sda
    vdodumpconfig /dev/sda
Starting VDO vdo
    dmsetup status vdo
    grep MemAvailable /proc/meminfo
    modprobe kvdo
    vdodumpconfig /dev/sda
    dmsetup create vdo --uuid VDO-ef88f279-f4ec-4dc5-bcc0-32674bae9bbc --table '0 12556528 vdo /dev/sda 4096 disabled 0 32768 16380 on auto vdo ack=1,bio=4,bioRotationInterval=64,cpu=2,hash=1,logical=1,physical=1'
    dmsetup status vdo
Starting compression on VDO vdo
    dmsetup message vdo 0 compression on
    dmsetup status vdo
VDO instance 1 volume is ready at /dev/mapper/vdo

# hexdump -C /dev/sda -n 4k
00000000  64 6d 76 64 6f 30 30 31  05 00 00 00 04 00 00 00  |dmvdo001........|
00000010  00 00 00 00 5d 00 00 00  00 00 00 00 09 01 02 00  |....]...........|
00000020  f1 44 60 b5 15 74 05 00  ef 88 f2 79 f4 ec 4d c5  |.D`..t.....y..M.|
00000030  bc c0 32 67 4b ae 9b bc  00 00 00 00 01 00 00 00  |..2gK...........|
00000040  00 00 00 00 01 00 00 00  d8 5c 0a 00 00 00 00 00  |.........\......|
00000050  00 ff ff ff 00 00 00 00  00 32 cf bc 99 00 00 00  |.........2......|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000 


Setting to regression because of this.
Comment 6 Jakub Krysl 2019-04-29 15:27:30 UTC 
kernel-3.10.0-1040.el7.x86_64
vdo-6.1.2.41-4.el7.x86_64
kmod-kvdo-6.1.2.41-5.el7.x86_64


# pvcreate /dev/sda
  Physical volume "/dev/sda" successfully created.

# hexdump -C /dev/sda -n 4k
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00000210  f0 5d b8 e6 20 00 00 00  4c 56 4d 32 20 30 30 31  |.].. ...LVM2 001|
00000220  4d 49 6b 39 50 53 6a 61  76 4e 47 34 62 57 55 5a  |MIk9PSjavNG4bWUZ|
00000230  6c 39 73 56 62 30 4a 73  6b 5a 5a 76 63 77 56 54  |l9sVb0JskZZvcwVT|
00000240  00 00 00 80 18 09 00 00  00 00 10 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 0f 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  02 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

# vdo create --name vdo --device /dev/sda --verbose
    mkdir -p /run/lock/vdo
    touch /run/lock/vdo/_etc_vdoconf.yml.lock
    chmod 644 /run/lock/vdo/_etc_vdoconf.yml.lock
    touch /run/lock/vdo-config-singletons
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate --config devices/scan_lvs=1 -qq --test /dev/sda
    blkid -p /dev/sda
vdo: ERROR - device is a physical volume; use --force to override

# vdo create --name vdo --device /dev/sda --verbose
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate --config devices/scan_lvs=1 -qq --test /dev/sda
    blkid -p /dev/sda
vdo: ERROR - device is a physical volume; use --force to override

# vdo create --name vdo --device /dev/sda --verbose --force
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --force /dev/disk/by-id/lvm-pv-uuid-MIk9PS-javN-G4bW-UZl9-sVb0-JskZ-ZvcwVT
vdo r    vdodumpconfig /dev/disk/by-id/lvm-pv-uuid-MIk9PS-javN-G4bW-UZl9-sVb0-JskZ-ZvcwVT
Removing VDO vdo
vdo: ERROR - vdodumpconfig: Failed to make FileLayer from '/dev/disk/by-id/lvm-pv-uuid-MIk9PS-javN-G4bW-UZl9-sVb0-JskZ-ZvcwVT' with No such file or directory

# hexdump -C /dev/sda -n 4k
00000000  64 6d 76 64 6f 30 30 31  05 00 00 00 04 00 00 00  |dmvdo001........|
00000010  00 00 00 00 5d 00 00 00  00 00 00 00 09 01 02 00  |....]...........|
00000020  f5 b9 5f d0 ac 87 05 00  b2 c4 7b 12 6a 7b 44 af  |.._.......{.j{D.|
00000030  85 b5 cd 6e 19 3f 24 0b  00 00 00 00 01 00 00 00  |...n.?$.........|
00000040  00 00 00 00 01 00 00 00  d8 5c 0a 00 00 00 00 00  |.........\......|
00000050  00 ff ff ff 00 00 00 00  00 34 c9 4a 56 00 00 00  |.........4.JV...|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000


So the new issue is fixed, there is a warning now. But the warning states "use --force to overwrite", which does not work. It correctly overwrites the PV, but as it takes the PV device from /dev/disk/by-id, it is nonexistent at that point and fail with $SUBJECT.
Although user can run the command again and it passes, it should pass on 1st run without errors. Returning to be fixed.
Comment 7 Dennis Keefe 2019-05-01 15:34:16 UTC 
The original issue was resolved, which is to day that vdo create will no longer overwrite the lvm pv header.
Running vdo create with --force sounds and getting an error sounds like a new bug.  
I would vote to close this bug and open a new one, which can be fixed in the next release.
The workaround can be to run vdo create --force twice.  We could document this in the release notes.
Comment 8 Jakub Krysl 2019-05-02 07:20:22 UTC 
(In reply to Dennis Keefe from comment #7)
> The original issue was resolved, which is to day that vdo create will no
> longer overwrite the lvm pv header.
> Running vdo create with --force sounds and getting an error sounds like a
> new bug.  
> I would vote to close this bug and open a new one, which can be fixed in the
> next release.
> The workaround can be to run vdo create --force twice.  We could document
> this in the release notes.

I returned it because the new behaviour seems exactly the same as the original one reported. We did not have --force at that time, so that is the only difference. But it does not change the behaviour. The VDO fails to create and the PV is overwritten. Thanks to --force there is the ability to run the command again as a workaround. So both ways (fix it here / close this with verified and new BZ) are fine with me.
Comment 9 Andy Walsh 2019-05-05 22:18:10 UTC 
I agree with Dennis that the behavior is related to a different bug.

Bug 1: If there is an LVM PV signature, VDO Manager would go ahead and overwrite the device with its own signature.
Bug 2: The resolved block device path (/dev/disk/by-id/*) should always resolve to something that doesn't rely on the contents of that device (like an LVM PV signature).

The reason I have this opinion is because you can create a PV on a volume, then run vdo create explicitly against a more valid /dev/disk/by-id path (like scsi-wwn, for example) and not trigger this condition.   Here's an example when there's a better by-id path available:

# vdo create --name vdo0 --device /dev/sdb1 --verbose
    mkdir -p /run/lock/vdo
    touch /run/lock/vdo/_etc_vdoconf.yml.lock
    chmod 644 /run/lock/vdo/_etc_vdoconf.yml.lock
    touch /run/lock/vdo-config-singletons
Creating VDO vdo0
    grep MemAvailable /proc/meminfo
    pvcreate --config devices/scan_lvs=1 -qq --test /dev/sdb1
    blkid -p --no-part-details /dev/sdb1
vdo: ERROR - device is a physical volume; use --force to override

# vdo create --name vdo0 --device /dev/sdb1 --verbose --force
Creating VDO vdo0
    grep MemAvailable /proc/meminfo
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --force /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1
    vdodumpconfig /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1
Starting VDO vdo0
    dmsetup status --target vdo vdo0
    grep MemAvailable /proc/meminfo
    modprobe kvdo
    vdodumpconfig /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1
    vdodumpconfig /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1
    dmsetup create vdo0 --uuid VDO-bbc27fb2-bcfb-4d63-b4e7-d2acfce2b9de --table '0 481423736 vdo V2 /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1 61049344 4096 32768 16380 on auto vdo0 maxDiscard 1 ack 1 bio 4 bioRotationInterval 64 cpu 2 hash 1 logical 1 physical 1'
    dmsetup status --target vdo vdo0
Starting compression on VDO vdo0
    dmsetup message vdo0 0 compression on
    vdodmeventd -r vdo0
    dmsetup status --target vdo vdo0
    dmsetup status --target vdo vdo0
VDO instance 0 volume is ready at /dev/mapper/vdo0


Either way to look at this, I don't think this should be a blocker+ for 7.7, just like I didn't for 7.6 mentioned in comment#4.
Comment 11 Andy Walsh 2019-05-07 15:29:09 UTC 
Moving back to ON_QA, after discussing with Jakub and having BZ1707481 opened to track the other behavior issue.
Comment 12 Jakub Krysl 2019-05-07 15:33:15 UTC 
From description:
"When creating VDO over device with PV, VDO does not see the PV and overwrites it. Than it fails to dump config with $SUBJECT error."
Setting to VERIFIED as the 1st part of this issue is gone, VDO no longer overwrites PV by itself, VDO now requires --force from user. This fails on 1st try, which is target of BZ 1707481 to be fixed.
Comment 14 errata-xmlrpc 2019-08-06 13:08:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2233
Note You need to log in before you can comment on or make changes to this bug.