Bug 1620572

Summary: vdodumpconfig: Failed to make FileLayer from '/dev/disk/by-id/*' with No such file or directory
Product: Red Hat Enterprise Linux 7 Reporter: Jakub Krysl <jkrysl>
Component: vdoAssignee: Thomas Jaskiewicz <tjaskiew>
Status: CLOSED ERRATA QA Contact: Jakub Krysl <jkrysl>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.6CC: awalsh, bgurney, dkeefe, jkrysl, jochapma, jomurphy, limershe, pasik
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 6.1.2.17 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1627859 (view as bug list) Environment:
Last Closed: 2019-08-06 13:08:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1627859    

Description Jakub Krysl 2018-08-23 08:06:17 UTC 
Description of problem:
When creating VDO over device with PV, VDO does not see the PV and overwrites it. Than it fails to dump config with $SUBJECT error.

# pvcreate /dev/sda
  Physical volume "/dev/sda" successfully created.

# hexdump -C /dev/sda -n 4K
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00000210  4f 00 30 11 20 00 00 00  4c 56 4d 32 20 30 30 31  |O.0. ...LVM2 001|
00000220  75 4f 4c 42 54 43 31 75  41 6a 37 75 61 57 4b 50  |uOLBTC1uAj7uaWKP|
00000230  76 6d 4b 33 6a 59 6a 4a  36 67 46 58 41 55 56 38  |vmK3jYjJ6gFXAUV8|
00000240  00 00 00 80 18 09 00 00  00 00 10 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 0f 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  02 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

# vdo create --name vdo --device /dev/sda --verbose
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate -qq --test /dev/sda
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 /dev/disk/by-id/lvm-pv-uuid-uOLBTC-1uAj-7uaW-KPvm-K3jY-jJ6g-FXAUV8
    vdodumpconfig /dev/disk/by-id/lvm-pv-uuid-uOLBTC-1uAj-7uaW-KPvm-K3jY-jJ6g-FXAUV8
Removing VDO vdo
Stopping VDO vdo
    dmsetup status vdo
vdo: ERROR - vdodumpconfig: Failed to make FileLayer from '/dev/disk/by-id/lvm-pv-uuid-uOLBTC-1uAj-7uaW-KPvm-K3jY-jJ6g-FXAUV8' with No such file or directory

# hexdump -C /dev/sda -n 4K
00000000  64 6d 76 64 6f 30 30 31  05 00 00 00 04 00 00 00  |dmvdo001........|
00000010  00 00 00 00 5d 00 00 00  00 00 00 00 09 01 02 00  |....]...........|
00000020  2a ae 5c 8a 15 74 05 00  b0 a2 b2 36 62 dc 4b 9f  |*.\..t.....6b.K.|
00000030  81 90 00 eb 29 ad e8 1e  00 00 00 00 01 00 00 00  |....)...........|
00000040  00 00 00 00 01 00 00 00  d8 5c 0a 00 00 00 00 00  |.........\......|
00000050  00 ff ff ff 00 00 00 00  00 17 80 b6 c1 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

Version-Release number of selected component (if applicable):
vdo-6.1.1.120-3.el7.x86_64 

How reproducible:
100%

Steps to Reproduce:
1. pvcreate /dev/sda
2. vdo create --name vdo --device /dev/sda

Actual results:
VDO is not created and existing PV is overwritten

Expected results:
VDO is created

Additional info:
Comment 2 Jakub Krysl 2018-08-23 08:10:22 UTC 
Lvm handles pvcreate by running pvcreate on existing PV creates the PV again. Because of this behaviour 'pvcreate -qq --test' passes and the whole creation continues. But changing dumpconfig to /dev/disk/by-id' points to nonexistent device now, in 7.5 this was not an issue:

# pvcreate /dev/sda
   Physical volume "/dev/sda" successfully created.

# hexdump -C /dev/sda -n 4k
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00000210  82 fc 41 16 20 00 00 00  4c 56 4d 32 20 30 30 31  |..A. ...LVM2 001|
00000220  43 4c 62 57 49 71 52 30  66 53 73 39 6b 63 59 39  |CLbWIqR0fSs9kcY9|
00000230  78 30 6f 6c 58 37 69 6c  63 5a 49 56 4c 65 77 43  |x0olX7ilcZIVLewC|
00000240  00 00 00 80 02 00 00 00  00 00 10 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 0f 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  02 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

# vdo create --name vdo --device /dev/sda --verbose
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate -qq --test /dev/sda
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 /dev/sda
    vdodumpconfig /dev/sda
Starting VDO vdo
    dmsetup status vdo
    grep MemAvailable /proc/meminfo
    modprobe kvdo
    vdodumpconfig /dev/sda
    dmsetup create vdo --uuid VDO-ef88f279-f4ec-4dc5-bcc0-32674bae9bbc --table '0 12556528 vdo /dev/sda 4096 disabled 0 32768 16380 on auto vdo ack=1,bio=4,bioRotationInterval=64,cpu=2,hash=1,logical=1,physical=1'
    dmsetup status vdo
Starting compression on VDO vdo
    dmsetup message vdo 0 compression on
    dmsetup status vdo
VDO instance 1 volume is ready at /dev/mapper/vdo

# hexdump -C /dev/sda -n 4k
00000000  64 6d 76 64 6f 30 30 31  05 00 00 00 04 00 00 00  |dmvdo001........|
00000010  00 00 00 00 5d 00 00 00  00 00 00 00 09 01 02 00  |....]...........|
00000020  f1 44 60 b5 15 74 05 00  ef 88 f2 79 f4 ec 4d c5  |.D`..t.....y..M.|
00000030  bc c0 32 67 4b ae 9b bc  00 00 00 00 01 00 00 00  |..2gK...........|
00000040  00 00 00 00 01 00 00 00  d8 5c 0a 00 00 00 00 00  |.........\......|
00000050  00 ff ff ff 00 00 00 00  00 32 cf bc 99 00 00 00  |.........2......|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000 


Setting to regression because of this.
Comment 6 Jakub Krysl 2019-04-29 15:27:30 UTC 
kernel-3.10.0-1040.el7.x86_64
vdo-6.1.2.41-4.el7.x86_64
kmod-kvdo-6.1.2.41-5.el7.x86_64


# pvcreate /dev/sda
  Physical volume "/dev/sda" successfully created.

# hexdump -C /dev/sda -n 4k
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00000210  f0 5d b8 e6 20 00 00 00  4c 56 4d 32 20 30 30 31  |.].. ...LVM2 001|
00000220  4d 49 6b 39 50 53 6a 61  76 4e 47 34 62 57 55 5a  |MIk9PSjavNG4bWUZ|
00000230  6c 39 73 56 62 30 4a 73  6b 5a 5a 76 63 77 56 54  |l9sVb0JskZZvcwVT|
00000240  00 00 00 80 18 09 00 00  00 00 10 00 00 00 00 00  |................|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 0f 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 00 00 00 00  02 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000

# vdo create --name vdo --device /dev/sda --verbose
    mkdir -p /run/lock/vdo
    touch /run/lock/vdo/_etc_vdoconf.yml.lock
    chmod 644 /run/lock/vdo/_etc_vdoconf.yml.lock
    touch /run/lock/vdo-config-singletons
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate --config devices/scan_lvs=1 -qq --test /dev/sda
    blkid -p /dev/sda
vdo: ERROR - device is a physical volume; use --force to override

# vdo create --name vdo --device /dev/sda --verbose
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    pvcreate --config devices/scan_lvs=1 -qq --test /dev/sda
    blkid -p /dev/sda
vdo: ERROR - device is a physical volume; use --force to override

# vdo create --name vdo --device /dev/sda --verbose --force
Creating VDO vdo
    grep MemAvailable /proc/meminfo
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --force /dev/disk/by-id/lvm-pv-uuid-MIk9PS-javN-G4bW-UZl9-sVb0-JskZ-ZvcwVT
vdo r    vdodumpconfig /dev/disk/by-id/lvm-pv-uuid-MIk9PS-javN-G4bW-UZl9-sVb0-JskZ-ZvcwVT
Removing VDO vdo
vdo: ERROR - vdodumpconfig: Failed to make FileLayer from '/dev/disk/by-id/lvm-pv-uuid-MIk9PS-javN-G4bW-UZl9-sVb0-JskZ-ZvcwVT' with No such file or directory

# hexdump -C /dev/sda -n 4k
00000000  64 6d 76 64 6f 30 30 31  05 00 00 00 04 00 00 00  |dmvdo001........|
00000010  00 00 00 00 5d 00 00 00  00 00 00 00 09 01 02 00  |....]...........|
00000020  f5 b9 5f d0 ac 87 05 00  b2 c4 7b 12 6a 7b 44 af  |.._.......{.j{D.|
00000030  85 b5 cd 6e 19 3f 24 0b  00 00 00 00 01 00 00 00  |...n.?$.........|
00000040  00 00 00 00 01 00 00 00  d8 5c 0a 00 00 00 00 00  |.........\......|
00000050  00 ff ff ff 00 00 00 00  00 34 c9 4a 56 00 00 00  |.........4.JV...|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000


So the new issue is fixed, there is a warning now. But the warning states "use --force to overwrite", which does not work. It correctly overwrites the PV, but as it takes the PV device from /dev/disk/by-id, it is nonexistent at that point and fail with $SUBJECT.
Although user can run the command again and it passes, it should pass on 1st run without errors. Returning to be fixed.
Comment 7 Dennis Keefe 2019-05-01 15:34:16 UTC 
The original issue was resolved, which is to day that vdo create will no longer overwrite the lvm pv header.
Running vdo create with --force sounds and getting an error sounds like a new bug.  
I would vote to close this bug and open a new one, which can be fixed in the next release.
The workaround can be to run vdo create --force twice.  We could document this in the release notes.
Comment 8 Jakub Krysl 2019-05-02 07:20:22 UTC 
(In reply to Dennis Keefe from comment #7)
> The original issue was resolved, which is to day that vdo create will no
> longer overwrite the lvm pv header.
> Running vdo create with --force sounds and getting an error sounds like a
> new bug.  
> I would vote to close this bug and open a new one, which can be fixed in the
> next release.
> The workaround can be to run vdo create --force twice.  We could document
> this in the release notes.

I returned it because the new behaviour seems exactly the same as the original one reported. We did not have --force at that time, so that is the only difference. But it does not change the behaviour. The VDO fails to create and the PV is overwritten. Thanks to --force there is the ability to run the command again as a workaround. So both ways (fix it here / close this with verified and new BZ) are fine with me.
Comment 9 Andy Walsh 2019-05-05 22:18:10 UTC 
I agree with Dennis that the behavior is related to a different bug.

Bug 1: If there is an LVM PV signature, VDO Manager would go ahead and overwrite the device with its own signature.
Bug 2: The resolved block device path (/dev/disk/by-id/*) should always resolve to something that doesn't rely on the contents of that device (like an LVM PV signature).

The reason I have this opinion is because you can create a PV on a volume, then run vdo create explicitly against a more valid /dev/disk/by-id path (like scsi-wwn, for example) and not trigger this condition.   Here's an example when there's a better by-id path available:

# vdo create --name vdo0 --device /dev/sdb1 --verbose
    mkdir -p /run/lock/vdo
    touch /run/lock/vdo/_etc_vdoconf.yml.lock
    chmod 644 /run/lock/vdo/_etc_vdoconf.yml.lock
    touch /run/lock/vdo-config-singletons
Creating VDO vdo0
    grep MemAvailable /proc/meminfo
    pvcreate --config devices/scan_lvs=1 -qq --test /dev/sdb1
    blkid -p --no-part-details /dev/sdb1
vdo: ERROR - device is a physical volume; use --force to override

# vdo create --name vdo0 --device /dev/sdb1 --verbose --force
Creating VDO vdo0
    grep MemAvailable /proc/meminfo
    modprobe kvdo
    vdoformat --uds-checkpoint-frequency=0 --uds-memory-size=0.25 --force /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1
    vdodumpconfig /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1
Starting VDO vdo0
    dmsetup status --target vdo vdo0
    grep MemAvailable /proc/meminfo
    modprobe kvdo
    vdodumpconfig /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1
    vdodumpconfig /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1
    dmsetup create vdo0 --uuid VDO-bbc27fb2-bcfb-4d63-b4e7-d2acfce2b9de --table '0 481423736 vdo V2 /dev/disk/by-id/ata-ST9250610NS_9XE0QXRA-part1 61049344 4096 32768 16380 on auto vdo0 maxDiscard 1 ack 1 bio 4 bioRotationInterval 64 cpu 2 hash 1 logical 1 physical 1'
    dmsetup status --target vdo vdo0
Starting compression on VDO vdo0
    dmsetup message vdo0 0 compression on
    vdodmeventd -r vdo0
    dmsetup status --target vdo vdo0
    dmsetup status --target vdo vdo0
VDO instance 0 volume is ready at /dev/mapper/vdo0


Either way to look at this, I don't think this should be a blocker+ for 7.7, just like I didn't for 7.6 mentioned in comment#4.
Comment 11 Andy Walsh 2019-05-07 15:29:09 UTC 
Moving back to ON_QA, after discussing with Jakub and having BZ1707481 opened to track the other behavior issue.
Comment 12 Jakub Krysl 2019-05-07 15:33:15 UTC 
From description:
"When creating VDO over device with PV, VDO does not see the PV and overwrites it. Than it fails to dump config with $SUBJECT error."
Setting to VERIFIED as the 1st part of this issue is gone, VDO no longer overwrites PV by itself, VDO now requires --force from user. This fails on 1st try, which is target of BZ 1707481 to be fixed.
Comment 14 errata-xmlrpc 2019-08-06 13:08:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2233