Bug 1961291
| Summary: | Enable forward error correction (FEC) feature for dm-verity | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Ondrej Kozina <okozina> |
| Component: | cryptsetup | Assignee: | Ondrej Kozina <okozina> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | guazhang <guazhang> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | high | ||
| Version: | 9.0 | CC: | agk, guazhang, jbrassow, okozina, prajnoha |
| Target Milestone: | beta | Keywords: | FutureFeature, Triaged |
| Target Release: | 9.0 Beta | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | cryptsetup-2.3.6-1.el9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-12-07 21:35:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1990465 | ||
| Bug Blocks: | |||
|
Comment 2
guazhang@redhat.com
2021-05-18 00:13:50 UTC
Hi
veritysetup format --data-blocks '65536' --hash-offset '268435456' --data-block-size '4096' --fec-device '/dev/mapper/vg01-lv01' --fec-offset '272629760' /dev/mapper/vg01-lv01 /dev/mapper/vg01-lv01
VERITY header information for /dev/mapper/vg01-lv01
UUID: 9565647c-e119-4048-902b-b3550ce58886
Hash type: 1
Data blocks: 65536
Data block size: 4096
Hash block size: 4096
Hash algorithm: sha256
Salt: 5e33c22366f941a6503f91cd0879368995a6b78f9093e34c2093745c70ba08f2
Root hash: 6e928ee53db049d6c4ea324447518fe4e1c6fb42978a9e70be53adea430cb9df
[root@storageqe-69 ~]# dd if=/dev/mapper/vg01-lv01 bs=4096 count=65536 | sha256sum
65536+0 records in
65536+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 0.565716 s, 475 MB/s
d67df5ac6b6ad138883fc74b2bf3527de27137d821c514136caf104b146df6b5 -
[root@storageqe-69 ~]#
[root@storageqe-69 ~]# dd if=/dev/mapper/vg01-lv01 skip=65536 bs=4096 count=1024 | sha256sum
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.00953271 s, 440 MB/s
739ffd1fa2e61ef0f23b789c7e37ec00b1058af61b2383c755cd2d3d44a4e358 -
[root@storageqe-69 ~]#
[root@storageqe-69 ~]# dd if=/dev/mapper/verity_name | sha256sum
524288+0 records in
524288+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 0.74616 s, 360 MB/s
a6d72ac7690f53be6ae46ba88506bd97302a093f7108472bd9efc3cefda06484 -
[root@storageqe-69 ~]#
from the result, the data_digest != verity_digest and can not find the log like "device-mapper: verity-fec: 7:0: FEC 630784: corrected 1526 errors" in /var/log/messages
please have a look the test steps
INFO: [2021-06-21 21:26:46] Running: 'veritysetup format --data-blocks '65536' --hash-offset '268435456' --data-block-size '4096' --fec-device '/dev/mapper/vg01-lv01' --fec-offset '272629760' /dev/mapper/vg01-lv01 /dev/mapper/vg01-lv01'...
VERITY header information for /dev/mapper/vg01-lv01
UUID: 98878ced-f09b-41bc-8328-a80546764750
Hash type: 1
Data blocks: 65536
Data block size: 4096
Hash block size: 4096
Hash algorithm: sha256
Salt: e78bacfefdd810ae46721e6a45c184744752cb001c89dbfc89ede20bf8273f3b
Root hash: dbb65e8aeb29cf440ff25bf7b0b85f132830f5c171f6ea96a7ddd2e06182de68
{'UUID': '98878ced-f09b-41bc-8328-a80546764750', 'Hash_type': '1', 'Data_blocks': '65536', 'Data_block_size': '4096', 'Hash_block_size': '4096', 'Hash_algorithm': 'sha256', 'Salt': 'e78bacfefdd810ae46721e6a45c184744752cb001c89dbfc89ede20bf8273f3b', 'Root_hash': 'dbb65e8aeb29cf440ff25bf7b0b85f132830f5c171f6ea96a7ddd2e06182de68', 'data_disk': '/dev/mapper/vg01-lv01', 'hash_disk': '/dev/mapper/vg01-lv01', 'spce': (), 'data_blocks': 65536, 'hash_offset': '268435456', 'data_block_size': 4096, 'fec_device': '/dev/mapper/vg01-lv01', 'fec_offset': 272629760}
data_digest is a6d72ac7690f53be6ae46ba88506bd97302a093f7108472bd9efc3cefda06484
hash_digest is f470efbc06ab6e211799d6099911220e06e486e6815d2268f0a40ca66764f0fe
INFO: [2021-06-21 21:26:49] Running: 'veritysetup open --data-blocks '65536' --hash-offset '268435456' --data-block-size '4096' --fec-device '/dev/mapper/vg01-lv01' --fec-offset '272629760' /dev/mapper/vg01-lv01 verity_name /dev/mapper/vg01-lv01 dbb65e8aeb29cf440ff25bf7b0b85f132830f5c171f6ea96a7ddd2e06182de68'...
INFO: [2021-06-21 21:26:49] Running: 'lsblk'...
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 15G 0 loop
loop1 7:1 0 15G 0 loop
loop2 7:2 0 15G 0 loop
loop3 7:3 0 15G 0 loop
loop4 7:4 0 15G 0 loop
vg01-lv01 253:0 0 10G 0 lvm
verity_name 253:1 0 256M 1 crypt
loop5 7:5 0 15G 0 loop
sda 8:0 0 1.8T 0 disk
sda1 8:1 0 1G 0 part /boot
sda2 8:2 0 7.8G 0 part [SWAP]
sda3 8:3 0 1.8T 0 part /
sdb 8:16 0 1.8T 0 disk
sdc 8:32 0 1.8T 0 disk
sr0 11:0 1 1024M 0 rom
verity_digest is e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
FAIL: data_digest != verity_digest
INFO: [2021-06-21 21:26:49] Running: 'dd if=/dev/urandom of=/dev/mapper/vg01-lv01 seek=1233 count=3 bs=512 conv=notrunc'...
3+0 records in
3+0 records out
1536 bytes (1.5 kB, 1.5 KiB) copied, 0.00014392 s, 10.7 MB/s
INFO: corrupted_data_digest is fed7d608d0964f6d7335e8c2822befbbf270c710c7e3b62ab10a570828b33265
INFO: the corrupted_data_digest != data_digest
INFO: [2021-06-21 21:26:50] Running: 'veritysetup verify --data-blocks '65536' --hash-offset '268435456' --data-block-size '4096' --fec-device '/dev/mapper/vg01-lv01' --fec-offset '272629760' /dev/mapper/vg01-lv01 /dev/mapper/vg01-lv01 dbb65e8aeb29cf440ff25bf7b0b85f132830f5c171f6ea96a7ddd2e06182de68'...
Verification failed at position 630784.
Verification of data area failed.
Found 1531 repairable errors with FEC device.
INFO: [2021-06-21 21:26:52] Running: 'dd if=/dev/zero of=/dev/mapper/vg01-lv01 bs=512 count=10'...
10+0 records in
10+0 records out
5120 bytes (5.1 kB, 5.0 KiB) copied, 4.7528e-05 s, 108 MB/s
INFO: [2021-06-21 21:26:52] Running: 'veritysetup verify --data-blocks '65536' --hash-offset '268435456' --data-block-size '4096' --fec-device '/dev/mapper/vg01-lv01' --fec-offset '272629760' /dev/mapper/vg01-lv01 /dev/mapper/vg01-lv01 dbb65e8aeb29cf440ff25bf7b0b85f132830f5c171f6ea96a7ddd2e06182de68'...
Verification failed at position 630784.
Verification of data area failed.
Found 1531 repairable errors with FEC device.
INFO: [2021-06-21 21:26:53] Running: 'dd if=/dev/zero of=/dev/mapper/vg01-lv01 bs=512 count=1'...
1+0 records in
1+0 records out
512 bytes copied, 3.4755e-05 s, 14.7 MB/s
INFO: [2021-06-21 21:26:53] Running: 'veritysetup verify --data-blocks '65536' --hash-offset '268435456' --data-block-size '4096' --fec-device '/dev/mapper/vg01-lv01' --fec-offset '272629760' /dev/mapper/vg01-lv01 /dev/mapper/vg01-lv01 dbb65e8aeb29cf440ff25bf7b0b85f132830f5c171f6ea96a7ddd2e06182de68'...
Verification failed at position 630784.
Verification of data area failed.
Found 1531 repairable errors with FEC device.
INFO: [2021-06-21 21:26:55] Running: 'veritysetup status verity_name '...
/dev/mapper/verity_name is active.
type: VERITY
status: verified
hash type: 1
data block: 4096
hash block: 4096
hash name: sha256
salt: e78bacfefdd810ae46721e6a45c184744752cb001c89dbfc89ede20bf8273f3b
data device: /dev/mapper/vg01-lv01
size: 524288 sectors
mode: readonly
hash device: /dev/mapper/vg01-lv01
hash offset: 524296 sectors
FEC device: /dev/mapper/vg01-lv01
FEC offset: 532480 sectors
FEC roots: 2
root hash: dbb65e8aeb29cf440ff25bf7b0b85f132830f5c171f6ea96a7ddd2e06182de68
(In reply to guazhang from comment #7) > Hi > > veritysetup format --data-blocks '65536' --hash-offset '268435456' > --data-block-size '4096' --fec-device '/dev/mapper/vg01-lv01' --fec-offset > '272629760' /dev/mapper/vg01-lv01 /dev/mapper/vg01-lv01 > VERITY header information for /dev/mapper/vg01-lv01 > UUID: 9565647c-e119-4048-902b-b3550ce58886 > Hash type: 1 > Data blocks: 65536 > Data block size: 4096 > Hash block size: 4096 > Hash algorithm: sha256 > Salt: > 5e33c22366f941a6503f91cd0879368995a6b78f9093e34c2093745c70ba08f2 > Root hash: > 6e928ee53db049d6c4ea324447518fe4e1c6fb42978a9e70be53adea430cb9df > > [root@storageqe-69 ~]# dd if=/dev/mapper/vg01-lv01 bs=4096 count=65536 | > sha256sum > 65536+0 records in > 65536+0 records out > 268435456 bytes (268 MB, 256 MiB) copied, 0.565716 s, 475 MB/s > d67df5ac6b6ad138883fc74b2bf3527de27137d821c514136caf104b146df6b5 - > [root@storageqe-69 ~]# > [root@storageqe-69 ~]# dd if=/dev/mapper/vg01-lv01 skip=65536 bs=4096 > count=1024 | sha256sum > 1024+0 records in > 1024+0 records out > 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.00953271 s, 440 MB/s > 739ffd1fa2e61ef0f23b789c7e37ec00b1058af61b2383c755cd2d3d44a4e358 - > [root@storageqe-69 ~]# > [root@storageqe-69 ~]# dd if=/dev/mapper/verity_name | sha256sum > 524288+0 records in > 524288+0 records out > 268435456 bytes (268 MB, 256 MiB) copied, 0.74616 s, 360 MB/s > a6d72ac7690f53be6ae46ba88506bd97302a093f7108472bd9efc3cefda06484 - > [root@storageqe-69 ~]# > > > > from the result, the data_digest != verity_digest and can not find the log > like "device-mapper: verity-fec: 7:0: FEC 630784: corrected 1526 errors" in > /var/log/messages > please have a look the test steps Not sure what went wrong with digests (perhaps try example script below) but you will definitely not see any errors/corrections with FEC in kernel log because there's not 'corruption' in data/hash area yet. First corrupt the data area or hash area, read dm-verity later and you should see notifications about FEC being triggered. I've added 'direct' flag and run this script in loop. It worked as expected for me. #!/bin/bash set -xe # format device veritysetup format /dev/loop0 /dev/loop0 --fec-device /dev/loop0 --data-block-size 4096 --data-blocks 65536 --hash-offset 268435456 --fec-offset 272629760 | tee /tmp/format ROOT=$(grep -e "Root hash:" /tmp/format | cut -f 2) echo "root: $ROOT" dd if=/dev/loop0 bs=4096 count=65536 iflag=direct | sha256sum | cut -d " " -f 1 | tee /tmp/data_digest dd if=/dev/loop0 bs=4096 skip=65536 count=1024 iflag=direct | sha256sum | cut -d " " -f 1 | tee /tmp/hash_digest # open dm-verity device veritysetup open /dev/loop0 dmv1 /dev/loop0 $ROOT --fec-device /dev/loop0 --hash-offset 268435456 --fec-offset 272629760 dd if=/dev/mapper/dmv1 bs=4096 iflag=direct | sha256sum | cut -d " " -f 1 | tee /tmp/verity_digest # must match diff /tmp/data_digest /tmp/verity_digest # corrupt data area dd if=/dev/urandom of=/dev/loop0 bs=512 seek=1233 count=3 oflag=direct dd if=/dev/loop0 bs=4096 count=65536 iflag=direct | sha256sum | cut -d " " -f 1 | tee /tmp/corrupted_data_digest # must not match ! diff /tmp/data_digest /tmp/corrupted_data_digest # read dm-verity dd if=/dev/mapper/dmv1 bs=4096 iflag=direct | sha256sum | cut -d " " -f 1 | tee /tmp/verity_new_digest # must match diff /tmp/verity_new_digest /tmp/data_digest # corrupt hash area dd if=/dev/urandom of=/dev/loop0 bs=512 seek=524301 count=7 oflag=direct dd if=/dev/loop0 bs=4096 skip=65536 count=1024 iflag=direct | sha256sum | cut -d " " -f 1 | tee /tmp/corrupted_hash_digest # must not match ! diff /tmp/hash_digest /tmp/corrupted_hash_digest # read dm-verity dd if=/dev/mapper/dmv1 bs=4096 iflag=direct | sha256sum | cut -d " " -f 1 | tee /tmp/verity_new_digest # must match diff /tmp/verity_new_digest /tmp/data_digest veritysetup close dmv1 Thanks for the details testing. test pass with the script, move to verified. |