Bug 2007018
Summary: | Can not add data device after init-cache is called | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 9 | Reporter: | Dennis Keefe <dkeefe> |
Component: | stratisd | Assignee: | Bryan Gurney <bgurney> |
Status: | CLOSED ERRATA | QA Contact: | Filip Suba <fsuba> |
Severity: | unspecified | Docs Contact: | Kristina Slaveykova <kslaveyk> |
Priority: | medium | ||
Version: | 9.1 | CC: | amulhern, bgurney, cwei, dkeefe, fsuba, kslaveyk, mgandhi |
Target Milestone: | rc | Keywords: | Triaged |
Target Release: | 9.1 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | stratisd-3.1.0-2.el9 | Doc Type: | Bug Fix |
Doc Text: |
.Adding a data device no longer triggers assertion failure
Previously, when adding additional devices to the cache, Stratis did not use cache immediately after initialization. As a consequence, the `stratisd` service returned an assertion failure message whenever a user attempted to add additional data devices to a pool. With this fix, cache is now used immediately after initialization and no assertion failures occur.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2022-11-15 10:24:49 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Dennis Keefe
2021-09-22 20:02:51 UTC
Upstream issue: https://github.com/stratis-storage/stratisd/issues/2789 Output from testing GitHub PR: https://github.com/stratis-storage/stratisd/pull/2790 [root@vm]# stratis --version 3.0.0 [root@vm]# stratis pool create p1 /dev/loop1 [root@vm]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop1 7:1 0 10G 0 loop └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub 253:0 0 10G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta 253:1 0 16M 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 9.9G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata 253:2 0 9.9G 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 9.9G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv 253:3 0 16M 0 stratis loop2 7:2 0 10G 0 loop loop3 7:3 0 10G 0 loop loop4 7:4 0 10G 0 loop sr0 11:0 1 470K 0 rom zram0 251:0 0 1.8G 0 disk [SWAP] vda 252:0 0 20G 0 disk └─vda1 252:1 0 20G 0 part / [root@vm]# stratis pool init-cache p1 /dev/loop2 [root@vm]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop1 7:1 0 10G 0 loop └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub 253:0 0 10G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache 253:7 0 10G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta 253:1 0 16M 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 9.9G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata 253:2 0 9.9G 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 9.9G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv 253:3 0 16M 0 stratis loop2 7:2 0 10G 0 loop ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-metasub 253:5 0 512M 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache 253:7 0 10G 0 stratis │ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta 253:1 0 16M 0 stratis │ │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 9.9G 0 stratis │ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata 253:2 0 9.9G 0 stratis │ │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 9.9G 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv 253:3 0 16M 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cachesub 253:6 0 9.5G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache 253:7 0 10G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta 253:1 0 16M 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 9.9G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata 253:2 0 9.9G 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 9.9G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv 253:3 0 16M 0 stratis loop3 7:3 0 10G 0 loop loop4 7:4 0 10G 0 loop sr0 11:0 1 470K 0 rom zram0 251:0 0 1.8G 0 disk [SWAP] vda 252:0 0 20G 0 disk └─vda1 252:1 0 20G 0 part / [root@vm]# lsblk --merge NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop1 7:1 0 10G 0 loop ┌┈▶ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub 253:0 0 10G 0 stratis ┆ loop2 7:2 0 10G 0 loop ├┈▶ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-metasub 253:5 0 512M 0 stratis └┬▶ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cachesub 253:6 0 9.5G 0 stratis └┈┈stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache 253:7 0 10G 0 stratis ┌┈▶ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta 253:1 0 16M 0 stratis └┬▶ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata 253:2 0 9.9G 0 stratis ┆ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv 253:3 0 16M 0 stratis └┈┈stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 9.9G 0 stratis loop3 7:3 0 10G 0 loop loop4 7:4 0 10G 0 loop sr0 11:0 1 470K 0 rom zram0 251:0 0 1.8G 0 disk [SWAP] vda 252:0 0 20G 0 disk └─vda1 252:1 0 20G 0 part / [root@vm]# stratis pool add-data p1 /dev/loop3 [root@vm]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop1 7:1 0 10G 0 loop └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub 253:0 0 20G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache 253:7 0 20G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta 253:1 0 16M 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 19.9G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata 253:2 0 19.9G 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 19.9G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv 253:3 0 16M 0 stratis loop2 7:2 0 10G 0 loop ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-metasub 253:5 0 512M 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache 253:7 0 20G 0 stratis │ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta 253:1 0 16M 0 stratis │ │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 19.9G 0 stratis │ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata 253:2 0 19.9G 0 stratis │ │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 19.9G 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv 253:3 0 16M 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cachesub 253:6 0 9.5G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache 253:7 0 20G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta 253:1 0 16M 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 19.9G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata 253:2 0 19.9G 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 19.9G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv 253:3 0 16M 0 stratis loop3 7:3 0 10G 0 loop └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub 253:0 0 20G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache 253:7 0 20G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta 253:1 0 16M 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 19.9G 0 stratis ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata 253:2 0 19.9G 0 stratis │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4 0 19.9G 0 stratis └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv 253:3 0 16M 0 stratis loop4 7:4 0 10G 0 loop sr0 11:0 1 470K 0 rom zram0 251:0 0 1.8G 0 disk [SWAP] vda 252:0 0 20G 0 disk └─vda1 252:1 0 20G 0 part / [root@vm]# stratis pool Name Total Physical Properties UUID Alerts p1 20 GiB / 41.64 MiB / 19.96 GiB Ca,~Cr 98e808a4-2701-47a3-b589-f52c2c5570ec [root@vm]# stratis blockdev Pool Name Device Node Physical Size Tier p1 /dev/loop1 10 GiB DATA p1 /dev/loop2 10 GiB CACHE p1 /dev/loop3 10 GiB DATA Fix merged in: https://github.com/stratis-storage/stratisd/pull/2790 Further notes on the original problem: The problem was discovered when the cache was initialized, and a user then issued a command to add an additional data device to the pool, resulting in the following assertion failure: stratisd[6378]: thread 'stratis-wt-6' panicked at 'assertion failed: `(left == right)` stratisd[6378]: left: `Device { major: 252, minor: 7 }`, stratisd[6378]: right: `Device { major: 252, minor: 0 }`', src/engine/strat_engine/thinpool/thinpool.rs:516:9 The assertion that failed was: assert_eq!( backstore.device().expect( "thinpool exists and has been allocated to, so backstore must have a cap device" ), self.backstore_device ); This assertion checks that the device that the thinpool allocates its component devices from is the same as the one that the backstore considers to be its cap device, the one to be allocated from. The assertion itself was quite correct, and effective in identifying the bug. The assertion failed because the backstore's device was the cache device, but the thinpool was allocating from the cap device it was using before the cache was initialized and bypassing the cache. There is no risk of data corruption in this case; the problem is that the cache is unused and so no performance benefit is gained. In the example above, where the operation is correct, before the cache is initialized the thinpool devices, 253:1 and 253:2, are allocated from the cap device, 253:0. After the cache is initialized, the thinpool devices are allocated from the cache device, 253:7, instead. This is the correct configuration. Previously, 253:1 and 253:2 would continue to allocate from 253:0 even though the cache device, 253:7, was constructed properly. Because the pool metadata had been written properly, any action that destroyed and rebuilt the device stack, would cause the thinpool devices to be set up properly to allocate from the cache device. So, a reboot would certainly cause the device stack to be reconstructed correctly. Because of the particular nature of the code defect that caused the bug, adding an additional device to the cache would cause the thinpool devices to be allocated properly from the cache device. Verified with stratisd-3.1.0-2.el9. # stratis pool create pool1 /dev/sda # stratis pool init-cache pool1 /dev/sdb # stratis pool add-data pool1 /dev/sdc # stratis pool Name Total / Used / Free Properties UUID Alerts pool1 60 GiB / 522.66 MiB / 59.49 GiB Ca,~Cr, Op 877bbaf9-cf33-43ae-a4ed-0b4b37274562 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (stratisd bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8124 |