RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2007018 - Can not add data device after init-cache is called
Summary: Can not add data device after init-cache is called
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: stratisd
Version: 9.1
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: 9.1
Assignee: Bryan Gurney
QA Contact: Filip Suba
Kristina Slaveykova
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-22 20:02 UTC by Dennis Keefe
Modified: 2022-11-15 11:26 UTC (History)
7 users (show)

Fixed In Version: stratisd-3.1.0-2.el9
Doc Type: Bug Fix
Doc Text:
.Adding a data device no longer triggers assertion failure Previously, when adding additional devices to the cache, Stratis did not use cache immediately after initialization. As a consequence, the `stratisd` service returned an assertion failure message whenever a user attempted to add additional data devices to a pool. With this fix, cache is now used immediately after initialization and no assertion failures occur.
Clone Of:
Environment:
Last Closed: 2022-11-15 10:24:49 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-97974 0 None None None 2021-09-22 20:04:41 UTC
Red Hat Product Errata RHBA-2022:8124 0 None None None 2022-11-15 10:24:59 UTC

Description Dennis Keefe 2021-09-22 20:02:51 UTC 
stratis pool init-cache operation does not complete necessary re-organization of devicemapper devices

Related to: https://github.com/stratis-storage/stratisd/issues/2789

Version of Stratis = 2.4.2

How reproducible = 100%

Steps to Reproduce:
yum -y install stratis-cli
systemctl enable stratisd --now
truncate -s 10g /root/stratis.trunc1
truncate -s 10g /root/stratis.trunc2
truncate -s 10g /root/stratis.trunc3
losetup  /dev/loop1 /root/stratis.trunc1
losetup  /dev/loop2 /root/stratis.trunc2
losetup  /dev/loop3 /root/stratis.trunc3
stratis pool create p1 /dev/loop1
stratis pool init-cache p1 /dev/loop2
stratis pool add-data p1 /dev/loop3
(cli will hang for about 120 seconds until it times out)

Output:
From CLI 

Execution failed:
stratis attempted communication with the daemon, stratisd, over the D-Bus, but stratisd did not respond in the allowed time.

From Logs

stratisd[6378]: thread 'stratis-wt-6' panicked at 'assertion failed: `(left == right)`
stratisd[6378]:   left: `Device { major: 252, minor: 7 }`,
stratisd[6378]:  right: `Device { major: 252, minor: 0 }`', src/engine/strat_engine/thinpool/thinpool.rs:516:9
stratisd[6378]: stack backtrace:
stratisd[6378]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


Expected results:
Pool can be extended after init-cache operation.

Workaround:
For new pools that did not experience a failure to grow. 
1. Expand the cache with a second device before expanding the pool
stratis pool create p1 /dev/loop1
stratis pool init-cache p1 /dev/loop2
stratis pool add-cache p1 /dev/loop3
stratis pool add-data p1 /dev/loop4

2. Pool that failed to grow.
Clear the device that failed to be added to the pool 
(warning: make sure that you are clearing the correct device)
wipefs -a /dev/loop3
Reboot
Add new data device: 
stratis pool add-data p1 /dev/loop3
Comment 1 mulhern 2021-09-22 21:47:47 UTC 
Upstream issue: https://github.com/stratis-storage/stratisd/issues/2789
Comment 2 Dennis Keefe 2021-09-22 21:57:50 UTC 
Output from testing GitHub PR: https://github.com/stratis-storage/stratisd/pull/2790

[root@vm]# stratis --version
3.0.0
[root@vm]# stratis pool create p1 /dev/loop1
[root@vm]# lsblk
NAME                                                                    MAJ:MIN RM  SIZE RO TYPE    MOUNTPOINT
loop1                                                                     7:1    0   10G  0 loop    
└─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub 253:0    0   10G  0 stratis 
  ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta    253:1    0   16M  0 stratis 
  │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool  253:4    0  9.9G  0 stratis 
  ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata    253:2    0  9.9G  0 stratis 
  │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool  253:4    0  9.9G  0 stratis 
  └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv         253:3    0   16M  0 stratis 
loop2                                                                     7:2    0   10G  0 loop    
loop3                                                                     7:3    0   10G  0 loop    
loop4                                                                     7:4    0   10G  0 loop    
sr0                                                                      11:0    1  470K  0 rom     
zram0                                                                   251:0    0  1.8G  0 disk    [SWAP]
vda                                                                     252:0    0   20G  0 disk    
└─vda1                                                                  252:1    0   20G  0 part    /
[root@vm]# stratis pool init-cache p1 /dev/loop2
[root@vm]# lsblk
NAME                                                                     MAJ:MIN RM  SIZE RO TYPE    MOUNTPOINT
loop1                                                                      7:1    0   10G  0 loop    
└─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub  253:0    0   10G  0 stratis 
  └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache    253:7    0   10G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta   253:1    0   16M  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0  9.9G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata   253:2    0  9.9G  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0  9.9G  0 stratis 
    └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv        253:3    0   16M  0 stratis 
loop2                                                                      7:2    0   10G  0 loop    
├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-metasub    253:5    0  512M  0 stratis 
│ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache    253:7    0   10G  0 stratis 
│   ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta   253:1    0   16M  0 stratis 
│   │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0  9.9G  0 stratis 
│   ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata   253:2    0  9.9G  0 stratis 
│   │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0  9.9G  0 stratis 
│   └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv        253:3    0   16M  0 stratis 
└─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cachesub   253:6    0  9.5G  0 stratis 
  └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache    253:7    0   10G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta   253:1    0   16M  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0  9.9G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata   253:2    0  9.9G  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0  9.9G  0 stratis 
    └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv        253:3    0   16M  0 stratis 
loop3                                                                      7:3    0   10G  0 loop    
loop4                                                                      7:4    0   10G  0 loop    
sr0                                                                       11:0    1  470K  0 rom     
zram0                                                                    251:0    0  1.8G  0 disk    [SWAP]
vda                                                                      252:0    0   20G  0 disk    
└─vda1                                                                   252:1    0   20G  0 part    /
[root@vm]# lsblk --merge
    NAME                                                                    MAJ:MIN RM  SIZE RO TYPE    MOUNTPOINT
    loop1                                                                     7:1    0   10G  0 loop    
┌┈▶ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub 253:0    0   10G  0 stratis 
┆   loop2                                                                     7:2    0   10G  0 loop    
├┈▶ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-metasub   253:5    0  512M  0 stratis 
└┬▶ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cachesub  253:6    0  9.5G  0 stratis 
 └┈┈stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache       253:7    0   10G  0 stratis 
┌┈▶ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta      253:1    0   16M  0 stratis 
└┬▶ ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata      253:2    0  9.9G  0 stratis 
 ┆  └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv           253:3    0   16M  0 stratis 
 └┈┈stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool        253:4    0  9.9G  0 stratis 
    loop3                                                                     7:3    0   10G  0 loop    
    loop4                                                                     7:4    0   10G  0 loop    
    sr0                                                                      11:0    1  470K  0 rom     
    zram0                                                                   251:0    0  1.8G  0 disk    [SWAP]
    vda                                                                     252:0    0   20G  0 disk    
    └─vda1                                                                  252:1    0   20G  0 part    /
[root@vm]# stratis pool add-data p1 /dev/loop3
[root@vm]# lsblk
NAME                                                                     MAJ:MIN RM  SIZE RO TYPE    MOUNTPOINT
loop1                                                                      7:1    0   10G  0 loop    
└─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub  253:0    0   20G  0 stratis 
  └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache    253:7    0   20G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta   253:1    0   16M  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0 19.9G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata   253:2    0 19.9G  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0 19.9G  0 stratis 
    └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv        253:3    0   16M  0 stratis 
loop2                                                                      7:2    0   10G  0 loop    
├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-metasub    253:5    0  512M  0 stratis 
│ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache    253:7    0   20G  0 stratis 
│   ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta   253:1    0   16M  0 stratis 
│   │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0 19.9G  0 stratis 
│   ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata   253:2    0 19.9G  0 stratis 
│   │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0 19.9G  0 stratis 
│   └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv        253:3    0   16M  0 stratis 
└─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cachesub   253:6    0  9.5G  0 stratis 
  └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache    253:7    0   20G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta   253:1    0   16M  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0 19.9G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata   253:2    0 19.9G  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0 19.9G  0 stratis 
    └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv        253:3    0   16M  0 stratis 
loop3                                                                      7:3    0   10G  0 loop    
└─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-originsub  253:0    0   20G  0 stratis 
  └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-physical-cache    253:7    0   20G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thinmeta   253:1    0   16M  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0 19.9G  0 stratis 
    ├─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-thindata   253:2    0 19.9G  0 stratis 
    │ └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-thinpool-pool 253:4    0 19.9G  0 stratis 
    └─stratis-1-private-98e808a4270147a3b589f52c2c5570ec-flex-mdv        253:3    0   16M  0 stratis 
loop4                                                                      7:4    0   10G  0 loop    
sr0                                                                       11:0    1  470K  0 rom     
zram0                                                                    251:0    0  1.8G  0 disk    [SWAP]
vda                                                                      252:0    0   20G  0 disk    
└─vda1                                                                   252:1    0   20G  0 part    /
[root@vm]# stratis pool
Name                   Total Physical   Properties                                   UUID   Alerts
p1     20 GiB / 41.64 MiB / 19.96 GiB       Ca,~Cr   98e808a4-2701-47a3-b589-f52c2c5570ec         
[root@vm]# stratis blockdev
Pool Name   Device Node   Physical Size    Tier
p1          /dev/loop1           10 GiB    DATA
p1          /dev/loop2           10 GiB   CACHE
p1          /dev/loop3           10 GiB    DATA
Comment 3 Dennis Keefe 2021-10-14 16:16:24 UTC 
Fix merged in: https://github.com/stratis-storage/stratisd/pull/2790
Comment 4 mulhern 2021-11-17 00:40:33 UTC 
Further notes on the original problem:

The problem was discovered when the cache was initialized,
and a user then issued a command to add an additional data device to
the pool, resulting in the following assertion failure:

stratisd[6378]: thread 'stratis-wt-6' panicked at 'assertion failed: `(left == right)`
stratisd[6378]:   left: `Device { major: 252, minor: 7 }`,
stratisd[6378]:  right: `Device { major: 252, minor: 0 }`', src/engine/strat_engine/thinpool/thinpool.rs:516:9

The assertion that failed was:

        assert_eq!(
            backstore.device().expect(
                "thinpool exists and has been allocated to, so backstore must have a cap device"
            ),
            self.backstore_device
        );

This assertion checks that the device that the thinpool allocates its component devices from is
the same as the one that the backstore considers to be its cap device, the one to be allocated from.

The assertion itself was quite correct, and effective in identifying the bug.

The assertion failed because the backstore's device was the cache device, but the thinpool
was allocating from the cap device it was using before the cache was initialized and bypassing
the cache. There is no risk of data corruption in this case; the problem is that the cache is
unused and so no performance benefit is gained.

In the example above, where the operation is correct, before the cache is initialized the thinpool
devices, 253:1 and 253:2, are allocated from the cap device, 253:0. After the cache is initialized,
the thinpool devices are allocated from the cache device, 253:7, instead. This is the correct
configuration.

Previously, 253:1 and 253:2 would continue to allocate from 253:0 even though the cache
device, 253:7, was constructed properly.

Because the pool metadata had been written properly, any action that destroyed and rebuilt
the device stack, would cause the thinpool devices to be set up properly to allocate from
the cache device. So, a reboot would certainly cause the device stack to be reconstructed
correctly.

Because of the particular nature of the code defect that caused the bug, adding an additional device to
the cache would cause the thinpool devices to be allocated properly from the cache device.
Comment 9 Filip Suba 2022-06-20 12:43:52 UTC 
Verified with stratisd-3.1.0-2.el9.
# stratis pool create pool1 /dev/sda
# stratis pool init-cache pool1 /dev/sdb
# stratis pool add-data pool1 /dev/sdc
# stratis pool
Name                Total / Used / Free    Properties                                   UUID   Alerts
pool1   60 GiB / 522.66 MiB / 59.49 GiB    Ca,~Cr, Op   877bbaf9-cf33-43ae-a4ed-0b4b37274562
Comment 14 errata-xmlrpc 2022-11-15 10:24:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (stratisd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8124
Note You need to log in before you can comment on or make changes to this bug.