1707461 – stratisd tries to extremely grow thin metadata device on large disk

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1707461 - stratisd tries to extremely grow thin metadata device on large disk

Summary: stratisd tries to extremely grow thin metadata device on large disk

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	stratisd
Sub Component:
Version:	8.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	8.5
Assignee:	mulhern
QA Contact:	Filip Suba
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-05-07 15:07 UTC by Jakub Krysl
Modified:	2023-09-07 20:00 UTC (History)
CC List:	3 users (show)
Fixed In Version:	stratisd-2.4.0-1.el8
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-11-09 18:16:03 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2021:4217	0	None	None	None	2021-11-09 18:16:12 UTC

Description Jakub Krysl 2019-05-07 15:07:36 UTC

Description of problem:
When creating stratis pool over very large device (27.3T RAID5 in my case - 4 9.1T 4K block disks) syslog is spammed with stratisd trying to extend metadata device to size that dm-thin does not allow. Plus the requested size is so big, thinpool runs out of space and cannot further extend it.
The metadata end up being 28GB big:
# lsblk
NAME                                                                      MAJ:MIN RM   SIZE RO TYPE    MOUNTPOINT
sdb                                                                         8:16   0   9.1T  0 disk
└─md127                                                                     9:127  0  27.3T  0 raid5
  └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-physical-originsub 253:0    0  27.3T  0 stratis
    ├─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-thinmeta    253:1    0    28G  0 stratis
    │ └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-thinpool-pool  253:4    0  27.3T  0 stratis
    ├─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-thindata    253:2    0  27.3T  0 stratis
    │ └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-thinpool-pool  253:4    0  27.3T  0 stratis
    └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-mdv         253:3    0    16M  0 stratis
sdc                                                                         8:32   0   9.1T  0 disk
└─md127                                                                     9:127  0  27.3T  0 raid5
  └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-physical-originsub 253:0    0  27.3T  0 stratis
    ├─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-thinmeta    253:1    0    28G  0 stratis
    │ └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-thinpool-pool  253:4    0  27.3T  0 stratis
    ├─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-thindata    253:2    0  27.3T  0 stratis
    │ └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-thinpool-pool  253:4    0  27.3T  0 stratis
    └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-mdv         253:3    0    16M  0 stratis
sdd                                                                         8:48   0   9.1T  0 disk
└─md127                                                                     9:127  0  27.3T  0 raid5
  └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-physical-originsub 253:0    0  27.3T  0 stratis
    ├─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-thinmeta    253:1    0    28G  0 stratis
    │ └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-thinpool-pool  253:4    0  27.3T  0 stratis
    ├─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-thindata    253:2    0  27.3T  0 stratis
    │ └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-thinpool-pool  253:4    0  27.3T  0 stratis
    └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-mdv         253:3    0    16M  0 stratis
sde                                                                         8:64   0   9.1T  0 disk
└─md127                                                                     9:127  0  27.3T  0 raid5
  └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-physical-originsub 253:0    0  27.3T  0 stratis
    ├─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-thinmeta    253:1    0    28G  0 stratis
    │ └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-thinpool-pool  253:4    0  27.3T  0 stratis
    ├─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-thindata    253:2    0  27.3T  0 stratis
    │ └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-thinpool-pool  253:4    0  27.3T  0 stratis
    └─stratis-1-private-8449f2f9275b4e9db5a98e41aaa332aa-flex-mdv         253:3    0    16M  0 stratis

syslog:
May  7 16:52:43 storageqe-90 kernel: XFS (dm-3): Mounting V5 Filesystem
May  7 16:52:43 storageqe-90 kernel: XFS (dm-3): Ending clean mount
May  7 16:52:44 storageqe-90 kernel: XFS (dm-3): Unmounting Filesystem
May  7 16:52:46 storageqe-90 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
May  7 16:52:46 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Attempting to extend thinpool metadata sub-device belonging to pool 8449f2f9275b4e9db5a98e41aaa332aa by 58565048 sectors
May  7 16:52:46 storageqe-90 kernel: device-mapper: thin: Metadata device dm-1 is larger than 33292800 sectors: excess space will not be used.
May  7 16:52:46 storageqe-90 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
May  7 16:52:46 storageqe-90 kernel: device-mapper: thin: Metadata device dm-1 is larger than 33292800 sectors: excess space will not be used.
May  7 16:52:46 storageqe-90 kernel: device-mapper: thin: 253:4: growing the metadata device from 4096 to 4161600 blocks
May  7 16:52:47 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Extended thinpool metadata sub-device belonging to pool with uuid 8449f2f9275b4e9db5a98e41aaa332aa by 58556416 sectors
May  7 16:52:47 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Attempting to extend thinpool data sub-device belonging to pool 8449f2f9275b4e9db5a98e41aaa332aa by 58537592832 sectors
May  7 16:52:47 storageqe-90 kernel: device-mapper: thin: Metadata device dm-1 is larger than 33292800 sectors: excess space will not be used.
May  7 16:52:47 storageqe-90 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
May  7 16:52:47 storageqe-90 kernel: device-mapper: thin: 253:4: growing the data device from 768 to 28583577 blocks
May  7 16:52:48 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Extended thinpool data sub-device belonging to pool with uuid 8449f2f9275b4e9db5a98e41aaa332aa by 58537592832 sectors
May  7 16:52:48 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Data tier percent used: 0
May  7 16:52:48 storageqe-90 kernel: device-mapper: thin: Metadata device dm-1 is larger than 33292800 sectors: excess space will not be used.
May  7 16:52:48 storageqe-90 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
May  7 16:52:49 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Attempting to extend thinpool metadata sub-device belonging to pool 8449f2f9275b4e9db5a98e41aaa332aa by 25305016 sectors
May  7 16:52:49 storageqe-90 stratisd[75817]: WARN libstratis::engine::strat_engine::thinpool::thinpool: Insufficient free space available in backstore; extended thinpool metadata sub-device belonging to pool with uuid 8449f2f9275b4e9db5a98e41aaa332aa by 0 sectors, request was 25305016 sectors
May  7 16:52:49 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Data tier percent used: 0
May  7 16:52:49 storageqe-90 kernel: device-mapper: thin: Metadata device dm-1 is larger than 33292800 sectors: excess space will not be used.
May  7 16:52:49 storageqe-90 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
May  7 16:52:52 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Attempting to extend thinpool metadata sub-device belonging to pool 8449f2f9275b4e9db5a98e41aaa332aa by 25305016 sectors
May  7 16:52:52 storageqe-90 stratisd[75817]: WARN libstratis::engine::strat_engine::thinpool::thinpool: Insufficient free space available in backstore; extended thinpool metadata sub-device belonging to pool with uuid 8449f2f9275b4e9db5a98e41aaa332aa by 0 sectors, request was 25305016 sectors
May  7 16:52:52 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Data tier percent used: 0
May  7 16:52:52 storageqe-90 kernel: device-mapper: thin: Metadata device dm-1 is larger than 33292800 sectors: excess space will not be used.
May  7 16:52:52 storageqe-90 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
May  7 16:52:53 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Attempting to extend thinpool metadata sub-device belonging to pool 8449f2f9275b4e9db5a98e41aaa332aa by 25305016 sectors
May  7 16:52:53 storageqe-90 stratisd[75817]: WARN libstratis::engine::strat_engine::thinpool::thinpool: Insufficient free space available in backstore; extended thinpool metadata sub-device belonging to pool with uuid 8449f2f9275b4e9db5a98e41aaa332aa by 0 sectors, request was 25305016 sectors
May  7 16:52:53 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Data tier percent used: 0
May  7 16:52:53 storageqe-90 kernel: device-mapper: thin: Metadata device dm-1 is larger than 33292800 sectors: excess space will not be used.
May  7 16:52:53 storageqe-90 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
May  7 16:52:54 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Attempting to extend thinpool metadata sub-device belonging to pool 8449f2f9275b4e9db5a98e41aaa332aa by 25305016 sectors
May  7 16:52:54 storageqe-90 stratisd[75817]: WARN libstratis::engine::strat_engine::thinpool::thinpool: Insufficient free space available in backstore; extended thinpool metadata sub-device belonging to pool with uuid 8449f2f9275b4e9db5a98e41aaa332aa by 0 sectors, request was 25305016 sectors
May  7 16:52:54 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Data tier percent used: 0
May  7 16:52:54 storageqe-90 kernel: device-mapper: thin: Metadata device dm-1 is larger than 33292800 sectors: excess space will not be used.
May  7 16:52:54 storageqe-90 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.
May  7 16:52:55 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Attempting to extend thinpool metadata sub-device belonging to pool 8449f2f9275b4e9db5a98e41aaa332aa by 25305016 sectors
May  7 16:52:55 storageqe-90 stratisd[75817]: WARN libstratis::engine::strat_engine::thinpool::thinpool: Insufficient free space available in backstore; extended thinpool metadata sub-device belonging to pool with uuid 8449f2f9275b4e9db5a98e41aaa332aa by 0 sectors, request was 25305016 sectors
May  7 16:52:55 storageqe-90 stratisd[75817]: INFO libstratis::engine::strat_engine::thinpool::thinpool: Data tier percent used: 0
May  7 16:52:55 storageqe-90 kernel: device-mapper: thin: Metadata device dm-1 is larger than 33292800 sectors: excess space will not be used.
May  7 16:52:55 storageqe-90 kernel: device-mapper: thin: Data device (dm-2) discard unsupported: Disabling discard passdown.

Version-Release number of selected component (if applicable):
kernel-4.18.0-80.el8.x86_64
stratisd-1.0.3-1.el8.x86_64
stratis-cli-1.0.2-1.el8.noarch

How reproducible:
100%

Steps to Reproduce:
1. mdadm --create --level 5 --raid-devices 4 md5 /dev/sd[bcde]
2. stratis pool create test_pool /dev/md127

Actual results:
stratis attemps and fail to create too large metadata

Expected results:
stratis creates reasonably large metadata without failing

Additional info:
default LVM thin in the same situation:
# lvs
  LV        VG Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  test_pool vg twi-a-tz-- <27.29t             0.00   10.41
# lsblk
NAME                   MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                      8:0    0   9.1T  0 disk
sdb                      8:16   0   9.1T  0 disk
└─md127                  9:127  0  27.3T  0 raid5
  ├─vg-test_pool_tmeta 253:0    0   112M  0 lvm
  │ └─vg-test_pool     253:2    0  27.3T  0 lvm
  └─vg-test_pool_tdata 253:1    0  27.3T  0 lvm
    └─vg-test_pool     253:2    0  27.3T  0 lvm
sdc                      8:32   0   9.1T  0 disk
└─md127                  9:127  0  27.3T  0 raid5
  ├─vg-test_pool_tmeta 253:0    0   112M  0 lvm
  │ └─vg-test_pool     253:2    0  27.3T  0 lvm
  └─vg-test_pool_tdata 253:1    0  27.3T  0 lvm
    └─vg-test_pool     253:2    0  27.3T  0 lvm
sdd                      8:48   0   9.1T  0 disk
└─md127                  9:127  0  27.3T  0 raid5
  ├─vg-test_pool_tmeta 253:0    0   112M  0 lvm
  │ └─vg-test_pool     253:2    0  27.3T  0 lvm
  └─vg-test_pool_tdata 253:1    0  27.3T  0 lvm
    └─vg-test_pool     253:2    0  27.3T  0 lvm
sde                      8:64   0   9.1T  0 disk
└─md127                  9:127  0  27.3T  0 raid5
  ├─vg-test_pool_tmeta 253:0    0   112M  0 lvm
  │ └─vg-test_pool     253:2    0  27.3T  0 lvm
  └─vg-test_pool_tdata 253:1    0  27.3T  0 lvm
    └─vg-test_pool     253:2    0  27.3T  0 lvm

Comment 1 mulhern 2019-05-07 20:08:55 UTC

Some notes:

In drivers/md/dm-thin-metadata.h:

/*
 * The metadata device is currently limited in size.
 */
#define THIN_METADATA_MAX_SECTORS DM_SM_METADATA_MAX_SECTORS

Most relevant commit:

commit 7d48935eff401bb7970e73e822871a10e3643df1
Author: Mike Snitzer <snitzer>
Date:   Wed Feb 12 23:58:15 2014 -0500

    dm thin: allow metadata space larger than supported to go unused
    
    It was always intended that a user could provide a thin metadata device
    that is larger than the max supported by the on-disk format.  The extra
    space would just go unused.
    
    Unfortunately that never worked.  If the user attempted to use a larger
    metadata device on creation they would get an error like the following:
    
     device-mapper: space map common: space map too large
     device-mapper: transaction manager: couldn't create metadata space map
     device-mapper: thin metadata: tm_create_with_sm failed
     device-mapper: table: 252:17: thin-pool: Error creating metadata object
     device-mapper: ioctl: error adding target to table
    
    Fix this by allowing the initial metadata space map creation to cap its
    size at the max number of blocks supported (DM_SM_METADATA_MAX_BLOCKS).
    get_metadata_dev_size() must also impose DM_SM_METADATA_MAX_BLOCKS (via
    THIN_METADATA_MAX_SECTORS), otherwise extending metadata would cap at
    THIN_METADATA_MAX_SECTORS_WARNING (which is larger than supported).
    
    Also, the calculation for THIN_METADATA_MAX_SECTORS didn't account for
    the sizeof the disk_bitmap_header.  So the supported maximum metadata
    size is a bit smaller (reduced from 33423360 to 33292800 sectors).
    
    Lastly, remove the "excess space will not be used" warning message from
    get_metadata_dev_size(); it resulted in printing the warning multiple
    times.  Factor out warn_if_metadata_device_too_big(), call it from
    pool_ctr() and maybe_resize_metadata_dev().


It looks like the value has been stable since 2014.

Generally speaking, we don't want to restrict our code based on a max
value that may change in our dependency. Ideally, our dependency would
publish that value in some programmatic way. But likely this one will
not change, so we could incorporate the knowledge that this is the DM max into
our source, if we have to.

Comment 2 Jakub Krysl 2019-10-02 11:32:58 UTC

Mass migration to Guangwu.

Comment 3 mulhern 2020-01-10 16:18:10 UTC

Upstream issue: https://github.com/stratis-storage/stratisd/issues/1495.

Comment 7 mulhern 2021-02-01 17:20:42 UTC

Fix merged upstream.

Comment 10 Filip Suba 2021-05-27 14:44:21 UTC

Verified with stratisd-2.4.0-3.el8.

May 27 16:33:31 storageqe-90 kernel: XFS (dm-113): Mounting V5 Filesystem
May 27 16:33:31 storageqe-90 kernel: XFS (dm-113): Ending clean mount
May 27 16:33:31 storageqe-90 systemd[1]: run-stratisd-.mdv\x2d66d8df924c754eda8f7cf712c55fd601.mount: Succeeded.
May 27 16:33:31 storageqe-90 systemd[53459]: run-stratisd-.mdv\x2d66d8df924c754eda8f7cf712c55fd601.mount: Succeeded.
May 27 16:33:31 storageqe-90 kernel: XFS (dm-113): Unmounting Filesystem
May 27 16:33:31 storageqe-90 stratisd[38584]: [2021-05-27T14:33:31Z INFO  libstratis::engine::strat_engine::thinpool::thinpool] Attempting to extend thinpool metadata sub-device belonging to pool 66d8df92-4c75-4eda-8f7c-f712c55fd601 by 33260032 sectors
May 27 16:33:31 storageqe-90 kernel: device-mapper: thin: 253:114: growing the metadata device from 4096 to 4161536 blocks
May 27 16:33:31 storageqe-90 stratisd[38584]: [2021-05-27T14:33:31Z INFO  libstratis::engine::strat_engine::thinpool::thinpool] Extended thinpool metadata sub-device belonging to pool with uuid 66d8df92-4c75-4eda-8f7c-f712c55fd601 by 33259520 sectors
May 27 16:33:31 storageqe-90 stratisd[38584]: [2021-05-27T14:33:31Z INFO  libstratis::engine::strat_engine::thinpool::thinpool] Attempting to extend thinpool data sub-device belonging to pool 66d8df92-4c75-4eda-8f7c-f712c55fd601 by 58562889728 sectors
May 27 16:33:32 storageqe-90 kernel: device-mapper: thin: 253:114: growing the data device from 768 to 28595929 blocks
May 27 16:33:32 storageqe-90 stratisd[38584]: [2021-05-27T14:33:32Z INFO  libstratis::engine::strat_engine::thinpool::thinpool] Extended thinpool data sub-device belonging to pool with uuid 66d8df92-4c75-4eda-8f7c-f712c55fd601 by 58562889728 sectors
May 27 16:33:32 storageqe-90 stratisd[38584]: [2021-05-27T14:33:32Z INFO  libstratis::engine::strat_engine::thinpool::thinpool] Status of thinpool device with "devicemapper name: stratis-1-private-66d8df924c754eda8f7cf712c55fd601-thinpool-pool, device number: 253:114, device node: /dev/dm-114" changed from "none" to "rw"
May 27 16:33:32 storageqe-90 stratisd[38584]: [2021-05-27T14:33:32Z INFO  libstratis::dbus_api::api::shared] Pool with UUID 66d8df92-4c75-4eda-8f7c-f712c55fd601 was created successfully
May 27 16:33:32 storageqe-90 stratisd[38584]: [2021-05-27T14:33:32Z DEBUG libstratis::engine::strat_engine::liminal::identify] Stratis block device with Stratis device description: Stratis pool UUID: "66d8df92-4c75-4eda-8f7c-f712c55fd601", Stratis device UUID: "37398669-953c-4dd8-94e4-11108b449dbf", device number: "9:127", devnode: "/dev/md127" identified
May 27 16:33:32 storageqe-90 stratisd[38584]: [2021-05-27T14:33:32Z DEBUG libstratis::engine::strat_engine::liminal::identify] Stratis block device with Stratis device description: Stratis pool UUID: "66d8df92-4c75-4eda-8f7c-f712c55fd601", Stratis device UUID: "37398669-953c-4dd8-94e4-11108b449dbf", device number: "9:127", devnode: "/dev/md127" identified

Comment 13 errata-xmlrpc 2021-11-09 18:16:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (stratis-cli bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4217

Note You need to log in before you can comment on or make changes to this bug.