Bug 1696442
| Summary: | drastic increase in pool size causes issues with pool metadata: "device-mapper: space map metadata: unable to allocate new metadata block" | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Corey Marthaler <cmarthal> | ||||
| Component: | lvm2 | Assignee: | Zdenek Kabelac <zkabelac> | ||||
| lvm2 sub component: | Thin Provisioning | QA Contact: | cluster-qe <cluster-qe> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | medium | ||||||
| Priority: | high | CC: | agk, heinzm, jbrassow, mcsontos, msnitzer, pasik, prajnoha, thornber, zkabelac | ||||
| Version: | 8.0 | Flags: | pm-rhel:
mirror+
|
||||
| Target Milestone: | rc | ||||||
| Target Release: | 8.1 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | lvm2-2.03.11-0.2.20201103git8801a86.el8 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-05-18 15:01:41 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Corey Marthaler
2019-04-04 20:43:18 UTC
Same scenario restarted but with "small" increases works fine. [root@hayes-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Meta% Devices [lvol0_pmspare] thinpool_4_5830 ewi------- 4.00m /dev/sdl1(0) thinpool_4_58300 thinpool_4_5830 twi-aotz-- 1.00g 0.00 11.43 thinpool_4_58300_tdata(0) [thinpool_4_58300_tdata] thinpool_4_5830 Twi-ao---- 1.00g /dev/sdl1(1) [thinpool_4_58300_tmeta] thinpool_4_5830 ewi-ao---- 4.00m /dev/sdf1(0) virt1 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt2 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt3 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt4 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt5 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 [root@hayes-02 ~]# lvextend -L +1G /dev/thinpool_4_5830/thinpool_4_58300 WARNING: Sum of all thin volume sizes (2.44 GiB) exceeds the size of thin pools (2.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Size of logical volume thinpool_4_5830/thinpool_4_58300_tdata changed from 1.00 GiB (256 extents) to 2.00 GiB (512 extents). Logical volume thinpool_4_5830/thinpool_4_58300_tdata successfully resized. [root@hayes-02 ~]# lvextend -L +1G /dev/thinpool_4_5830/thinpool_4_58300 Size of logical volume thinpool_4_5830/thinpool_4_58300_tdata changed from 2.00 GiB (512 extents) to 3.00 GiB (768 extents). Logical volume thinpool_4_5830/thinpool_4_58300_tdata successfully resized. [root@hayes-02 ~]# lvextend -L +1G /dev/thinpool_4_5830/thinpool_4_58300 Size of logical volume thinpool_4_5830/thinpool_4_58300_tdata changed from 3.00 GiB (768 extents) to 4.00 GiB (1024 extents). Logical volume thinpool_4_5830/thinpool_4_58300_tdata successfully resized. [root@hayes-02 ~]# lvextend -L +1G /dev/thinpool_4_5830/thinpool_4_58300 Size of logical volume thinpool_4_5830/thinpool_4_58300_tdata changed from 4.00 GiB (1024 extents) to 5.00 GiB (1280 extents). Logical volume thinpool_4_5830/thinpool_4_58300_tdata successfully resized. [root@hayes-02 ~]# lvextend -L +1G /dev/thinpool_4_5830/thinpool_4_58300 Size of logical volume thinpool_4_5830/thinpool_4_58300_tdata changed from 5.00 GiB (1280 extents) to 6.00 GiB (1536 extents). Logical volume thinpool_4_5830/thinpool_4_58300_tdata successfully resized. Created attachment 1552099 [details]
verbose lvextend attempt
FWIW - this happens whether or not thin_pool_autoextend_threshold/thin_pool_autoextend_percent are turned on. patient: "doctor it hurts when I slam my head in a car door" ;) doctor: "maybe you should extend the thin-pool's metadata (from its undersized 4mb) before extending its data (from 1gb to 2tb)?" Note - upstream recently welcomed this commit: https://www.redhat.com/archives/lvm-devel/2019-April/msg00003.html Which essentially up-sizes metadata first to match data - though the patch will still need some tunning as it can be made better. Considering this solved for RH 8.3 as we upsize thin-pool metadata (_tmeta) to match thin-pool data (_tdata) and we do resize metadata first. Fix verified in the latest .d rpms. kernel-4.18.0-246.el8.dt2 BUILT: Mon Nov 9 07:22:41 CST 2020 lvm2-2.03.11-0.2.20201103git8801a86.el8 BUILT: Wed Nov 4 07:04:46 CST 2020 lvm2-libs-2.03.11-0.2.20201103git8801a86.el8 BUILT: Wed Nov 4 07:04:46 CST 2020 device-mapper-1.02.175-0.2.20201103git8801a86.el8 BUILT: Wed Nov 4 07:04:46 CST 2020 device-mapper-libs-1.02.175-0.2.20201103git8801a86.el8 BUILT: Wed Nov 4 07:04:46 CST 2020 device-mapper-event-1.02.175-0.2.20201103git8801a86.el8 BUILT: Wed Nov 4 07:04:46 CST 2020 device-mapper-event-libs-1.02.175-0.2.20201103git8801a86.el8 BUILT: Wed Nov 4 07:04:46 CST 2020 [root@hayes-02 ~]# lvcreate --activate y --thinpool thinpool_4_58300 -L 1G thinpool_4_5830 Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data. Logical volume "thinpool_4_58300" created. [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt1 Logical volume "virt1" created. [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt2 Logical volume "virt2" created. [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt3 WARNING: Sum of all thin volume sizes (1.46 GiB) exceeds the size of thin pool thinpool_4_5830/thinpool_4_58300 (1.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "virt3" created. [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt4 WARNING: Sum of all thin volume sizes (1.95 GiB) exceeds the size of thin pool thinpool_4_5830/thinpool_4_58300 (1.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "virt4" created. [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt5 WARNING: Sum of all thin volume sizes (2.44 GiB) exceeds the size of thin pool thinpool_4_5830/thinpool_4_58300 (1.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "virt5" created. [root@hayes-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices [lvol0_pmspare] thinpool_4_5830 ewi------- 4.00m /dev/sdl1(0) thinpool_4_58300 thinpool_4_5830 twi-aotz-- 1.00g 0.00 11.43 thinpool_4_58300_tdata(0) [thinpool_4_58300_tdata] thinpool_4_5830 Twi-ao---- 1.00g /dev/sdl1(1) [thinpool_4_58300_tmeta] thinpool_4_5830 ewi-ao---- 4.00m /dev/sdp1(0) virt1 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt2 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt3 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt4 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt5 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 [root@hayes-02 ~]# lvextend -l 528154 /dev/thinpool_4_5830/thinpool_4_58300 Rounding size to boundary between physical extents: <2.02 GiB. WARNING: Sum of all thin volume sizes (2.44 GiB) exceeds the size of thin pools (1.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Size of logical volume thinpool_4_5830/thinpool_4_58300_tmeta changed from 4.00 MiB (1 extents) to <2.02 GiB (516 extents). Size of logical volume thinpool_4_5830/thinpool_4_58300_tdata changed from 1.00 GiB (256 extents) to 2.01 TiB (528154 extents). Logical volume thinpool_4_5830/thinpool_4_58300_tdata successfully resized. Fix verified in the latest rpms. kernel-4.18.0-274.el8 BUILT: Thu Jan 14 01:49:32 CST 2021 lvm2-2.03.11-1.el8 BUILT: Fri Jan 8 05:21:07 CST 2021 lvm2-libs-2.03.11-1.el8 BUILT: Fri Jan 8 05:21:07 CST 2021 device-mapper-1.02.175-1.el8 BUILT: Fri Jan 8 05:21:07 CST 2021 device-mapper-libs-1.02.175-1.el8 BUILT: Fri Jan 8 05:21:07 CST 2021 device-mapper-event-1.02.175-1.el8 BUILT: Fri Jan 8 05:21:07 CST 2021 device-mapper-event-libs-1.02.175-1.el8 BUILT: Fri Jan 8 05:21:07 CST 2021 [root@hayes-02 ~]# vgcreate thinpool_4_5830 /dev/sdl1 /dev/sdn1 /dev/sdm1 /dev/sdo1 /dev/sdp1 /dev/sdj1 /dev/sde1 /dev/sdf1 WARNING: Devices have inconsistent physical block sizes (4096 and 512). WARNING: Devices have inconsistent physical block sizes (4096 and 512). WARNING: Devices have inconsistent physical block sizes (4096 and 512). Volume group "thinpool_4_5830" successfully created [root@hayes-02 ~]# vgs VG #PV #LV #SN Attr VSize VFree thinpool_4_5830 8 0 0 wz--n- <7.64t <7.64t [root@hayes-02 ~]# lvcreate --activate y --thinpool thinpool_4_58300 -L 1G thinpool_4_5830 Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data. Logical volume "thinpool_4_58300" created. [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt1 Logical volume "virt1" created. [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt2 Logical volume "virt2" created. [root@hayes-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices [lvol0_pmspare] thinpool_4_5830 ewi------- 4.00m /dev/sdl1(0) thinpool_4_58300 thinpool_4_5830 twi-aotz-- 1.00g 0.00 11.13 thinpool_4_58300_tdata(0) [thinpool_4_58300_tdata] thinpool_4_5830 Twi-ao---- 1.00g /dev/sdl1(1) [thinpool_4_58300_tmeta] thinpool_4_5830 ewi-ao---- 4.00m /dev/sdf1(0) virt1 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt2 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt3 WARNING: Sum of all thin volume sizes (1.46 GiB) exceeds the size of thin pool thinpool_4_5830/thinpool_4_58300 (1.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "virt3" created. [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt4 WARNING: Sum of all thin volume sizes (1.95 GiB) exceeds the size of thin pool thinpool_4_5830/thinpool_4_58300 (1.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "virt4" created. [root@hayes-02 ~]# lvcreate --activate y --virtualsize 500M -T thinpool_4_5830/thinpool_4_58300 -n virt5 WARNING: Sum of all thin volume sizes (2.44 GiB) exceeds the size of thin pool thinpool_4_5830/thinpool_4_58300 (1.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "virt5" created. [root@hayes-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices [lvol0_pmspare] thinpool_4_5830 ewi------- 4.00m /dev/sdl1(0) thinpool_4_58300 thinpool_4_5830 twi-aotz-- 1.00g 0.00 11.43 thinpool_4_58300_tdata(0) [thinpool_4_58300_tdata] thinpool_4_5830 Twi-ao---- 1.00g /dev/sdl1(1) [thinpool_4_58300_tmeta] thinpool_4_5830 ewi-ao---- 4.00m /dev/sdf1(0) virt1 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt2 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt3 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt4 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt5 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 [root@hayes-02 ~]# lvextend -l 528154 /dev/thinpool_4_5830/thinpool_4_58300 Rounding size to boundary between physical extents: <2.02 GiB. WARNING: Sum of all thin volume sizes (2.44 GiB) exceeds the size of thin pools (1.00 GiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Size of logical volume thinpool_4_5830/thinpool_4_58300_tmeta changed from 4.00 MiB (1 extents) to <2.02 GiB (516 extents). Size of logical volume thinpool_4_5830/thinpool_4_58300_tdata changed from 1.00 GiB (256 extents) to 2.01 TiB (528154 extents). Logical volume thinpool_4_5830/thinpool_4_58300_tdata successfully resized. Jan 18 17:33:58 hayes-02 dmeventd[2765]: No longer monitoring thin pool thinpool_4_5830-thinpool_4_58300-tpool. Jan 18 17:33:58 hayes-02 kernel: device-mapper: thin: 253:2: growing the metadata device from 1024 to 528384 blocks Jan 18 17:33:58 hayes-02 lvm[2765]: Monitoring thin pool thinpool_4_5830-thinpool_4_58300-tpool. Jan 18 17:33:58 hayes-02 dmeventd[2765]: No longer monitoring thin pool thinpool_4_5830-thinpool_4_58300-tpool. Jan 18 17:33:58 hayes-02 kernel: device-mapper: thin: Data device (dm-1) discard unsupported: Disabling discard passdown. Jan 18 17:33:58 hayes-02 kernel: device-mapper: thin: 253:2: growing the data device from 16384 to 33801856 blocks Jan 18 17:33:58 hayes-02 lvm[2765]: Monitoring thin pool thinpool_4_5830-thinpool_4_58300-tpool. [root@hayes-02 ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices [lvol0_pmspare] thinpool_4_5830 ewi------- <2.02g /dev/sdl1(0) [lvol0_pmspare] thinpool_4_5830 ewi------- <2.02g /dev/sdl1(257) thinpool_4_58300 thinpool_4_5830 twi-aotz-- 2.01t 0.00 1.18 thinpool_4_58300_tdata(0) [thinpool_4_58300_tdata] thinpool_4_5830 Twi-ao---- 2.01t /dev/sdl1(1) [thinpool_4_58300_tdata] thinpool_4_5830 Twi-ao---- 2.01t /dev/sdl1(772) [thinpool_4_58300_tdata] thinpool_4_5830 Twi-ao---- 2.01t /dev/sdj1(0) [thinpool_4_58300_tmeta] thinpool_4_5830 ewi-ao---- <2.02g /dev/sdf1(0) virt1 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt2 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt3 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt4 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 virt5 thinpool_4_5830 Vwi-a-tz-- 500.00m thinpool_4_58300 0.00 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1659 |