+++ This bug was initially created as a clone of Bug #2129414 +++ Description of problem: In RHCS 5.x there is no blueFS spillover health warning generated when the RocksDB starts consuming block (slower) device space. Version-Release number of selected component (if applicable): RHCS 5.0z4 and RHCS 5.2 How reproducible: Always Steps to Reproduce: 1. Deploy a fresh RHCS 5 or upgrade a cluster from RHCS 4 to RHCS 5 with smaller block DB size (Like 10 Mib or 30 Mib) - For example: ~~~ service_type: osd service_id: osd_nodeXY_paths service_name: osd.osd_nodeXY_paths placement: hosts: - nodeX - nodeY spec: block_db_size: 10485760 <---- data_devices: paths: - /dev/sdb - /dev/sdc db_devices: paths: - /dev/sdd filter_logic: AND objectstore: bluestore ~~~ 2. Add some data into the cluster using RBD 3. Collect the output of the below command and look for the "slow_used_bytes" parameter. ~~~ $ ceph daemon osd.<id> perf dump bluefs ~~~ - If using non-colocated OSDs, then also verify using the below command and look for "SLOW" Column ~~~ $ ceph daemon osd.<id> bluefs stats ~~~ *NOTE*: non-colocated: OSDs having DB and Data on separate devices Actual results: No bluefs spillover health warning Expected results: It should show the bluefs spillover health warning Additional info: Tried to reproduce this issue in RHCS 4.2z4 and successfully able to get the bluefs spillover health warning --- Additional comment from Kritik Sachdeva on 2022-09-23 17:02:23 UTC --- Hello team, Here is the results of the "perf dump" and "bluefs stats" command from the lab environment. + bluefs stats ~~~ [ceph: root@node2 /]# ceph daemon osd.0 bluefs stats 1 : device size 0x7fe000 : using 0x700000(7 MiB) 2 : device size 0x27fc00000 : using 0x8137000(129 MiB) RocksDBBlueFSVolumeSelector: wal_total:0, db_total:7969177, slow_total:10196562739, db_avail:0 Usage matrix: DEV/LEV WAL DB SLOW <-- * * REAL FILES LOG 0 B 4 MiB 0 B 0 B 0 B 688 KiB 1 WAL 0 B 1 MiB 64 KiB 0 B 0 B 1.1 MiB 1 DB 0 B 2 MiB 704 KiB 0 B 0 B 85 KiB 14 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTALS 0 B 7 MiB 768 KiB 0 B 0 B 0 B 16 MAXIMUMS: LOG 0 B 4 MiB 0 B 0 B 0 B 688 KiB WAL 0 B 1 MiB 64 KiB 0 B 0 B 1.1 MiB DB 0 B 3 MiB 768 KiB 0 B 0 B 121 KiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTALS 0 B 7 MiB 832 KiB 0 B 0 B 0 B ~~~ + perf dump bluefs ~~~ [ceph: root@node2 /]# ceph daemon osd.0 perf dump bluefs { "bluefs": { "db_total_bytes": 8380416, "db_used_bytes": 7340032, "wal_total_bytes": 0, "wal_used_bytes": 0, "slow_total_bytes": 10733223936, "slow_used_bytes": 786432, <---- "num_files": 16, "log_bytes": 704512, "log_compactions": 0, "logged_bytes": 499712, "files_written_wal": 1, "files_written_sst": 4, "bytes_written_wal": 1564672, "bytes_written_sst": 20480, "bytes_written_slow": 139264, "max_bytes_wal": 0, "max_bytes_db": 7340032, "max_bytes_slow": 786432, "read_random_count": 41, "read_random_bytes": 11755, "read_random_disk_count": 1, "read_random_disk_bytes": 4168, "read_random_buffer_count": 40, "read_random_buffer_bytes": 7587, "read_count": 70, "read_bytes": 259859, "read_prefetch_count": 7, "read_prefetch_bytes": 7318, "read_zeros_candidate": 0, "read_zeros_errors": 0 } } ~~~ + Cluster health status ~~~ # ceph health detail HEALTH_OK ~~~ + ceph osd df tree ~~~ [ceph: root@node2 /]# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 0.03918 - 40 GiB 1.9 GiB 1.9 GiB 0 B 39 MiB 38 GiB 4.82 1.00 - root default -3 0.01959 - 20 GiB 987 MiB 965 MiB 0 B 20 MiB 19 GiB 4.82 1.00 - host node2 0 hdd 0.00980 1.00000 10 GiB 137 MiB 128 MiB 0 B 7.8 MiB 9.9 GiB 1.34 0.28 2 up osd.0 1 hdd 0.00980 1.00000 10 GiB 850 MiB 837 MiB 0 B 12 MiB 9.2 GiB 8.29 1.72 7 up osd.1 -5 0.01959 - 20 GiB 987 MiB 965 MiB 0 B 20 MiB 19 GiB 4.82 1.00 - host node3 2 hdd 0.00980 1.00000 10 GiB 252 MiB 241 MiB 0 B 10 MiB 9.8 GiB 2.46 0.51 2 up osd.2 3 hdd 0.00980 1.00000 10 GiB 735 MiB 725 MiB 0 B 9.1 MiB 9.3 GiB 7.17 1.49 7 up osd.3 TOTAL 40 GiB 1.9 GiB 1.9 GiB 0 B 39 MiB 38 GiB 4.82 MIN/MAX VAR: 0.28/1.72 STDDEV: 2.97 ~~~ Regards, Kritik Sachdeva --- Additional comment from Adam Kupczyk on 2022-09-26 13:52:37 UTC --- I confirm that code to set spillover health warn has been removed. This is an accidental byproduct of an improvement https://github.com/ceph/ceph/pull/30838 . A new place where this trigger should be added must be devised. --- Additional comment from Vikhyat Umrao on 2022-09-27 17:32:19 UTC --- Good find, Kritik! Adam - what is our plan? Do you think it can be fixed soon? as this is regression? Also, if you have created an upstream tracker please let me know I will attach it to the bug. --- Additional comment from RHEL Program Management on 2022-09-27 17:32:29 UTC --- This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is being proposed as a blocker for this release. Please resolve \triage ASAP. --- Additional comment from Red Hat Bugzilla on 2022-12-31 19:09:42 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2022-12-31 19:13:33 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2022-12-31 19:32:42 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2022-12-31 20:00:05 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2022-12-31 22:43:36 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2022-12-31 23:43:39 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2022-12-31 23:45:58 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2023-01-01 05:35:27 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2023-01-01 06:27:15 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2023-01-01 06:29:08 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2023-01-01 08:39:00 UTC --- Account disabled by LDAP Audit --- Additional comment from Red Hat Bugzilla on 2023-01-01 08:39:47 UTC --- Account disabled by LDAP Audit --- Additional comment from Red Hat Bugzilla on 2023-01-01 08:48:44 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2023-01-01 08:50:17 UTC --- Account disabled by LDAP Audit --- Additional comment from Red Hat Bugzilla on 2023-01-01 08:52:14 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2023-01-09 08:29:27 UTC --- Account disabled by LDAP Audit for extended failure --- Additional comment from Kritik Sachdeva on 2023-02-16 04:10:41 UTC --- Hello team, After discussing with Michael & Vikhyat, attaching the test results from RHCS 5.3 environment where the issue is still present. ## Steps to reproduce ## -------------------------- 1. Deploy a fresh RHCS 5 with smaller block DB size (Like 10 Mib or 30 Mib) - For example: ~~~ service_type: osd service_id: osd_nodeXY_paths service_name: osd.osd_nodeXY_paths placement: hosts: - nodeX - nodeY spec: block_db_size: 10485760 <---- data_devices: paths: - /dev/sdb - /dev/sdc db_devices: paths: - /dev/sdd filter_logic: AND objectstore: bluestore ~~~ 2. Add some data into the cluster using RBD 3. Collect the output of the below command and look for the "slow_used_bytes" parameter. ~~~ $ ceph daemon osd.<id> perf dump bluefs ~~~ - If using non-colocated OSDs, then also verify using the below command and look for "SLOW" Column ~~~ $ ceph daemon osd.<id> bluefs stats ~~~ ======================================================================================================= ## Testing Results ## --------------------- [root@node1ceph5 ~]# ceph -s cluster: id: 3357af48-acf0-11ed-a09d-001a4a00064d health: HEALTH_WARN 1 slow ops, oldest one blocked for 302 sec, mon.node3ceph5 has slow ops services: mon: 3 daemons, quorum node1ceph5,node2ceph5,node3ceph5 (age 12m) mgr: node1ceph5.msurmv(active, since 23m), standbys: node2ceph5.bvkilb osd: 5 osds: 5 up (since 3m), 5 in (since 3m) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 45 MiB used, 110 GiB / 110 GiB avail pgs: 1 active+clean [root@node1ceph5 ~]# ceph versions { "mon": { "ceph version 16.2.10-94.el8cp (48ce8ed67474ea50f10c019b9445be7f49749d23) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.10-94.el8cp (48ce8ed67474ea50f10c019b9445be7f49749d23) pacific (stable)": 2 }, "osd": { "ceph version 16.2.10-94.el8cp (48ce8ed67474ea50f10c019b9445be7f49749d23) pacific (stable)": 5 }, "mds": {}, "overall": { "ceph version 16.2.10-94.el8cp (48ce8ed67474ea50f10c019b9445be7f49749d23) pacific (stable)": 10 } } [root@node1ceph5 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.10748 root default -7 0.02930 host node1ceph5 4 hdd 0.02930 osd.4 up 1.00000 1.00000 -5 0.03909 host node2ceph5 1 hdd 0.02930 osd.1 up 1.00000 1.00000 3 hdd 0.00980 osd.3 up 1.00000 1.00000 -3 0.03909 host node3ceph5 0 hdd 0.02930 osd.0 up 1.00000 1.00000 2 hdd 0.00980 osd.2 up 1.00000 1.00000 [ceph: root@node2ceph5 /]# ceph daemon osd.1 perf dump bluefs { "bluefs": { "db_total_bytes": 8380416, "db_used_bytes": 7340032, "wal_total_bytes": 0, "wal_used_bytes": 0, "slow_total_bytes": 32208060416, "slow_used_bytes": 786432, <----- Non-empty value "num_files": 12, "log_bytes": 307200, "log_compactions": 0, "logged_bytes": 139264, "files_written_wal": 1, "files_written_sst": 3, "bytes_written_wal": 491520, "bytes_written_sst": 8192, "bytes_written_slow": 540672, "max_bytes_wal": 0, "max_bytes_db": 7340032, "max_bytes_slow": 720896, "read_random_count": 18, "read_random_bytes": 3522, "read_random_disk_count": 0, "read_random_disk_bytes": 0, "read_random_buffer_count": 18, "read_random_buffer_bytes": 3522, "read_count": 57, "read_bytes": 218464, "read_prefetch_count": 3, "read_prefetch_bytes": 3328, "read_zeros_candidate": 0, "read_zeros_errors": 0 } } [ceph: root@node2ceph5 /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 110 GiB 110 GiB 45 MiB 45 MiB 0.04 TOTAL 110 GiB 110 GiB 45 MiB 45 MiB 0.04 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 35 GiB [ceph: root@node2ceph5 /]# ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 4 hdd 0.02930 1.00000 30 GiB 8.9 MiB 248 KiB 0 B 7.7 MiB 30 GiB 0.03 0.73 1 up 1 hdd 0.02930 1.00000 30 GiB 9.0 MiB 248 KiB 0 B 7.8 MiB 30 GiB 0.03 0.73 1 up 3 hdd 0.00980 1.00000 10 GiB 8.9 MiB 248 KiB 0 B 7.7 MiB 10 GiB 0.09 2.19 0 up 0 hdd 0.02930 1.00000 30 GiB 9.1 MiB 248 KiB 0 B 7.8 MiB 30 GiB 0.03 0.74 1 up 2 hdd 0.00980 1.00000 10 GiB 9.0 MiB 248 KiB 0 B 7.8 MiB 10 GiB 0.09 2.20 0 up TOTAL 110 GiB 45 MiB 1.2 MiB 0 B 39 MiB 110 GiB 0.04 MIN/MAX VAR: 0.73/2.20 STDDEV: 0.03 [ceph: root@node2ceph5 /]# ceph daemon osd.1 bluefs stats 1 : device size 0x7fe000 : using 0x700000(7 MiB) 2 : device size 0x77fc00000 : using 0x100000(1 MiB) RocksDBBlueFSVolumeSelector: wal_total:0, db_total:7969177, slow_total:30597657395, db_avail:0 Usage matrix: DEV/LEV WAL DB SLOW <--- * * REAL FILES LOG 0 B 4 MiB 0 B 0 B 0 B 300 KiB 1 WAL 0 B 0 B 384 KiB 0 B 0 B 372 KiB 1 DB 0 B 3 MiB 384 KiB 0 B 0 B 76 KiB 10 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTALS 0 B 7 MiB 768 KiB 0 B 0 B 0 B 12 MAXIMUMS: LOG 0 B 4 MiB 0 B 0 B 0 B 300 KiB WAL 0 B 0 B 384 KiB 0 B 0 B 372 KiB DB 0 B 3 MiB 512 KiB 0 B 0 B 112 KiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTALS 0 B 7 MiB 768 KiB 0 B 0 B 0 B // Still no heath warning for bluefs Spillover // [ceph: root@node2ceph5 /]# ceph -s cluster: id: 3357af48-acf0-11ed-a09d-001a4a00064d health: HEALTH_WARN 1 slow ops, oldest one blocked for 627 sec, mon.node3ceph5 has slow ops services: mon: 3 daemons, quorum node1ceph5,node2ceph5,node3ceph5 (age 18m) mgr: node1ceph5.msurmv(active, since 28m), standbys: node2ceph5.bvkilb osd: 5 osds: 5 up (since 8m), 5 in (since 8m) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 45 MiB used, 110 GiB / 110 GiB avail pgs: 1 active+clean // Even after restarting of the OSDs similar results in bluefs stats output and no health warning // [ceph: root@node2ceph5 /]# ceph orch restart osd.osd_node23_paths Scheduled to restart osd.1 on host 'node2ceph5' Scheduled to restart osd.3 on host 'node2ceph5' Scheduled to restart osd.0 on host 'node3ceph5' Scheduled to restart osd.2 on host 'node3ceph5' [ceph: root@node2ceph5 /]# [ceph: root@node2ceph5 /]# ceph daemon osd.1 perf dump bluefs { "bluefs": { "db_total_bytes": 8380416, "db_used_bytes": 7340032, "wal_total_bytes": 0, "wal_used_bytes": 0, "slow_total_bytes": 32208060416, "slow_used_bytes": 983040, <---- Non-empty value "num_files": 19, "log_bytes": 475136, "log_compactions": 0, "logged_bytes": 143360, "files_written_wal": 1, "files_written_sst": 7, "bytes_written_wal": 311296, "bytes_written_sst": 49152, "bytes_written_slow": 176128, "max_bytes_wal": 0, "max_bytes_db": 7340032, "max_bytes_slow": 1376256, "read_random_count": 59, "read_random_bytes": 38730, "read_random_disk_count": 1, "read_random_disk_bytes": 20621, "read_random_buffer_count": 58, "read_random_buffer_bytes": 18109, "read_count": 117, "read_bytes": 812174, "read_prefetch_count": 10, "read_prefetch_bytes": 10286, "read_zeros_candidate": 0, "read_zeros_errors": 0 } } [ceph: root@node2ceph5 /]# ceph daemon osd.1 bluefs stats 1 : device size 0x7fe000 : using 0x700000(7 MiB) 2 : device size 0x77fc00000 : using 0x15a000(1.4 MiB) RocksDBBlueFSVolumeSelector: wal_total:0, db_total:7969177, slow_total:30597657395, db_avail:0 Usage matrix: DEV/LEV WAL DB SLOW <--- * * REAL FILES LOG 0 B 4 MiB 0 B 0 B 0 B 476 KiB 1 WAL 0 B 1 MiB 64 KiB 0 B 0 B 239 KiB 1 DB 0 B 2 MiB 896 KiB 0 B 0 B 111 KiB 17 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTALS 0 B 7 MiB 960 KiB 0 B 0 B 0 B 19 MAXIMUMS: LOG 0 B 4 MiB 0 B 0 B 0 B 476 KiB WAL 0 B 1 MiB 448 KiB 0 B 0 B 417 KiB DB 0 B 3 MiB 960 KiB 0 B 0 B 147 KiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTALS 0 B 7 MiB 1.4 MiB 0 B 0 B 0 B [ceph: root@node2ceph5 /]# ceph -s cluster: id: 3357af48-acf0-11ed-a09d-001a4a00064d health: HEALTH_OK services: mon: 3 daemons, quorum node1ceph5,node2ceph5,node3ceph5 (age 22m) mgr: node1ceph5.msurmv(active, since 32m), standbys: node2ceph5.bvkilb osd: 5 osds: 5 up (since 50s), 5 in (since 12m) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 47 MiB used, 110 GiB / 110 GiB avail pgs: 1 active+clean Regards, Kritik Sachdeva --- Additional comment from Adam Kupczyk on 2023-03-29 13:36:58 UTC --- I think we will be fixing the problem with this: https://github.com/ceph/ceph/pull/49987 + backports. --- Additional comment from Red Hat Bugzilla on 2023-06-20 18:45:35 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Red Hat Bugzilla on 2023-06-20 18:45:46 UTC --- remove performed by PnT Account Manager <pnt-expunge> --- Additional comment from Vikhyat Umrao on 2023-08-30 17:17:07 UTC --- Harsh, if you want you can mark this verified as you already saw in workload-dfg clusters this warning :) --- Additional comment from Vikhyat Umrao on 2023-08-30 17:17:49 UTC --- (In reply to Vikhyat Umrao from comment #25) > Harsh, if you want you can mark this verified as you already saw in > workload-dfg clusters this warning :) I mean when the bz goes ON_QA :) --- Additional comment from errata-xmlrpc on 2023-08-30 19:40:00 UTC --- This bug has been added to advisory RHBA-2023:118213 by Ken Dreyer (kdreyer) --- Additional comment from errata-xmlrpc on 2023-08-30 19:40:00 UTC --- Bug report changed to ON_QA status by Errata System. A QE request has been submitted for advisory RHBA-2023:118213-01 https://errata.devel.redhat.com/advisory/118213 --- Additional comment from Pawan on 2023-08-31 15:52:00 UTC --- Verified the fix. Deployed OSD with data + db device, and observed the spillover . Created very small DB device intentionally to check the warning generation. [ceph: root@ceph-pdhiran-66-bq3qsx-node1-installer /]# ceph orch daemon add osd ceph-pdhiran-66-bq3qsx-node9:data_devices=/dev/data_vg/data-lv1,db_devices=/dev/db_vg/db-lv1 Created osd(s) 1 on host 'ceph-pdhiran-66-bq3qsx-node9' [ceph: root@ceph-pdhiran-66-bq3qsx-node1-installer /]# ceph -s cluster: id: c4a335be-47f3-11ee-a24b-fa163e51ef57 health: HEALTH_WARN 1 OSD(s) experiencing BlueFS spillover 1 stray daemon(s) not managed by cephadm Degraded data redundancy: 9555/120660 objects degraded (7.919%), 27 pgs degraded services: mon: 5 daemons, quorum ceph-pdhiran-66-bq3qsx-node1-installer,ceph-pdhiran-66-bq3qsx-node2,ceph-pdhiran-66-bq3qsx-node11,ceph-pdhiran-66-bq3qsx-node8,ceph-pdhiran-66-bq3qsx-node6 (age 3h) mgr: ceph-pdhiran-66-bq3qsx-node1-installer.cxxbck(active, since 4h), standbys: ceph-pdhiran-66-bq3qsx-node6.twpvla, ceph-pdhiran-66-bq3qsx-node2.mnzoos osd: 17 osds: 17 up (since 17s), 17 in (since 3h); 28 remapped pgs data: pools: 2 pools, 129 pgs objects: 40.22k objects, 38 GiB usage: 118 GiB used, 306 GiB / 425 GiB avail pgs: 9555/120660 objects degraded (7.919%) 247/120660 objects misplaced (0.205%) 101 active+clean 27 active+recovery_wait+undersized+degraded+remapped 1 active+recovering+undersized+remapped io: recovery: 78 MiB/s, 79 objects/s [ceph: root@ceph-pdhiran-66-bq3qsx-node1-installer /]# ceph health detail HEALTH_WARN 1 OSD(s) experiencing BlueFS spillover; 1 stray daemon(s) not managed by cephadm; Degraded data redundancy: 8952/120660 objects degraded (7.419%), 25 pgs degraded [WRN] BLUEFS_SPILLOVER: 1 OSD(s) experiencing BlueFS spillover osd.1 spilled over 65 MiB metadata from 'db' device (3 MiB used of 4.0 MiB) to slow device # ceph version ceph version 18.2.0-2.el9cp (cbed2329dbd9e1c06cf77afcfad901ae16cc5e6a) reef (stable) --- Additional comment from Pawan on 2023-09-07 03:18:33 UTC --- Hello Neha, Vikhyat, Do we have backports of this bug, for the issue to be fixed in 5.x & 6.x? I feel that we should include this fix in other releases as well. --- Additional comment from Vikhyat Umrao on 2023-09-07 13:18:31 UTC --- (In reply to Pawan from comment #30) > Hello Neha, Vikhyat, Do we have backports of this bug, for the issue to be > fixed in 5.x & 6.x? I feel that we should include this fix in other releases > as well. I see in upstream we do have backports for pacific and quincy we could do it for 5.3.z and 6.1z. I agree it would be needed. Thank you for highlighting it.
quincy backport - https://github.com/ceph/ceph/pull/50931
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Ceph Storage 6.1 security, enhancements, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:7740
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days