Bug 2342752 - After performing the osd resize test the osd pods failed to recover and the cluster ceph health was not OK
Summary: After performing the osd resize test the osd pods failed to recover and the c...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RADOS
Version: 9.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 8.1
Assignee: Adam Kupczyk
QA Contact: Harsh Kumar
URL:
Whiteboard:
Depends On:
Blocks: 2342817
TreeView+ depends on / blocked
 
Reported: 2025-01-29 08:19 UTC by Parth Arora
Modified: 2025-06-26 12:24 UTC (History)
9 users (show)

Fixed In Version: ceph-19.2.1-3.el9cp
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2342817 (view as bug list)
Environment:
Last Closed: 2025-06-26 12:24:27 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ceph ceph pull 61470 0 None open os/bluestore: CBT bluefs-bdev-expand addendum 2025-01-29 14:46:07 UTC
Github ceph ceph pull 61671 0 None Merged squid: os/bluestore: Create additional bdev labels when expanding block device. 2025-04-16 20:02:25 UTC
Red Hat Issue Tracker DFBUGS-1467 0 None None None 2025-01-29 08:20:36 UTC
Red Hat Issue Tracker RHCEPH-10527 0 None None None 2025-01-29 08:19:51 UTC
Red Hat Product Errata RHSA-2025:9775 0 None None None 2025-06-26 12:24:35 UTC

Description Parth Arora 2025-01-29 08:19:31 UTC
Description of problem:

The osd resize is failing in 4.18 branches intermittently,
`ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-x`

Errros from different clusters:
```
2025-01-10T14:52:34.944+0000 7fb4b70d2940 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
```

```
2025-01-27T15:45:05.164888150Z 2025-01-27T15:45:05.164+0000 7f60b29c5940 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 62: Decoder at 'void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=2 cannot decode v=41 minimal_decoder=66: Malformed input [buffer:3]
```

```

Errors:
IBM cloud
1)

2025-01-10T14:52:34.944+0000 7fb4b70d2940 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
2025-01-27T15:45:05.164888150Z 2025-01-27T15:45:05.164+0000 7f60b29c5940 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 62: Decoder at 'void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=2 cannot decode v=41 minimal_decoder=66: Malformed input [buffer:3]
Aws

2025-01-13T15:13:38.683+0000 7fa5eb2d4940 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 62: Decoder at 'void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=2 cannot decode v=1 minimal_decoder=104: Malformed input [buffer:3]
```
Probably coming from here:https://github.com/ceph/ceph/blame/squid-release/src/os/bluestore/BlueStore.cc#L6612

For more info: https://github.com/rook/rook/pull/15251#issuecomment-2618453748

Interesting thing, in failed cases if we update the osd image to upstream `quay.io/ceph/ceph:v18.2.4`, the resizing worked,


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 8 errata-xmlrpc 2025-06-26 12:24:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775


Note You need to log in before you can comment on or make changes to this bug.