Description of problem: ceph-objectstore-tool and ceph-bluestore-tool fail with I/O mount error in a brownfield deployment Observed when cluster was upgraded in these scenarios: 5.3 (16.2.10-247) -> 6.1 (17.2.6-190) 5.3 (16.2.10-247) -> 7.0 GA (18.2.0-131) 6.1 (17.2.6-189) -> 7.0 GA (18.2.0-131) Version-Release number of selected component (if applicable): ceph version 18.2.0-131.el9cp (d2f32f94f1c60fec91b161c8a1f200fca2bb8858) reef (stable) How reproducible: 5/5 Steps to Reproduce: 1. Deploy a RHCS 5.3 or RHCS 6.1 cluster 2. Upgrade the cluster to RHCS 7.0 GA https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/7/html/upgrade_guide/upgrade-a-red-hat-ceph-storage-cluster-using-cephadm#upgrading-the-red-hat-ceph-storage-cluster_upgrade 3. Post successful upgrade, choose an OSD at random 4. On the osd host machine, stop the osd service through systemctl # systemctl stop ceph-65b38b76-b148-11ee-9c08-fa163e93f383.service 5. Through cephadm get inside the osd container's shell cephadm shell --name osd.# 6. Execute ceph-objectstore-tool or ceph-bluestore-tool commands Actual results: [root@ceph-upgrade-cot-3qfewv-node4 ~]# cephadm shell --name osd.0 Inferring fsid e2f98e88-b255-11ee-8c99-fa163ed2a99b Inferring config /var/lib/ceph/e2f98e88-b255-11ee-8c99-fa163ed2a99b/osd.0/config Using ceph image with id '024e63562755' and tag '<none>' created on 2024-01-11 23:23:22 +0000 UTC registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1d2572ec0dd321c9f6327b3687b25579eb42e37d9cc607a70ae554cbcc1b970d [ceph: root@ceph-upgrade-cot-3qfewv-node4 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list Mount failed with '(5) Input/output error' [root@ceph-hakumar-fvlbhb-node4 yum.repos.d]# systemctl stop ceph-2ca31110-b118-11ee-a618-fa163e02665f.service [root@ceph-hakumar-fvlbhb-node4 yum.repos.d]# cephadm shell --name osd.2 Inferring fsid 2ca31110-b118-11ee-a618-fa163e02665f Inferring config /var/lib/ceph/2ca31110-b118-11ee-a618-fa163e02665f/osd.2/config Using ceph image with id '7accf94326fc' and tag '<none>' created on 2024-01-11 06:35:02 +0000 UTC registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:48496e4a881e0a4e6dcafd9dadf77fe2f41f43246dc5645af89b27d22dd813d3 [ceph: root@ceph-hakumar-fvlbhb-node4 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2/ --op list Mount failed with '(5) Input/output error' [ceph: root@ceph-hakumar-fvlbhb-node4 /]# exit exit [root@ceph-hakumar-fvlbhb-node4 yum.repos.d]# cephadm version cephadm version 18.2.0-131.el9cp (d2f32f94f1c60fec91b161c8a1f200fca2bb8858) reef (stable) [ceph: root@ceph-hakumar-12u5ze-node4 /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-8/ fsck 2024-01-11T21:37:51.871+0000 7f8a1ddb5540 -1 bluefs _replay 0x426000: stop: unrecognized op 12 2024-01-11T21:37:51.871+0000 7f8a1ddb5540 -1 bluefs mount failed to replay log: (5) Input/output error 2024-01-11T21:37:51.871+0000 7f8a1ddb5540 -1 bluestore(/var/lib/ceph/osd/ceph-8) _open_bluefs failed bluefs mount: (5) Input/output error 2024-01-11T21:37:51.871+0000 7f8a1ddb5540 -1 bluestore(/var/lib/ceph/osd/ceph-8) _open_db failed to prepare db environment: fsck failed: (5) Input/output error Expected results: [ceph: root@ceph-hakumar-n69ot0-node3 /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-3/ fsck fsck success [ceph: root@ceph-hakumar-n69ot0-node3 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --op list ["6.1f",{"oid":"notify.0","key":"","snapid":-2,"hash":1126365855,"max":0,"pool":6,"namespace":"","max":0}] ["5.19",{"oid":"obj_delete_at_hint.0000000100","key":"","snapid":-2,"hash":27468569,"max":0,"pool":5,"namespace":"","max":0}] ["5.19",{"oid":"lc.9","key":"","snapid":-2,"hash":2339156633,"max":0,"pool":5,"namespace":"lc","max":0}] ["5.19",{"oid":"gc.0","key":"","snapid":-2,"hash":52147801,"max":0,"pool":5,"namespace":"gc","max":0}] ["5.19",{"oid":"obj_delete_at_hint.0000000093","key":"","snapid":-2,"hash":103627321,"max":0,"pool":5,"namespace":"","max":0}] ["5.19",{"oid":"lc.15","key":"","snapid":-2,"hash":877642937,"max":0,"pool":5,"namespace":"lc","max":0}] ["5.19",{"oid":"lc.19","key":"","snapid":-2,"hash":4147430329,"max":0,"pool":5,"namespace":"lc","max":0}] ["5.19",{"oid":"obj_delete_at_hint.0000000004","key":"","snapid":-2,"hash":3987729273,"max":0,"pool":5,"namespace":"","max":0}] ["5.19",{"oid":"obj_delete_at_hint.0000000063","key":"","snapid":-2,"hash":2378447737,"max":0,"pool":5,"namespace":"","max":0}] ["5.19",{"oid":"obj_delete_at_hint.0000000122","key":"","snapid":-2,"hash":2989065465,"max":0,"pool":5,"namespace":"","max":0}] ["5.19",{"oid":"lc.21","key":"","snapid":-2,"hash":305547001,"max":0,"pool":5,"namespace":"lc","max":0}] ["5.19",{"oid":"obj_delete_at_hint.0000000057","key":"","snapid":-2,"hash":2350856185,"max":0,"pool":5,"namespace":"","max":0}] ["2.13",{"oid":"mds0_inotable","key":"","snapid":-2,"hash":3092428947,"max":0,"pool":2,"namespace":"","max":0}] ["2.13",{"oid":"100.00000000","key":"","snapid":-2,"hash":3307625139,"max":0,"pool":2,"namespace":"","max":0}] ["5.14",{"oid":"reshard.0000000001","key":"","snapid":-2,"hash":3291492404,"max":0,"pool":5,"namespace":"reshard","max":0}] ["5.14",{"oid":"obj_delete_at_hint.0000000088","key":"","snapid":-2,"hash":2187825332,"max":0,"pool":5,"namespace":"","max":0}] ["5.14",{"oid":"obj_delete_at_hint.0000000080","key":"","snapid":-2,"hash":2088300980,"max":0,"pool":5,"namespace":"","max":0}] ["5.14",{"oid":"obj_delete_at_hint.0000000090","key":"","snapid":-2,"hash":1552334260,"max":0,"pool":5,"namespace":"","max":0}] ["5.14",{"oid":"lc.22","key":"","snapid":-2,"hash":455327348,"max":0,"pool":5,"namespace":"lc","max":0}] ["5.14",{"oid":"obj_delete_at_hint.0000000110","key":"","snapid":-2,"hash":3549983604,"max":0,"pool":5,"namespace":"","max":0}] ["5.14",{"oid":"lc.17","key":"","snapid":-2,"hash":293272820,"max":0,"pool":5,"namespace":"lc","max":0}] ["2.d",{"oid":"mds_snaptable","key":"","snapid":-2,"hash":3640815789,"max":0,"pool":2,"namespace":"","max":0}] ["5.12",{"oid":"obj_delete_at_hint.0000000064","key":"","snapid":-2,"hash":2380537618,"max":0,"pool":5,"namespace":"","max":0}] ["5.12",{"oid":"obj_delete_at_hint.0000000071","key":"","snapid":-2,"hash":2801038418,"max":0,"pool":5,"namespace":"","max":0}] ["5.12",{"oid":"obj_delete_at_hint.0000000074","key":"","snapid":-2,"hash":769383634,"max":0,"pool":5,"namespace":"","max":0}] ["5.12",{"oid":"gc.8","key":"","snapid":-2,"hash":1167531762,"max":0,"pool":5,"namespace":"gc","max":0}] ["5.17",{"oid":"reshard.0000000000","key":"","snapid":-2,"hash":492705943,"max":0,"pool":5,"namespace":"reshard","max":0}] ["5.17",{"oid":"obj_delete_at_hint.0000000050","key":"","snapid":-2,"hash":2363050551,"max":0,"pool":5,"namespace":"","max":0}] ["5.17",{"oid":"obj_delete_at_hint.0000000102","key":"","snapid":-2,"hash":3543557943,"max":0,"pool":5,"namespace":"","max":0}] ["5.17",{"oid":"lc.1","key":"","snapid":-2,"hash":1269515895,"max":0,"pool":5,"namespace":"lc","max":0}] ["5.7",{"oid":"gc.15","key":"","snapid":-2,"hash":312892935,"max":0,"pool":5,"namespace":"gc","max":0}] ["5.7",{"oid":"obj_delete_at_hint.0000000086","key":"","snapid":-2,"hash":2649155847,"max":0,"pool":5,"namespace":"","max":0}] ["5.7",{"oid":"reshard.0000000008","key":"","snapid":-2,"hash":2959979399,"max":0,"pool":5,"namespace":"reshard","max":0}] ["5.7",{"oid":"lc.28","key":"","snapid":-2,"hash":4206672071,"max":0,"pool":5,"namespace":"lc","max":0}] ["5.7",{"oid":"gc.29","key":"","snapid":-2,"hash":1154039335,"max":0,"pool":5,"namespace":"gc","max":0}] ["5.7",{"oid":"gc.20","key":"","snapid":-2,"hash":1458521191,"max":0,"pool":5,"namespace":"gc","max":0}] Additional info: Could be similar to https://tracker.ceph.com/issues/15376 Cluster access: ssh -l root 10.0.208.194 pass: passwd Upgraded from Quincy to Reef
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Critical: Red Hat Ceph Storage 7.1 security, enhancements, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:3925