Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2227063

Summary: [RFE] cephadm: support osd_objectstore variable in OSD deployment configuration
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Harsh Kumar <hakumar>
Component: CephadmAssignee: Adam King <adking>
Status: CLOSED UPSTREAM QA Contact: Harsh Kumar <hakumar>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.1CC: adking, akupczyk, bhubbard, ceph-eng-bugs, cephqe-warriors, ngangadh, nojha, sabose, sapillai, saraut, vumrao
Target Milestone: ---Keywords: FutureFeature
Target Release: 10.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2026-03-04 08:51:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Harsh Kumar 2023-07-27 16:03:50 UTC 
Description of problem:
Observed during the testing of BZ2226657
As per the explanation available in these comments - BZ2226657#c13 & BZ2226657#c9
when the OSDs are initialized with osd_objectstore parameter already set to bluestore-rdr, it is expected that "ceph config show" would return bluestore-rdr as the value against osd_objectstore.

Moreoever, OSD backend type post deployment and initialization of OSD should also return "bluestore-rdr"
And likewise, the "type" metadata label should also be "bluestore-rdr"

Currently, ceph config show displays the correct value of bluestore-rdr
However, both OSD backend type and "type" entry in metadata label return "bluestore" instead of "bluestore-rdr"

This would essentially mean that bluestore-rdr is not taking effect

Version-Release number of selected component (if applicable):
ceph version 17.2.6-100.el9cp (ea4e3ef8df2cf26540aae06479df031dcfc80343) quincy (stable)

How reproducible:
2/2

Steps to Reproduce:
1. Deploy a Ceph cluster with initializing OSD 
2. Set the osd_objectstore config to bluestore-rdr
3. Initialize the OSDs
4. Verify osd_objectstore value in ceph config show
5. Verify the osd backend type from any of the OSDs
6. List the metadata label for any of the OSDs using ceph-bluestore-tool

Actual results:
Currently, ceph config show displays the correct value of bluestore-rdr for osd_objectstore
However, both OSD backend type and "type" entry in metadata label return "bluestore" instead of "bluestore-rdr"

Expected results:
Value of osd_objectstore in "ceph config show", OSD backend type, and "type" metadata label should all be 'bluestore-rdr' if OSDs are initialized with this parameter.


Additional info:
# ceph version
ceph version 17.2.6-100.el9cp (ea4e3ef8df2cf26540aae06479df031dcfc80343) quincy (stable)

[ceph: root@ceph-hakumar-4mmunb-node1-installer /]# ceph status
  cluster:
    id:     e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04
    health: HEALTH_WARN
            1 MDSs report slow metadata IOs
            Reduced data availability: 3 pgs inactive
            OSD count 0 < osd_pool_default_size 3
 
  services:
    mon: 3 daemons, quorum ceph-hakumar-4mmunb-node1-installer,ceph-hakumar-4mmunb-node6,ceph-hakumar-4mmunb-node2 (age 15m)
    mgr: ceph-hakumar-4mmunb-node1-installer.sowakk(active, since 19m), standbys: ceph-hakumar-4mmunb-node2.aoopfk, ceph-hakumar-4mmunb-node6.qgobwn
    mds: 1/1 daemons up, 1 standby
    osd: 0 osds: 0 up, 0 in
 
  data:
    volumes: 1/1 healthy
    pools:   3 pools, 3 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     100.000% pgs unknown
             3 unknown
 
  progress:
    Global Recovery Event (0s)
      [............................] 
 
[ceph: root@ceph-hakumar-4mmunb-node1-installer /]# ceph config get osd osd_objectstore
bluestore

[ceph: root@ceph-hakumar-4mmunb-node1-installer /]# ceph config set osd osd_objectstore bluestore-rdr

[ceph: root@ceph-hakumar-4mmunb-node1-installer /]# ceph config get osd osd_objectstore
bluestore-rdr

[ceph: root@ceph-hakumar-4mmunb-node1-installer /]# ceph orch apply osd  --all-available-devices
Scheduled osd.all-available-devices update...

# ceph config show osd.0 | grep objectstore
osd_objectstore                                  bluestore-rdr                                                                                                                                  mon

# cat /var/lib/ceph/e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04/osd.0/type 
bluestore

[root@ceph-hakumar-4mmunb-node5 ~]# systemctl -a | grep osd.0
  ceph-e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04.service                                                                                                                                                         loaded    active   running   Ceph osd.0 for e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04

[root@ceph-hakumar-4mmunb-node5 ~]# systemctl stop ceph-e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04.service

[root@ceph-hakumar-4mmunb-node5 ~]# cephadm shell --name osd.0 -- ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0
Inferring fsid e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04
Inferring config /var/lib/ceph/e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04/osd.0/config
Using ceph image with id 'c9eb30bbf30e' and tag '<none>' created on 2023-07-26 17:33:06 +0000 UTC
registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2851c553bd88cb643546bab6058eccf7ebdee389ca29790abcfbeab805e77682
inferring bluefs devices from bluestore path
{
    "/var/lib/ceph/osd/ceph-0/block": {
        "osd_uuid": "bcf95a28-9d62-421a-9b72-232f23d9963b",
        "size": 26839351296,
        "btime": "2023-07-27T15:34:15.452488+0000",
        "description": "main",
        "bfm_blocks": "6552576",
        "bfm_blocks_per_key": "128",
        "bfm_bytes_per_block": "4096",
        "bfm_size": "26839351296",
        "bluefs": "1",
        "ceph_fsid": "e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04",
        "ceph_version_when_created": "ceph version 17.2.6-100.el9cp (ea4e3ef8df2cf26540aae06479df031dcfc80343) quincy (stable)",
        "created_at": "2023-07-27T15:34:17.072286Z",
        "kv_backend": "rocksdb",
        "magic": "ceph osd volume v026",
        "mkfs_done": "yes",
        "osd_key": "AQB2jsJkNlkiFhAABPvtKiQh4yoaxO66DUlIfg==",
        "osdspec_affinity": "all-available-devices",
        "ready": "ready",
        "require_osd_release": "17",
        "type": "bluestore",
        "whoami": "0"
    }
}

# cephadm shell --name osd.0 -- ceph-bluestore-tool prime-osd-dir --dev /dev/ceph-59639f9d-afd0-4e61-a025-a671284ad4fb/osd-block-bcf95a28-9d62-421a-9b72-232f23d9963b --path /var/lib/ceph/osd/ceph-0
Inferring fsid e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04
Inferring config /var/lib/ceph/e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04/osd.0/config
Using ceph image with id 'c9eb30bbf30e' and tag '<none>' created on 2023-07-26 17:33:06 +0000 UTC
registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:2851c553bd88cb643546bab6058eccf7ebdee389ca29790abcfbeab805e77682

[root@ceph-hakumar-4mmunb-node5 ~]# cat /var/lib/ceph/e0399ebe-2c8f-11ee-9a9f-fa163e7a4d04/osd.0/type
bluestore
Comment 7 Santosh Pillai 2023-07-31 16:08:20 UTC 
sh-4.4# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0
inferring bluefs devices from bluestore path
{
    "/var/lib/ceph/osd/ceph-0/block": {
        "osd_uuid": "bb74ec3d-172d-42de-b666-7a64dc750100",
        "size": 10737418240,
        "btime": "2023-07-31T15:31:23.112671+0000",
        "description": "main",
        "bfm_blocks": "2621440",
        "bfm_blocks_per_key": "128",
        "bfm_bytes_per_block": "4096",
        "bfm_size": "10737418240",
        "bluefs": "1",
        "ceph_fsid": "48e05304-fc23-4119-9332-32ac98668622",
        "ceph_version_when_created": "ceph version 17.2.6-377-ga98ae05d (a98ae05dba83682ea4ec75bd63be2f83b984d29c) quincy (stable)",
        "created_at": "2023-07-31T15:31:25.219628Z",
        "kv_backend": "rocksdb",
        "magic": "ceph osd volume v026",
        "mkfs_done": "yes",
        "osd_key": "AQDI08dkxgh4MRAAfyLAOHDokeYZt5wRy9wDGw==",
        "ready": "ready",
        "require_osd_release": "17",
        "type": "bluestore",
        "whoami": "0"
    }
}
sh-4.4# cat /var/lib/ceph/osd/ceph-0/type
bluestore
Comment 13 Santosh Pillai 2023-08-25 15:23:19 UTC 
`ceph osd metadata osd.0` command still shows `bluestore` even after OSD is migrated to use `bluestore-rdr` 


Before upgrade:

sh-4.4$ ceph osd metadata osd.0
{
    "id": 0,
    "arch": "x86_64",
    "back_addr": "[v2:10.244.1.9:6802/2446454422,v1:10.244.1.9:6803/2446454422]",
    "back_iface": "",
    "bluefs": "1",
    "bluefs_dedicated_db": "0",
    "bluefs_dedicated_wal": "0",
    "bluefs_single_shared_device": "1",
    "bluestore_bdev_access_mode": "blk",
    "bluestore_bdev_block_size": "4096",
    "bluestore_bdev_dev_node": "/dev/dm-0",
    "bluestore_bdev_devices": "vdc",
    "bluestore_bdev_driver": "KernelDevice",
    "bluestore_bdev_optimal_io_size": "0",
    "bluestore_bdev_partition_path": "/dev/dm-0",
    "bluestore_bdev_rotational": "1",
    "bluestore_bdev_size": "10720641024",
    "bluestore_bdev_support_discard": "1",
    "bluestore_bdev_type": "hdd",
    "bluestore_min_alloc_size": "4096",
    "ceph_release": "quincy",
    "ceph_version": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
    "ceph_version_short": "17.2.6-373-g5406f4b1",
    "ceph_version_when_created": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
    "container_hostname": "rook-ceph-osd-0-6466b5bcb5-d8lmm",
    "container_image": "quay.io/guits/ceph-volume:bs-rdr",
    "cpu": "Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz",
    "created_at": "2023-08-25T15:08:41.150964Z",
    "default_device_class": "hdd",
    "device_ids": "",
    "device_paths": "",
    "devices": "vdc",
    "distro": "centos",
    "distro_description": "CentOS Stream 8",
    "distro_version": "8",
    "front_addr": "[v2:10.244.1.9:6800/2446454422,v1:10.244.1.9:6801/2446454422]",
    "front_iface": "",
    "hb_back_addr": "[v2:10.244.1.9:6806/2446454422,v1:10.244.1.9:6807/2446454422]",
    "hb_front_addr": "[v2:10.244.1.9:6804/2446454422,v1:10.244.1.9:6805/2446454422]",
    "hostname": "minikube-m02",
    "journal_rotational": "1",
    "kernel_description": "#1 SMP Fri Jan 27 18:05:35 UTC 2023",
    "kernel_version": "5.10.57",
    "mem_swap_kb": "0",
    "mem_total_kb": "2165900",
    "network_numa_unknown_ifaces": "back_iface,front_iface",
    "objectstore_numa_unknown_devices": "vdc",
    "os": "Linux",
    "osd_data": "/var/lib/ceph/osd/ceph-0",
    "osd_objectstore": "bluestore",
    "osdspec_affinity": "",
    "pod_name": "rook-ceph-osd-0-6466b5bcb5-d8lmm",
    "pod_namespace": "rook-ceph",
    "rotational": "1"
}
sh-4.4$ 


sh-4.4# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0
inferring bluefs devices from bluestore path
{
    "/var/lib/ceph/osd/ceph-0/block": {
        "osd_uuid": "7add3b6c-b8e9-42b0-8dc4-feb1af6e5394",
        "size": 10720641024,
        "btime": "2023-08-25T15:08:38.966496+0000",
        "description": "main",
        "bfm_blocks": "2617344",
        "bfm_blocks_per_key": "128",
        "bfm_bytes_per_block": "4096",
        "bfm_size": "10720641024",
        "bluefs": "1",
        "ceph_fsid": "e0c985eb-b626-4807-9681-b7a836209793",
        "ceph_version_when_created": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
        "created_at": "2023-08-25T15:08:41.150964Z",
        "kv_backend": "rocksdb",
        "magic": "ceph osd volume v026",
        "mkfs_done": "yes",
        "osd_key": "AQDqw+hkvhDHKhAA5r+WYvA++VOyplGBHynBOg==",
        "ready": "ready",
        "require_osd_release": "17",
        "type": "bluestore",
        "whoami": "0"
    }
}


sh-4.4# cat /var/lib/ceph/osd/ceph-0/type
bluestore


-----------------------------------------------------------------------------------

After Upgrade:

sh-4.4$ ceph osd metadata osd.0
{
    "id": 0,
    "arch": "x86_64",
    "back_addr": "[v2:10.244.1.14:6802/2651410559,v1:10.244.1.14:6803/2651410559]",
    "back_iface": "",
    "bluefs": "1",
    "bluefs_dedicated_db": "0",
    "bluefs_dedicated_wal": "0",
    "bluefs_single_shared_device": "1",
    "bluestore_bdev_access_mode": "blk",
    "bluestore_bdev_block_size": "4096",
    "bluestore_bdev_dev_node": "/dev/dm-0",
    "bluestore_bdev_devices": "vdc",
    "bluestore_bdev_driver": "KernelDevice",
    "bluestore_bdev_optimal_io_size": "0",
    "bluestore_bdev_partition_path": "/dev/dm-0",
    "bluestore_bdev_rotational": "1",
    "bluestore_bdev_size": "10720641024",
    "bluestore_bdev_support_discard": "1",
    "bluestore_bdev_type": "hdd",
    "bluestore_min_alloc_size": "4096",
    "ceph_release": "quincy",
    "ceph_version": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
    "ceph_version_short": "17.2.6-373-g5406f4b1",
    "ceph_version_when_created": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
    "container_hostname": "rook-ceph-osd-0-85db7bfdd8-vxq65",
    "container_image": "quay.io/guits/ceph-volume:bs-rdr",
    "cpu": "Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz",
    "created_at": "2023-08-25T15:15:51.148428Z",
    "default_device_class": "hdd",
    "device_ids": "",
    "device_paths": "",
    "devices": "vdc",
    "distro": "centos",
    "distro_description": "CentOS Stream 8",
    "distro_version": "8",
    "front_addr": "[v2:10.244.1.14:6800/2651410559,v1:10.244.1.14:6801/2651410559]",
    "front_iface": "",
    "hb_back_addr": "[v2:10.244.1.14:6806/2651410559,v1:10.244.1.14:6807/2651410559]",
    "hb_front_addr": "[v2:10.244.1.14:6804/2651410559,v1:10.244.1.14:6805/2651410559]",
    "hostname": "minikube-m02",
    "journal_rotational": "1",
    "kernel_description": "#1 SMP Fri Jan 27 18:05:35 UTC 2023",
    "kernel_version": "5.10.57",
    "mem_swap_kb": "0",
    "mem_total_kb": "2165900",
    "network_numa_unknown_ifaces": "back_iface,front_iface",
    "objectstore_numa_unknown_devices": "vdc",
    "os": "Linux",
    "osd_data": "/var/lib/ceph/osd/ceph-0",
    "osd_objectstore": "bluestore",
    "osdspec_affinity": "",
    "pod_name": "rook-ceph-osd-0-85db7bfdd8-vxq65",
    "pod_namespace": "rook-ceph",
    "rotational": "1"
}
sh-4.4$ 

sh-4.4# cat /var/lib/ceph/osd/ceph-0/type
bluestore-rdr

sh-4.4# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0
inferring bluefs devices from bluestore path
{
    "/var/lib/ceph/osd/ceph-0/block": {
        "osd_uuid": "16f9170b-8ef0-4287-96ed-60f745781128",
        "size": 10720641024,
        "btime": "2023-08-25T15:15:49.513276+0000",
        "description": "main",
        "bfm_blocks": "2617344",
        "bfm_blocks_per_key": "128",
        "bfm_bytes_per_block": "4096",
        "bfm_size": "10720641024",
        "bluefs": "1",
        "ceph_fsid": "e0c985eb-b626-4807-9681-b7a836209793",
        "ceph_version_when_created": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
        "created_at": "2023-08-25T15:15:51.148428Z",
        "kv_backend": "rocksdb",
        "magic": "ceph osd volume v026",
        "mkfs_done": "yes",
        "osd_key": "AQCVxehkJ+3qNhAAHf5zZWPKGZ6XMlm8SI6saw==",
        "ready": "ready",
        "require_osd_release": "17",
        "type": "bluestore-rdr",
        "whoami": "0"
    }
}
sh-4.4#
Comment 18 Abhishek Kane 2025-12-15 15:27:23 UTC 
Moving the RFE to 10.0.
Comment 19 Red Hat Bugzilla 2026-03-04 08:51:21 UTC 
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.
Comment 20 Red Hat Bugzilla 2026-03-05 04:26:00 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days or the product is inactive and locked