Bug 2226657

Summary: CephObjectStoreTool not reading type metadata label
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Santosh Pillai <sapillai>
Component: RADOSAssignee: Adam Kupczyk <akupczyk>
Status: CLOSED ERRATA QA Contact: Harsh Kumar <hakumar>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.1CC: akraj, akupczyk, bhubbard, ceph-eng-bugs, cephqe-warriors, hakumar, kdreyer, kseeger, nojha, vereddy, vumrao
Target Milestone: ---Keywords: TestBlocker
Target Release: 6.1z1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-17.2.6-100.el9cp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2226662 (view as bug list) Environment:
Last Closed: 2023-08-03 16:45:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2221020, 2225176, 2226662    

Description Santosh Pillai 2023-07-26 06:24:17 UTC 
Description of problem:
`ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-0` fails to read the bluestore type

Error:
`inferring bluefs devices from bluestore path
expected bluestore, but type is`

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.Deploy rook using downstream ceph (commit: b53d9dfff6b1021fbab8e28f2b873d7d49cf5665)
2.Observe the osd creation flow

Actual results:The init container for `expand-bluefs` fails
sh-5.1# ceph-bluestore-tool bluefs-bdev-expand --path  /var/lib/ceph/osd/ceph-0 --log-level 30
inferring bluefs devices from bluestore path
expected bluestore, but type is 
sh-5.1#

Expected results: The command should not fail.



Additional info: BZ
Comment 18 errata-xmlrpc 2023-08-03 16:45:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 6.1 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:4473
Comment 19 Santosh Pillai 2023-08-25 15:22:51 UTC 
`ceph osd metadata osd.0` command still shows `bluestore` even after OSD is migrated to use `bluestore-rdr` 


Before upgrade:

sh-4.4$ ceph osd metadata osd.0
{
    "id": 0,
    "arch": "x86_64",
    "back_addr": "[v2:10.244.1.9:6802/2446454422,v1:10.244.1.9:6803/2446454422]",
    "back_iface": "",
    "bluefs": "1",
    "bluefs_dedicated_db": "0",
    "bluefs_dedicated_wal": "0",
    "bluefs_single_shared_device": "1",
    "bluestore_bdev_access_mode": "blk",
    "bluestore_bdev_block_size": "4096",
    "bluestore_bdev_dev_node": "/dev/dm-0",
    "bluestore_bdev_devices": "vdc",
    "bluestore_bdev_driver": "KernelDevice",
    "bluestore_bdev_optimal_io_size": "0",
    "bluestore_bdev_partition_path": "/dev/dm-0",
    "bluestore_bdev_rotational": "1",
    "bluestore_bdev_size": "10720641024",
    "bluestore_bdev_support_discard": "1",
    "bluestore_bdev_type": "hdd",
    "bluestore_min_alloc_size": "4096",
    "ceph_release": "quincy",
    "ceph_version": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
    "ceph_version_short": "17.2.6-373-g5406f4b1",
    "ceph_version_when_created": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
    "container_hostname": "rook-ceph-osd-0-6466b5bcb5-d8lmm",
    "container_image": "quay.io/guits/ceph-volume:bs-rdr",
    "cpu": "Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz",
    "created_at": "2023-08-25T15:08:41.150964Z",
    "default_device_class": "hdd",
    "device_ids": "",
    "device_paths": "",
    "devices": "vdc",
    "distro": "centos",
    "distro_description": "CentOS Stream 8",
    "distro_version": "8",
    "front_addr": "[v2:10.244.1.9:6800/2446454422,v1:10.244.1.9:6801/2446454422]",
    "front_iface": "",
    "hb_back_addr": "[v2:10.244.1.9:6806/2446454422,v1:10.244.1.9:6807/2446454422]",
    "hb_front_addr": "[v2:10.244.1.9:6804/2446454422,v1:10.244.1.9:6805/2446454422]",
    "hostname": "minikube-m02",
    "journal_rotational": "1",
    "kernel_description": "#1 SMP Fri Jan 27 18:05:35 UTC 2023",
    "kernel_version": "5.10.57",
    "mem_swap_kb": "0",
    "mem_total_kb": "2165900",
    "network_numa_unknown_ifaces": "back_iface,front_iface",
    "objectstore_numa_unknown_devices": "vdc",
    "os": "Linux",
    "osd_data": "/var/lib/ceph/osd/ceph-0",
    "osd_objectstore": "bluestore",
    "osdspec_affinity": "",
    "pod_name": "rook-ceph-osd-0-6466b5bcb5-d8lmm",
    "pod_namespace": "rook-ceph",
    "rotational": "1"
}
sh-4.4$ 


sh-4.4# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0
inferring bluefs devices from bluestore path
{
    "/var/lib/ceph/osd/ceph-0/block": {
        "osd_uuid": "7add3b6c-b8e9-42b0-8dc4-feb1af6e5394",
        "size": 10720641024,
        "btime": "2023-08-25T15:08:38.966496+0000",
        "description": "main",
        "bfm_blocks": "2617344",
        "bfm_blocks_per_key": "128",
        "bfm_bytes_per_block": "4096",
        "bfm_size": "10720641024",
        "bluefs": "1",
        "ceph_fsid": "e0c985eb-b626-4807-9681-b7a836209793",
        "ceph_version_when_created": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
        "created_at": "2023-08-25T15:08:41.150964Z",
        "kv_backend": "rocksdb",
        "magic": "ceph osd volume v026",
        "mkfs_done": "yes",
        "osd_key": "AQDqw+hkvhDHKhAA5r+WYvA++VOyplGBHynBOg==",
        "ready": "ready",
        "require_osd_release": "17",
        "type": "bluestore",
        "whoami": "0"
    }
}


sh-4.4# cat /var/lib/ceph/osd/ceph-0/type
bluestore


-----------------------------------------------------------------------------------

After Upgrade:

sh-4.4$ ceph osd metadata osd.0
{
    "id": 0,
    "arch": "x86_64",
    "back_addr": "[v2:10.244.1.14:6802/2651410559,v1:10.244.1.14:6803/2651410559]",
    "back_iface": "",
    "bluefs": "1",
    "bluefs_dedicated_db": "0",
    "bluefs_dedicated_wal": "0",
    "bluefs_single_shared_device": "1",
    "bluestore_bdev_access_mode": "blk",
    "bluestore_bdev_block_size": "4096",
    "bluestore_bdev_dev_node": "/dev/dm-0",
    "bluestore_bdev_devices": "vdc",
    "bluestore_bdev_driver": "KernelDevice",
    "bluestore_bdev_optimal_io_size": "0",
    "bluestore_bdev_partition_path": "/dev/dm-0",
    "bluestore_bdev_rotational": "1",
    "bluestore_bdev_size": "10720641024",
    "bluestore_bdev_support_discard": "1",
    "bluestore_bdev_type": "hdd",
    "bluestore_min_alloc_size": "4096",
    "ceph_release": "quincy",
    "ceph_version": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
    "ceph_version_short": "17.2.6-373-g5406f4b1",
    "ceph_version_when_created": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
    "container_hostname": "rook-ceph-osd-0-85db7bfdd8-vxq65",
    "container_image": "quay.io/guits/ceph-volume:bs-rdr",
    "cpu": "Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz",
    "created_at": "2023-08-25T15:15:51.148428Z",
    "default_device_class": "hdd",
    "device_ids": "",
    "device_paths": "",
    "devices": "vdc",
    "distro": "centos",
    "distro_description": "CentOS Stream 8",
    "distro_version": "8",
    "front_addr": "[v2:10.244.1.14:6800/2651410559,v1:10.244.1.14:6801/2651410559]",
    "front_iface": "",
    "hb_back_addr": "[v2:10.244.1.14:6806/2651410559,v1:10.244.1.14:6807/2651410559]",
    "hb_front_addr": "[v2:10.244.1.14:6804/2651410559,v1:10.244.1.14:6805/2651410559]",
    "hostname": "minikube-m02",
    "journal_rotational": "1",
    "kernel_description": "#1 SMP Fri Jan 27 18:05:35 UTC 2023",
    "kernel_version": "5.10.57",
    "mem_swap_kb": "0",
    "mem_total_kb": "2165900",
    "network_numa_unknown_ifaces": "back_iface,front_iface",
    "objectstore_numa_unknown_devices": "vdc",
    "os": "Linux",
    "osd_data": "/var/lib/ceph/osd/ceph-0",
    "osd_objectstore": "bluestore",
    "osdspec_affinity": "",
    "pod_name": "rook-ceph-osd-0-85db7bfdd8-vxq65",
    "pod_namespace": "rook-ceph",
    "rotational": "1"
}
sh-4.4$ 

sh-4.4# cat /var/lib/ceph/osd/ceph-0/type
bluestore-rdr

sh-4.4# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0
inferring bluefs devices from bluestore path
{
    "/var/lib/ceph/osd/ceph-0/block": {
        "osd_uuid": "16f9170b-8ef0-4287-96ed-60f745781128",
        "size": 10720641024,
        "btime": "2023-08-25T15:15:49.513276+0000",
        "description": "main",
        "bfm_blocks": "2617344",
        "bfm_blocks_per_key": "128",
        "bfm_bytes_per_block": "4096",
        "bfm_size": "10720641024",
        "bluefs": "1",
        "ceph_fsid": "e0c985eb-b626-4807-9681-b7a836209793",
        "ceph_version_when_created": "ceph version 17.2.6-373-g5406f4b1 (5406f4b1729bf4f476338ad344cc86b126e0c5b3) quincy (stable)",
        "created_at": "2023-08-25T15:15:51.148428Z",
        "kv_backend": "rocksdb",
        "magic": "ceph osd volume v026",
        "mkfs_done": "yes",
        "osd_key": "AQCVxehkJ+3qNhAAHf5zZWPKGZ6XMlm8SI6saw==",
        "ready": "ready",
        "require_osd_release": "17",
        "type": "bluestore-rdr",
        "whoami": "0"
    }
}
sh-4.4#
Comment 20 Red Hat Bugzilla 2023-12-24 04:25:10 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days