Bug 1944527
| Summary: | [RADOS]: Cannot able to configure the OSD's with the model option in the osd_spec.yml | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | skanta |
| Component: | Cephadm | Assignee: | Juan Miguel Olmo <jolmomar> |
| Status: | CLOSED NOTABUG | QA Contact: | skanta |
| Severity: | medium | Docs Contact: | Karen Norteman <knortema> |
| Priority: | unspecified | ||
| Version: | 5.0 | CC: | jolmomar, sewagner, vereddy |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | 5.0 | Flags: | skanta:
needinfo+
skanta: needinfo- |
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-06-30 09:07:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
[ceph: root@magna045 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0 root default [ceph: root@magna045 /]# The spec file used makes impossible to create any osd. The spec file used says that you want to create OSDs in any host with the three different models present, and it seems that you do not have any hosts with that configuration. One important thing to remember is that a bluestore OSD can be splitted up to three different parts (data, db, wal) than can use different storage devices .... but ALL the parts MUST be present in the same host, you cannot distribute the different parts of an OSD between different hosts. Following clarification required for this scenario-
1. The example mentioned in the document as below-
service_type: osd
service_id: osd_spec_default
placement:
host_pattern: '*'
data_devices:
model: HDD-123-foo <- note that HDD-123 would also be valid
db_devices:
model: MC-55-44-XZ <- same here, MC-55-44 is valid
In the above, it is mentioned that "HDD-123" and "MC-55-44" should be valid.
2. What is exactly the model naming pattern need to follow in osd_spec.yml
For example: For a node in the following ways I am getting the model value
[root@depressa008 ubuntu]# lsblk -o NAME,FSTYPE,LABEL,MOUNTPOINT,SIZE,MODEL
NAME FSTYPE LABEL MOUNTPOINT SIZE MODEL
sda 447.1G Micron_5200_MTFD
└─sda1 ext4 / 447.1G
sdb 7T SAMSUNG MZ7LH7T6
sdc 7T SAMSUNG MZ7LH7T6
sdd 7T SAMSUNG MZ7LH7T6
nvme0n1 349.3G INTEL SSDPE21K375GA
nvme1n1 349.3G INTEL SSDPE21K375GA
[root@depressa008 ubuntu]#
With another command-
[root@depressa008 ubuntu]# smartctl -i /dev/sdb
smartctl 7.1 2020-04-05 r5049 [x86_64-linux-4.18.0-221.el8.x86_64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: SAMSUNG MZ7LH7T6HMLA-00005
Serial Number: S487NY0M801863
LU WWN Device Id: 5 002538 e09873fa7
Firmware Version: HXT7404Q
User Capacity: 7,681,501,126,656 bytes [7.68 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Mar 30 04:40:19 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
From which command do I need to take this value?
ceph orch device ls --wide (column model) I will check and update the status with the provided option. [ceph: root@magna045 /]# ceph orch device ls --wide Hostname Path Type Transport RPM Vendor Model Serial Size Health Ident Fault Available Reject Reasons depressa008 /dev/nvme0n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA PHKE91360145375AGN 375G Unknown N/A N/A Yes depressa008 /dev/nvme1n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA PHKE9136002P375AGN 375G Unknown N/A N/A Yes depressa008 /dev/sdb ssd Unknown Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801863 7681G Unknown N/A N/A Yes depressa008 /dev/sdc ssd Unknown Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801906 7681G Unknown N/A N/A Yes depressa008 /dev/sdd ssd Unknown Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801866 7681G Unknown N/A N/A Yes magna045 /dev/sdb hdd Unknown Unknown ATA Hitachi HUA72201 JPW9K0N21ETAME 1000G Unknown N/A N/A Yes magna045 /dev/sdc hdd Unknown Unknown ATA Hitachi HUA72201 JPW9M0N20BWRNE 1000G Unknown N/A N/A Yes magna045 /dev/sdd hdd Unknown Unknown ATA Hitachi HUA72201 JPW9M0N20BT1PE 1000G Unknown N/A N/A Yes magna046 /dev/sdb hdd Unknown Unknown ATA Hitachi HUA72201 JPW9K0N20D1NYE 1000G Unknown N/A N/A Yes magna046 /dev/sdc hdd Unknown Unknown ATA Hitachi HUA72201 JPW9K0N20D2H6E 1000G Unknown N/A N/A Yes magna046 /dev/sdd hdd Unknown Unknown ATA Hitachi HUA72201 JPW9K0N20D1U0E 1000G Unknown N/A N/A Yes magna047 /dev/sdb hdd Unknown Unknown ATA Hitachi HUA72201 JPW9K0N21ETAEE 1000G Unknown N/A N/A Yes magna047 /dev/sdc hdd Unknown Unknown ATA Hitachi HUA72201 JPW9K0N20D1NME 1000G Unknown N/A N/A Yes magna047 /dev/sdd hdd Unknown Unknown ATA Hitachi HUA72201 JPW9K0N20D1N7E 1000G Unknown N/A N/A Yes [ceph: root@magna045 /]# osd_spec.yml file [ceph: root@magna045 /]# cat osd_spec.yml service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: model: Hitachi HUA72201 db_devices: model: SAMSUNG MZ7LH7T6 [ceph: root@magna045 /]# output: [ceph: root@magna045 /]# ceph orch apply osd -i osd_spec.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+-------------+ +---------+------+--------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------------------+----------+----------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------------------+----------+----------+----+-----+ |osd |osd_spec_default |magna045 |/dev/sdb |- |- | |osd |osd_spec_default |magna045 |/dev/sdc |- |- | |osd |osd_spec_default |magna045 |/dev/sdd |- |- | |osd |osd_spec_default |magna046 |/dev/sdb |- |- | |osd |osd_spec_default |magna046 |/dev/sdc |- |- | |osd |osd_spec_default |magna046 |/dev/sdd |- |- | |osd |osd_spec_default |magna047 |/dev/sdb |- |- | |osd |osd_spec_default |magna047 |/dev/sdc |- |- | |osd |osd_spec_default |magna047 |/dev/sdd |- |- | +---------+------------------+----------+----------+----+-----+ [ceph: root@magna045 /]# The model "SAMSUNG MZ7LH7T6" not included in the OSD's or DB [ceph: root@magna045 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 8.18729 root default -7 2.72910 host magna045 1 hdd 0.90970 osd.1 up 1.00000 1.00000 4 hdd 0.90970 osd.4 up 1.00000 1.00000 8 hdd 0.90970 osd.8 up 1.00000 1.00000 -5 2.72910 host magna046 0 hdd 0.90970 osd.0 up 1.00000 1.00000 3 hdd 0.90970 osd.3 up 1.00000 1.00000 6 hdd 0.90970 osd.6 up 1.00000 1.00000 -3 2.72910 host magna047 2 hdd 0.90970 osd.2 up 1.00000 1.00000 5 hdd 0.90970 osd.5 up 1.00000 1.00000 7 hdd 0.90970 osd.7 up 1.00000 1.00000 [ceph: root@magna045 /]# The spec file used produces the expected result. You do not have any host with both "Hitachi HUA72201" and "SAMSUNG MZ7LH7T6" drives installed. So it is no possible to create osds with non-collocated components. Remember: In order to create an OSD with non-collocated components in the same host, you need to specify the different devices to use, and all these devices must be present in the host. The result you obtain is the only thing possible with the spec file provided, that is to create collocated osds in any hosts where we have "Hitachi HUA72201" installed. Checked with collocated components and working as expected- Step1: [ceph: root@depressa009 ~]# ceph orch device ls --wide Hostname Path Type Transport RPM Vendor Model Serial Size Health Ident Fault Available Reject Reasons depressa009 /dev/nvme0n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa009 /dev/nvme1n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa009 /dev/sdb ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801907 7681G Good N/A N/A Yes depressa009 /dev/sdc ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801785 7681G Good N/A N/A Yes depressa009 /dev/sdd ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801134 7681G Good N/A N/A Yes depressa010 /dev/nvme0n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa010 /dev/nvme1n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa010 /dev/sdb ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801864 7681G Good N/A N/A Yes depressa010 /dev/sdc ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801915 7681G Good N/A N/A Yes depressa011 /dev/nvme0n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa011 /dev/nvme1n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa011 /dev/sdb ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0MA00359 7681G Good N/A N/A Yes depressa011 /dev/sdc ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0MA00313 7681G Good N/A N/A Yes depressa011 /dev/sdd ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0MA00414 7681G Good N/A N/A Yes [ceph: root@depressa009 ~]# step2: [ceph: root@depressa009 ~]# cat osd_spec.yml service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: model: SAMSUNG MZ7LH7T6 db_devices: model: INTEL SSDPE21K375GA step3: [ceph: root@depressa009 ~]# ceph orch apply osd -i osd_spec.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+-------------+ +---------+------+--------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------------------+-------------+----------+--------------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------------------+-------------+----------+--------------+-----+ |osd |osd_spec_default |depressa009 |/dev/sdb |/dev/nvme1n1 |- | |osd |osd_spec_default |depressa009 |/dev/sdc |/dev/nvme0n1 |- | |osd |osd_spec_default |depressa009 |/dev/sdd |/dev/nvme0n1 |- | |osd |osd_spec_default |depressa010 |/dev/sdb |/dev/nvme1n1 |- | |osd |osd_spec_default |depressa010 |/dev/sdc |/dev/nvme0n1 |- | |osd |osd_spec_default |depressa011 |/dev/sdb |/dev/nvme1n1 |- | |osd |osd_spec_default |depressa011 |/dev/sdc |/dev/nvme0n1 |- | |osd |osd_spec_default |depressa011 |/dev/sdd |/dev/nvme0n1 |- | +---------+------------------+-------------+----------+--------------+-----+ [ceph: root@depressa009 ~]# I wrongly mentioned the Vendor related bug to the model parameter. Hence removed it from the dependent bug. I will update the status by checking the non-collocated scenario. @jolmomar - Thanks for clarifying. Verified the scenario and details mentiond below- [ceph: root@magna048 /]# ceph orch device ls --wide Hostname Path Type Transport RPM Vendor Model Serial Size Health Ident Fault Available Reject Reasons depressa010 /dev/nvme0n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa010 /dev/nvme1n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa010 /dev/sdb ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801864 7681G Good N/A N/A Yes depressa010 /dev/sdc ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0M801915 7681G Good N/A N/A Yes depressa011 /dev/nvme0n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa011 /dev/nvme1n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa011 /dev/sdb ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0MA00359 7681G Good N/A N/A Yes depressa011 /dev/sdc ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0MA00313 7681G Good N/A N/A Yes depressa011 /dev/sdd ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0MA00414 7681G Good N/A N/A Yes depressa012 /dev/nvme0n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa012 /dev/nvme1n1 ssd Unknown Unknown N/A INTEL SSDPE21K375GA Unknown 375G Unknown N/A N/A Yes depressa012 /dev/sdb ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0MA00311 7681G Good N/A N/A Yes depressa012 /dev/sdc ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0MA00419 7681G Good N/A N/A Yes depressa012 /dev/sdd ssd ATA/SATA Unknown ATA SAMSUNG MZ7LH7T6 S487NY0MA00420 7681G Good N/A N/A Yes magna048 /dev/sdb hdd ATA/SATA 7200 ATA Hitachi HUA72201 JPW9K0N21EGGHE 1000G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked magna048 /dev/sdc hdd ATA/SATA 7200 ATA Hitachi HUA72201 JPW9K0N20BX7DE 1000G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked magna048 /dev/sdd hdd ATA/SATA 7200 ATA Hitachi HUA72201 JPW9M0N20D0Z6E 1000G Good N/A N/A No Insufficient space (<10 extents) on vgs, LVM detected, locked magna049 /dev/sdb hdd ATA/SATA 7200 ATA Hitachi HUA72201 JPW9J0N20A9P0C 1000G Good N/A N/A Yes magna049 /dev/sdc hdd ATA/SATA 7200 ATA Hitachi HUA72201 JPW9M0N20BSWDE 1000G Good N/A N/A Yes magna049 /dev/sdd hdd ATA/SATA 7200 ATA Hitachi HUA72201 JPW9M0N20BNNYE 1000G Good N/A N/A Yes magna050 /dev/sdb hdd ATA/SATA 7200 ATA Hitachi HUA72201 JPW9K0N20D2HNE 1000G Good N/A N/A Yes magna050 /dev/sdc hdd ATA/SATA 7200 ATA Hitachi HUA72201 JPW9K0N20D1N8E 1000G Good N/A N/A Yes magna050 /dev/sdd hdd ATA/SATA 7200 ATA Hitachi HUA72201 JPW9K0N20D0ZLE 1000G Good N/A N/A Yes [ceph: root@magna048 /]# ====================================================================================================================================================================================================================== [ceph: root@magna048 /]# cat osd_spec.yml service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: model: SAMSUNG MZ7LH7T6 db_devices: model: INTEL SSDPE21K375GA [ceph: root@magna048 /]# ========================================================================================================================================================================================================================= [ceph: root@magna048 /]# ceph orch apply osd -i osd_spec.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+-------------+ +---------+------+--------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------------------+-------------+----------+--------------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------------------+-------------+----------+--------------+-----+ |osd |osd_spec_default |depressa010 |/dev/sdb |/dev/nvme1n1 |- | |osd |osd_spec_default |depressa010 |/dev/sdc |/dev/nvme0n1 |- | |osd |osd_spec_default |depressa011 |/dev/sdb |/dev/nvme1n1 |- | |osd |osd_spec_default |depressa011 |/dev/sdc |/dev/nvme0n1 |- | |osd |osd_spec_default |depressa011 |/dev/sdd |/dev/nvme0n1 |- | |osd |osd_spec_default |depressa012 |/dev/sdb |/dev/nvme1n1 |- | |osd |osd_spec_default |depressa012 |/dev/sdc |/dev/nvme0n1 |- | |osd |osd_spec_default |depressa012 |/dev/sdd |/dev/nvme0n1 |- | +---------+------------------+-------------+----------+--------------+-----+ [ceph: root@magna048 /]# verified the bug as mentioned in Comment #2 and comment #7. |
Description of problem: Cannot able to configure the OSD's with the model option in the OSD_spec file. Version-Release number of selected component (if applicable):ceph version 16.1.0-1323.el8cp (46ac37397f0332c20aceceb8022a1ac1ddf8fa73) pacific (rc) How reproducible: Steps to Reproduce: 1.Use cephadm and configure a cluster 2.Add the OSD's by using the osd_spec.yml file [ceph: root@magna045 /]# cat osd_spec.yml service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: model: Hitachi HUA72201-XZ db_devices: model: SAMSUNG MZ7LH7T6-foo wal_devices: model: INTEL SSDPE21K375GA-test [ceph: root@magna045 /]# To get the model name use the following- [root@depressa008 ubuntu]# lsblk -o NAME,FSTYPE,LABEL,MOUNTPOINT,SIZE,MODEL NAME FSTYPE LABEL MOUNTPOINT SIZE MODEL sda 447.1G Micron_5200_MTFD └─sda1 ext4 / 447.1G sdb 7T SAMSUNG MZ7LH7T6 sdc 7T SAMSUNG MZ7LH7T6 sdd 7T SAMSUNG MZ7LH7T6 nvme0n1 349.3G INTEL SSDPE21K375GA nvme1n1 349.3G INTEL SSDPE21K375GA [root@depressa008 ubuntu]# [root@magna046 ubuntu]# lsblk -o NAME,FSTYPE,LABEL,MOUNTPOINT,SIZE,MODEL NAME FSTYPE LABEL MOUNTPOINT SIZE MODEL sda 931.5G Hitachi HUA72201 └─sda1 ext4 / 931.5G sdb 931.5G Hitachi HUA72201 sdc 931.5G Hitachi HUA72201 sdd 931.5G Hitachi HUA72201 [root@magna046 ubuntu]# 3.ceph orch apply osd -i osd_spec.yml Actual results: OSD's are not configured Expected results: OSD's should configure as per mentioned details in the osd_spec.yml Additional info: Reference DOC- https://docs.ceph.com/en/latest/cephadm/osd/ With the following command also we can get the model details. [ceph: root@magna045 /]# ceph device ls DEVICE HOST:DEV DAEMONS WEAR LIFE EXPECTANCY Hitachi_HUA72201_JPW9K0N20B9WEE magna046:sda mon.magna046 Hitachi_HUA72201_JPW9K0N20D12EE magna047:sda mon.magna047 Hitachi_HUA72201_JPW9M0N20BUZHE magna045:sda mon.magna045 [ceph: root@magna045 /]# [root@depressa008 ubuntu]# smartctl -i /dev/sdb smartctl 7.1 2020-04-05 r5049 [x86_64-linux-4.18.0-221.el8.x86_64] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Samsung based SSDs Device Model: SAMSUNG MZ7LH7T6HMLA-00005 Serial Number: S487NY0M801863 LU WWN Device Id: 5 002538 e09873fa7 Firmware Version: HXT7404Q User Capacity: 7,681,501,126,656 bytes [7.68 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Mar 30 04:40:19 2021 UTC SMART support is: Available - device has SMART capability. SMART support is: Enabled