.OSDMap checks now ensure that a health warning is reported until the release flag is updated after a cluster upgrade
Previously, after all OSDs were upgraded to a new release, the `require-osd-release` flag in the OSDMap was updated to reflect the new release name. However, the check that verifies this flag against the running version was not updated appropriately to include the 'reef' release, so no cluster warning was raised once the upgrade was completed. As a result, users could mistakenly continue operations, risking catastrophic outcomes including cluster unavailability.
With this fix, the OSDMap check now includes the 'reef' release, ensuring that a health warning is reported until the `require-osd-release` flag is updated to the appropriate release after a cluster upgrade.
Description of problem:
When the upgrade is in progress and the osds are upgraded from Quincy to Reef, the warning message does not shown up in ceph status.
We observed the warning when the cluster was upgraded from :
1. Nautilus to Pacific.
# ceph health detail
HEALTH_WARN all OSDs are running pacific or later but require_osd_release < pacific
[WRN] OSD_UPGRADE_FINISHED: all OSDs are running pacific or later but require_osd_release < pacific
all OSDs are running pacific or later but require_osd_release < pacific
2. Pacific to Quincy.
"OSD_UPGRADE_FINISHED":{"severity":"HEALTH_WARN","summary":{"message":"all OSDs are running quincy or later but require_osd_release < quincy","count":0},"detail":[{"message":"all OSDs are running quincy or later but require_osd_release < quincy"}]
Test-run log : http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-LU2F6M/Upgrade_ceph_cluster_0.log
3. Pacific to Reef.
"OSD_UPGRADE_FINISHED":{"severity":"HEALTH_WARN","summary":{"message":"all OSDs are running quincy or later but require_osd_release < quincy","count":0},"detail":[{"message":"all OSDs are running quincy or later but require_osd_release < quincy"}],"muted":false}
Test-run log : http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-ILSU49/Upgrade_ceph_cluster_0.log
Version-Release number of selected component (if applicable):
ceph version 18.2.0-84.el9cp (4d8b4718f998b40ce8c0995ad6d2b3b3745756ea) reef (stable)
How reproducible:
3/3
Steps to Reproduce:
1. Deploy a RHCS 6.1 cluster
2. Upgrade the cluster to latest nightly 7.0 builds
3. Observe that no warning generated for mismatch b/w the require-osd-release flag during upgrades.
Actual results:
Warning not seen in ceph during upgrade from Quincy to Reef
Expected results:
Warning seen in ceph during upgrade from Quincy to Reef
Additional info:
Feature was introduced with bz : https://bugzilla.redhat.com/show_bug.cgi?id=1988773
Comment 1RHEL Program Management
2023-10-12 03:17:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Red Hat Ceph Storage 7.1 security and bug fix updates), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2025:9335
Description of problem: When the upgrade is in progress and the osds are upgraded from Quincy to Reef, the warning message does not shown up in ceph status. We observed the warning when the cluster was upgraded from : 1. Nautilus to Pacific. # ceph health detail HEALTH_WARN all OSDs are running pacific or later but require_osd_release < pacific [WRN] OSD_UPGRADE_FINISHED: all OSDs are running pacific or later but require_osd_release < pacific all OSDs are running pacific or later but require_osd_release < pacific 2. Pacific to Quincy. "OSD_UPGRADE_FINISHED":{"severity":"HEALTH_WARN","summary":{"message":"all OSDs are running quincy or later but require_osd_release < quincy","count":0},"detail":[{"message":"all OSDs are running quincy or later but require_osd_release < quincy"}] Test-run log : http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-LU2F6M/Upgrade_ceph_cluster_0.log 3. Pacific to Reef. "OSD_UPGRADE_FINISHED":{"severity":"HEALTH_WARN","summary":{"message":"all OSDs are running quincy or later but require_osd_release < quincy","count":0},"detail":[{"message":"all OSDs are running quincy or later but require_osd_release < quincy"}],"muted":false} Test-run log : http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-ILSU49/Upgrade_ceph_cluster_0.log Version-Release number of selected component (if applicable): ceph version 18.2.0-84.el9cp (4d8b4718f998b40ce8c0995ad6d2b3b3745756ea) reef (stable) How reproducible: 3/3 Steps to Reproduce: 1. Deploy a RHCS 6.1 cluster 2. Upgrade the cluster to latest nightly 7.0 builds 3. Observe that no warning generated for mismatch b/w the require-osd-release flag during upgrades. Actual results: Warning not seen in ceph during upgrade from Quincy to Reef Expected results: Warning seen in ceph during upgrade from Quincy to Reef Additional info: Feature was introduced with bz : https://bugzilla.redhat.com/show_bug.cgi?id=1988773