Bug 2310839

Summary:	[8.0 staggered Upgrade] : Upgrade of MGR in staggered approach also started upgrading NVMeoF service
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Sunil Kumar Nagaraju <sunnagar>
Component:	Cephadm	Assignee:	Adam King <adking>
Status:	CLOSED ERRATA	QA Contact:	Mohit Bisht <mobisht>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	8.0	CC:	adking, akraj, cephqe-warriors, rlepaksh, rpollack, tserlin, vereddy
Target Milestone:	---	Flags:	rlepaksh: needinfo-
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-19.1.1-8.el9cp	Doc Type:	Bug Fix
Doc Text:	.NVMe-oF daemons now upgrade after the MON daemons in staggered upgrade scenarios Previously, NVMe-oF depended on the version of the monitor daemon. As a result, in the previous staggered upgrade scenario, the NVMe-oF daemon would be upgraded as soon as the MGR daemons were upgraded. This placed the NVME-oF daemon before the MON daemons and caused the MON daemons to fail to deploy. With this fix, the upgrade of the NVMe-oF daemon is moved to the end of the upgrade order. NVMe-oF daemons now upgrade later in staggered upgrade scenarios, upgrading after the MON daemons and start up as expected.	Story Points:	---
Clone Of:	2278778	Environment:
Last Closed:	2024-11-25 09:09:18 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2278778
Bug Blocks:	2267614, 2298578, 2298579, 2317218

Description Sunil Kumar Nagaraju 2024-09-09 12:46:14 UTC

+++ This bug was initially created as a clone of Bug #2278778 +++


Currently in 8.0 dev build, with MGR upgrade the NVMeoF GW daemons also started getting upgraded to new version i.e 1.3.1-1 and all are working.

But the issue is nvmeof gw upgrade are still not supported in staggered upgrade workflow, which was partially fixed in cloned BZ 2278778, where nvmeof upgrade happned post ceph-exporter.

Whats needed is have this upgrade workflow supported as we get from error comment,

mgr -> mon -> crash -> osd -> mds -> rgw -> rbd-mirror -> cephfs-mirror -> ceph-exporter -> iscsi -> nfs -> nvmeof


So upgrade should support upgrading of NVMeoF gateways independently. 



[ceph: root@ceph-sunilkumar-02-meya5y-node1-installer /]# ceph orch upgrade start --image cp.stg.icr.io/cp/ibm-ceph/ceph-8-rhel9:8-27  --daemon-types nvmeof
Error EINVAL: Cannot start upgrade. Daemons with types earlier in upgrade order than given types need upgrading.
Please first upgrade mon.ceph-sunilkumar-02-meya5y-node3, osd.1, osd.9, osd.8, osd.5, mon.ceph-sunilkumar-02-meya5y-node1-installer, mgr.ceph-sunilkumar-02-meya5y-node2.ciqsnw, mon.ceph-sunilkumar-02-meya5y-node2, osd.4, osd.7, mgr.ceph-sunilkumar-02-meya5y-node1-installer.uhyezu, osd.2, osd.3, osd.0, osd.11, osd.6, osd.10
NOTE: Enforced upgrade order is: mgr -> mon -> crash -> osd -> mds -> rgw -> rbd-mirror -> cephfs-mirror -> ceph-exporter -> iscsi -> nfs -> nvmeof

Comment 10 errata-xmlrpc 2024-11-25 09:09:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:10216