2358304 – Both Fresh installation and Upgrade ( 8.0z3 --> 8.1) is failing with "PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process"

Bug 2358304 - Both Fresh installation and Upgrade ( 8.0z3 --> 8.1) is failing with "PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process"

Summary: Both Fresh installation and Upgrade ( 8.0z3 --> 8.1) is failing with "PyO3 mo...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	Cephadm
Sub Component:
Version:	8.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	8.1
Assignee:	Adam King
QA Contact:	Manisha Saini
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2025-04-08 16:32 UTC by Manisha Saini
Modified:	2025-06-26 12:23 UTC (History)
CC List:	10 users (show)
Fixed In Version:	ceph-19.2.1-113
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2025-06-26 12:23:11 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Ceph Project Bug Tracker	64213	None	None	None	2025-04-09 05:43:46 UTC
Github	ceph ceph pull 61737	None	open	pybind/mgr: Hack around the 'ImportError: PyO3 modules may only be in…	2025-04-09 05:43:46 UTC
Red Hat Issue Tracker	RHCEPH-11089	None	None	None	2025-04-08 16:34:14 UTC
Red Hat Product Errata	RHSA-2025:9775	None	None	None	2025-06-26 12:23:14 UTC

Description Manisha Saini 2025-04-08 16:32:56 UTC

Description of problem:
========

Performed upgrade from RHCS 8.0z3 LIVE builds to 8.1 builds ( 19.2.1-112.el9cp).
Upgrade got failed with "Module 'dashboard' has failed: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process" and cluster went into "HEALTH_ERR" state

[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph -s
  cluster:
    id:     63da4d80-1399-11f0-a553-fa163e07341b
    health: HEALTH_ERR
            2 mgr modules have failed dependencies
            Module 'dashboard' has failed: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process

  services:
    mon: 3 daemons, quorum ceph-manisaini-paxep6-node1-installer,ceph-manisaini-paxep6-node2,ceph-manisaini-paxep6-node3 (age 29h)
    mgr: ceph-manisaini-paxep6-node2.yshool(active, since 6s)
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 29h), 18 in (since 29h)
    rgw: 2 daemons active (2 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   9 pools, 721 pgs
    objects: 259 objects, 1.1 MiB
    usage:   1.6 GiB used, 268 GiB / 270 GiB avail
    pgs:     721 active+clean

  io:
    client:   42 KiB/s rd, 0 B/s wr, 19 op/s rd, 4 op/s wr




Version-Release number of selected component (if applicable):
------
19.2.1-112.el9cp

How reproducible:
-----
2/2


Steps to Reproduce:
======
1. Deploy RHCS 8.0z3 cluster
2. Upgrade the cluster to RHCS 8.1 builds.


Actual results:
======
Upgrade failed with "Module 'dashboard' has failed: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process"

Expected results:
=====
Upgrade should pass

Additional info:
=======
Test logs -->

==========
On 8.0z3 builds 
==========

[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph --version
ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph versions
{
    "mon": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 3
    },
    "mgr": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "osd": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 18
    },
    "mds": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "rgw": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "overall": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 27
    }
}
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# rpm -qa | grep nfs
libnfsidmap-2.5.4-27.el9.x86_64
nfs-utils-2.5.4-27.el9.x86_64
nfs-ganesha-selinux-6.5-9.el9cp.noarch
nfs-ganesha-6.5-9.el9cp.x86_64
nfs-ganesha-ceph-6.5-9.el9cp.x86_64
nfs-ganesha-rados-grace-6.5-9.el9cp.x86_64
nfs-ganesha-rados-urls-6.5-9.el9cp.x86_64
nfs-ganesha-rgw-6.5-9.el9cp.x86_64
nfs-ganesha-utils-6.5-9.el9cp.x86_64
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph -s
  cluster:
    id:     63da4d80-1399-11f0-a553-fa163e07341b
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph-manisaini-paxep6-node1-installer,ceph-manisaini-paxep6-node2,ceph-manisaini-paxep6-node3 (age 29h)
    mgr: ceph-manisaini-paxep6-node1-installer.aduzjk(active, since 19h), standbys: ceph-manisaini-paxep6-node2.yshool
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 29h), 18 in (since 29h)
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   9 pools, 721 pgs
    objects: 259 objects, 1.1 MiB
    usage:   1.6 GiB used, 268 GiB / 270 GiB avail
    pgs:     721 active+clean
 
  io:
    client:   170 B/s rd, 0 op/s rd, 0 op/s wr
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph versions
{
    "mon": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 3
    },
    "mgr": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "osd": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 18
    },
    "mds": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "rgw": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "overall": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 27
    }
}
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#
 


==============
Upgrade the cluster to RHCS 8.1 builds 
==============
 
[root@ceph-manisaini-paxep6-node1-installer yum.repos.d]# yum update
Updating Subscription Management repositories.
 
This system has release set to 9.5 and it receives updates only for this release.
 
RHCEPH-8.1-Tools-rpms                                                                                                    88 kB/s | 2.3 kB     00:00
Dependencies resolved.
========================================================================================================================================================
 Package                       Architecture                 Version                                   Repository                                   Size
========================================================================================================================================================
Upgrading:
 cephadm                       noarch                       2:19.2.1-112.el9cp                        RHCEPH-8.1-Tools-rpms                       1.1 M
 
Transaction Summary
========================================================================================================================================================
Upgrade  1 Package
 
Total size: 1.1 M
Is this ok [y/N]: y
Downloading Packages:
[SKIPPED] cephadm-19.2.1-112.el9cp.noarch.rpm: Already downloaded
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                1/1
  Running scriptlet: cephadm-2:19.2.1-112.el9cp.noarch                                                                                              1/2
  Upgrading        : cephadm-2:19.2.1-112.el9cp.noarch                                                                                              1/2
  Cleanup          : cephadm-2:19.2.0-124.el9cp.noarch                                                                                              2/2
  Running scriptlet: cephadm-2:19.2.0-124.el9cp.noarch                                                                                              2/2
  Verifying        : cephadm-2:19.2.1-112.el9cp.noarch                                                                                              1/2
  Verifying        : cephadm-2:19.2.0-124.el9cp.noarch                                                                                              2/2
Installed products updated.
 
Upgraded:
  cephadm-2:19.2.1-112.el9cp.noarch
 
Complete!
[root@ceph-manisaini-paxep6-node1-installer yum.repos.d]#
 
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade check registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.1-rhel-9-containers-candidate-68811-20250408133730
{
    "needs_update": {
        "ceph-exporter.ceph-manisaini-paxep6-node1-installer": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "ceph-exporter.ceph-manisaini-paxep6-node2": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "ceph-exporter.ceph-manisaini-paxep6-node3": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "crash.ceph-manisaini-paxep6-node1-installer": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "crash.ceph-manisaini-paxep6-node2": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "crash.ceph-manisaini-paxep6-node3": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mds.cephfs.ceph-manisaini-paxep6-node2.qyoufi": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mds.cephfs.ceph-manisaini-paxep6-node3.wbojjb": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mgr.ceph-manisaini-paxep6-node1-installer.aduzjk": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph:8-355",
            "current_version": "19.2.0-124.el9cp"
        },
        "mgr.ceph-manisaini-paxep6-node2.yshool": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mon.ceph-manisaini-paxep6-node1-installer": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph:8-355",
            "current_version": "19.2.0-124.el9cp"
        },
        "mon.ceph-manisaini-paxep6-node2": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mon.ceph-manisaini-paxep6-node3": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "nfs.nfsganesha.0.0.ceph-manisaini-paxep6-node2.tnrrca": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "6.5"
        },
        "nfs.nfsganesha.1.0.ceph-manisaini-paxep6-node3.bkqutd": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "6.5"
        },
        "osd.0": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.1": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.10": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.11": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.12": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.13": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.14": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.15": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.16": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.17": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.2": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.3": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.4": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.5": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.6": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.7": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.8": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.9": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "rgw.rgw.1.ceph-manisaini-paxep6-node2.zvoncq": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "rgw.rgw.1.ceph-manisaini-paxep6-node3.mkpdoe": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        }
    },
    "non_ceph_image_daemons": [
        "node-exporter.ceph-manisaini-paxep6-node1-installer",
        "alertmanager.ceph-manisaini-paxep6-node1-installer",
        "grafana.ceph-manisaini-paxep6-node1-installer",
        "prometheus.ceph-manisaini-paxep6-node1-installer",
        "node-exporter.ceph-manisaini-paxep6-node2",
        "node-exporter.ceph-manisaini-paxep6-node3"
    ],
    "target_digest": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:949b66fba0633336dfc4827148b3c766a1776296371743e111f965217ca32107",
    "target_id": "954799815da998e1bd9a18a2e3377ea2c744ff2e702fbc49e3e0e89b25bf6dca",
    "target_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.1-rhel-9-containers-candidate-68811-20250408133730",
    "target_version": "ceph version 19.2.1-112.el9cp (b42a340f4188319285c66f57d1a84ceb92b58e67) squid (stable)",
    "up_to_date": []
}
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade start registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.1-rhel-9-containers-candidate-68811-20250408133730
Initiating upgrade to registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.1-rhel-9-containers-candidate-68811-20250408133730
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#
 
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph -s
  cluster:
    id:     63da4d80-1399-11f0-a553-fa163e07341b
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph-manisaini-paxep6-node1-installer,ceph-manisaini-paxep6-node2,ceph-manisaini-paxep6-node3 (age 29h)
    mgr: ceph-manisaini-paxep6-node1-installer.aduzjk(active, since 19h), standbys: ceph-manisaini-paxep6-node2.yshool
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 29h), 18 in (since 29h)
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   9 pools, 721 pgs
    objects: 259 objects, 1.1 MiB
    usage:   1.6 GiB used, 268 GiB / 270 GiB avail
    pgs:     721 active+clean
 
  io:
    client:   341 B/s rd, 0 op/s rd, 0 op/s wr
 
  progress:
    Upgrade to 19.2.1-112.el9cp (3s)
      [............................]
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade status
{
    "target_image": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:949b66fba0633336dfc4827148b3c766a1776296371743e111f965217ca32107",
    "in_progress": true,
    "which": "Upgrading all daemon types on all hosts",
    "services_complete": [],
    "progress": "1/41 daemons upgraded",
    "message": "Currently upgrading mgr daemons",
    "is_paused": false
}
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade status
Error ENOENT: Module not found
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph -s
  cluster:
    id:     63da4d80-1399-11f0-a553-fa163e07341b
    health: HEALTH_ERR
            2 mgr modules have failed dependencies
            Module 'dashboard' has failed: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
 
  services:
    mon: 3 daemons, quorum ceph-manisaini-paxep6-node1-installer,ceph-manisaini-paxep6-node2,ceph-manisaini-paxep6-node3 (age 29h)
    mgr: ceph-manisaini-paxep6-node2.yshool(active, since 6s)
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 29h), 18 in (since 29h)
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   9 pools, 721 pgs
    objects: 259 objects, 1.1 MiB
    usage:   1.6 GiB used, 268 GiB / 270 GiB avail
    pgs:     721 active+clean
 
  io:
    client:   42 KiB/s rd, 0 B/s wr, 19 op/s rd, 4 op/s wr
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade status
Error ENOENT: Module not found
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#

Comment 16 Dona Mi 2025-04-10 05:30:07 UTC

This bug is a significant concern for teams relying on Cephadm for their Ceph deployments, and its prompt resolution will be crucial to maintaining the stability and reliability of the system. https://www-ipass.com

Comment 17 Justin Caratzas 2025-04-11 14:02:02 UTC

moving to ON_QA. QA please mark as verified

Comment 24 errata-xmlrpc 2025-06-26 12:23:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775

Note You need to log in before you can comment on or make changes to this bug.