Bug 2358304

Summary: Both Fresh installation and Upgrade ( 8.0z3 --> 8.1) is failing with "PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process"
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: CephadmAssignee: Adam King <adking>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.1CC: afrahman, anbehl, cephqe-warriors, dona545mi, epuertat, jcaratza, kdeb, mobisht, tserlin, vdas
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-19.2.1-113 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-06-26 12:23:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Manisha Saini 2025-04-08 16:32:56 UTC 
Description of problem:
========

Performed upgrade from RHCS 8.0z3 LIVE builds to 8.1 builds ( 19.2.1-112.el9cp).
Upgrade got failed with "Module 'dashboard' has failed: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process" and cluster went into "HEALTH_ERR" state

[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph -s
  cluster:
    id:     63da4d80-1399-11f0-a553-fa163e07341b
    health: HEALTH_ERR
            2 mgr modules have failed dependencies
            Module 'dashboard' has failed: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process

  services:
    mon: 3 daemons, quorum ceph-manisaini-paxep6-node1-installer,ceph-manisaini-paxep6-node2,ceph-manisaini-paxep6-node3 (age 29h)
    mgr: ceph-manisaini-paxep6-node2.yshool(active, since 6s)
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 29h), 18 in (since 29h)
    rgw: 2 daemons active (2 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   9 pools, 721 pgs
    objects: 259 objects, 1.1 MiB
    usage:   1.6 GiB used, 268 GiB / 270 GiB avail
    pgs:     721 active+clean

  io:
    client:   42 KiB/s rd, 0 B/s wr, 19 op/s rd, 4 op/s wr




Version-Release number of selected component (if applicable):
------
19.2.1-112.el9cp

How reproducible:
-----
2/2


Steps to Reproduce:
======
1. Deploy RHCS 8.0z3 cluster
2. Upgrade the cluster to RHCS 8.1 builds.


Actual results:
======
Upgrade failed with "Module 'dashboard' has failed: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process"

Expected results:
=====
Upgrade should pass

Additional info:
=======
Test logs -->

==========
On 8.0z3 builds 
==========

[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph --version
ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph versions
{
    "mon": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 3
    },
    "mgr": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "osd": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 18
    },
    "mds": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "rgw": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "overall": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 27
    }
}
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# rpm -qa | grep nfs
libnfsidmap-2.5.4-27.el9.x86_64
nfs-utils-2.5.4-27.el9.x86_64
nfs-ganesha-selinux-6.5-9.el9cp.noarch
nfs-ganesha-6.5-9.el9cp.x86_64
nfs-ganesha-ceph-6.5-9.el9cp.x86_64
nfs-ganesha-rados-grace-6.5-9.el9cp.x86_64
nfs-ganesha-rados-urls-6.5-9.el9cp.x86_64
nfs-ganesha-rgw-6.5-9.el9cp.x86_64
nfs-ganesha-utils-6.5-9.el9cp.x86_64
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph -s
  cluster:
    id:     63da4d80-1399-11f0-a553-fa163e07341b
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph-manisaini-paxep6-node1-installer,ceph-manisaini-paxep6-node2,ceph-manisaini-paxep6-node3 (age 29h)
    mgr: ceph-manisaini-paxep6-node1-installer.aduzjk(active, since 19h), standbys: ceph-manisaini-paxep6-node2.yshool
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 29h), 18 in (since 29h)
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   9 pools, 721 pgs
    objects: 259 objects, 1.1 MiB
    usage:   1.6 GiB used, 268 GiB / 270 GiB avail
    pgs:     721 active+clean
 
  io:
    client:   170 B/s rd, 0 op/s rd, 0 op/s wr
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph versions
{
    "mon": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 3
    },
    "mgr": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "osd": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 18
    },
    "mds": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "rgw": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 2
    },
    "overall": {
        "ceph version 19.2.0-124.el9cp (11e266251d845c59d6f46fc852124e45e0a772bd) squid (stable)": 27
    }
}
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#
 


==============
Upgrade the cluster to RHCS 8.1 builds 
==============
 
[root@ceph-manisaini-paxep6-node1-installer yum.repos.d]# yum update
Updating Subscription Management repositories.
 
This system has release set to 9.5 and it receives updates only for this release.
 
RHCEPH-8.1-Tools-rpms                                                                                                    88 kB/s | 2.3 kB     00:00
Dependencies resolved.
========================================================================================================================================================
 Package                       Architecture                 Version                                   Repository                                   Size
========================================================================================================================================================
Upgrading:
 cephadm                       noarch                       2:19.2.1-112.el9cp                        RHCEPH-8.1-Tools-rpms                       1.1 M
 
Transaction Summary
========================================================================================================================================================
Upgrade  1 Package
 
Total size: 1.1 M
Is this ok [y/N]: y
Downloading Packages:
[SKIPPED] cephadm-19.2.1-112.el9cp.noarch.rpm: Already downloaded
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                1/1
  Running scriptlet: cephadm-2:19.2.1-112.el9cp.noarch                                                                                              1/2
  Upgrading        : cephadm-2:19.2.1-112.el9cp.noarch                                                                                              1/2
  Cleanup          : cephadm-2:19.2.0-124.el9cp.noarch                                                                                              2/2
  Running scriptlet: cephadm-2:19.2.0-124.el9cp.noarch                                                                                              2/2
  Verifying        : cephadm-2:19.2.1-112.el9cp.noarch                                                                                              1/2
  Verifying        : cephadm-2:19.2.0-124.el9cp.noarch                                                                                              2/2
Installed products updated.
 
Upgraded:
  cephadm-2:19.2.1-112.el9cp.noarch
 
Complete!
[root@ceph-manisaini-paxep6-node1-installer yum.repos.d]#
 
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade check registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.1-rhel-9-containers-candidate-68811-20250408133730
{
    "needs_update": {
        "ceph-exporter.ceph-manisaini-paxep6-node1-installer": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "ceph-exporter.ceph-manisaini-paxep6-node2": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "ceph-exporter.ceph-manisaini-paxep6-node3": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "crash.ceph-manisaini-paxep6-node1-installer": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "crash.ceph-manisaini-paxep6-node2": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "crash.ceph-manisaini-paxep6-node3": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mds.cephfs.ceph-manisaini-paxep6-node2.qyoufi": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mds.cephfs.ceph-manisaini-paxep6-node3.wbojjb": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mgr.ceph-manisaini-paxep6-node1-installer.aduzjk": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph:8-355",
            "current_version": "19.2.0-124.el9cp"
        },
        "mgr.ceph-manisaini-paxep6-node2.yshool": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mon.ceph-manisaini-paxep6-node1-installer": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph:8-355",
            "current_version": "19.2.0-124.el9cp"
        },
        "mon.ceph-manisaini-paxep6-node2": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "mon.ceph-manisaini-paxep6-node3": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "nfs.nfsganesha.0.0.ceph-manisaini-paxep6-node2.tnrrca": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "6.5"
        },
        "nfs.nfsganesha.1.0.ceph-manisaini-paxep6-node3.bkqutd": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "6.5"
        },
        "osd.0": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.1": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.10": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.11": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.12": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.13": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.14": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.15": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.16": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.17": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.2": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.3": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.4": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.5": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.6": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.7": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.8": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "osd.9": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "rgw.rgw.1.ceph-manisaini-paxep6-node2.zvoncq": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        },
        "rgw.rgw.1.ceph-manisaini-paxep6-node3.mkpdoe": {
            "current_id": "2cdb8ab76018b0532889d2ae53ba51742b4ad2cafa30d1a5656379dba24ac948",
            "current_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:1669308564815f995bf62c40c780cd5357f5fb0d426e712ae6477a5ba2a983b0",
            "current_version": "19.2.0-124.el9cp"
        }
    },
    "non_ceph_image_daemons": [
        "node-exporter.ceph-manisaini-paxep6-node1-installer",
        "alertmanager.ceph-manisaini-paxep6-node1-installer",
        "grafana.ceph-manisaini-paxep6-node1-installer",
        "prometheus.ceph-manisaini-paxep6-node1-installer",
        "node-exporter.ceph-manisaini-paxep6-node2",
        "node-exporter.ceph-manisaini-paxep6-node3"
    ],
    "target_digest": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:949b66fba0633336dfc4827148b3c766a1776296371743e111f965217ca32107",
    "target_id": "954799815da998e1bd9a18a2e3377ea2c744ff2e702fbc49e3e0e89b25bf6dca",
    "target_name": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.1-rhel-9-containers-candidate-68811-20250408133730",
    "target_version": "ceph version 19.2.1-112.el9cp (b42a340f4188319285c66f57d1a84ceb92b58e67) squid (stable)",
    "up_to_date": []
}
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade start registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.1-rhel-9-containers-candidate-68811-20250408133730
Initiating upgrade to registry-proxy.engineering.redhat.com/rh-osbs/rhceph:ceph-8.1-rhel-9-containers-candidate-68811-20250408133730
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#
 
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph -s
  cluster:
    id:     63da4d80-1399-11f0-a553-fa163e07341b
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph-manisaini-paxep6-node1-installer,ceph-manisaini-paxep6-node2,ceph-manisaini-paxep6-node3 (age 29h)
    mgr: ceph-manisaini-paxep6-node1-installer.aduzjk(active, since 19h), standbys: ceph-manisaini-paxep6-node2.yshool
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 29h), 18 in (since 29h)
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   9 pools, 721 pgs
    objects: 259 objects, 1.1 MiB
    usage:   1.6 GiB used, 268 GiB / 270 GiB avail
    pgs:     721 active+clean
 
  io:
    client:   341 B/s rd, 0 op/s rd, 0 op/s wr
 
  progress:
    Upgrade to 19.2.1-112.el9cp (3s)
      [............................]
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade status
{
    "target_image": "registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:949b66fba0633336dfc4827148b3c766a1776296371743e111f965217ca32107",
    "in_progress": true,
    "which": "Upgrading all daemon types on all hosts",
    "services_complete": [],
    "progress": "1/41 daemons upgraded",
    "message": "Currently upgrading mgr daemons",
    "is_paused": false
}
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade status
Error ENOENT: Module not found
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph -s
  cluster:
    id:     63da4d80-1399-11f0-a553-fa163e07341b
    health: HEALTH_ERR
            2 mgr modules have failed dependencies
            Module 'dashboard' has failed: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
 
  services:
    mon: 3 daemons, quorum ceph-manisaini-paxep6-node1-installer,ceph-manisaini-paxep6-node2,ceph-manisaini-paxep6-node3 (age 29h)
    mgr: ceph-manisaini-paxep6-node2.yshool(active, since 6s)
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 29h), 18 in (since 29h)
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   9 pools, 721 pgs
    objects: 259 objects, 1.1 MiB
    usage:   1.6 GiB used, 268 GiB / 270 GiB avail
    pgs:     721 active+clean
 
  io:
    client:   42 KiB/s rd, 0 B/s wr, 19 op/s rd, 4 op/s wr
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#
 
[ceph: root@ceph-manisaini-paxep6-node1-installer /]# ceph orch upgrade status
Error ENOENT: Module not found
[ceph: root@ceph-manisaini-paxep6-node1-installer /]#
Comment 16 Dona Mi 2025-04-10 05:30:07 UTC 
This bug is a significant concern for teams relying on Cephadm for their Ceph deployments, and its prompt resolution will be crucial to maintaining the stability and reliability of the system. https://www-ipass.com
Comment 17 Justin Caratzas 2025-04-11 14:02:02 UTC 
moving to ON_QA. QA please mark as verified
Comment 24 errata-xmlrpc 2025-06-26 12:23:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775