Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2401206

Summary: [Tentacle] Ceph cluster bootstrap with RHEL 10 based container images on RHEL 9 host fails with error: “PyO3 modules do not yet support subinterpreters”
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: CephadmAssignee: Adam King <adking>
Status: CLOSED ERRATA QA Contact: Vinayak Papnoi <vpapnoi>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 9.0CC: akane, cephqe-warriors, jcaratza, mobisht, vereddy, vpapnoi
Target Milestone: ---   
Target Release: 9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-20.1.0-42.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2026-01-29 07:00:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Manisha Saini 2025-10-03 09:30:17 UTC 
Description of problem:
======================

While attempting to bootstrap a Ceph cluster using RHEL 10 based Ceph container images on a RHEL 9 host, the bootstrap process fails during the mgr module enable cephadm step - PyO3 modules do not yet support subinterpreters

Note - Since RHEL 10 RPMs are not yet available, the bootstrap was performed on a RHEL 9 host with RHEL 10 based container images.


2025-10-03 05:10:13,126 - cephci - ceph:1615 - INFO - Execute cephadm --image quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 bootstrap --mon-ip 10.0.66.126 on 10.0.66.126
2025-10-03 05:11:38,648 - cephci - ceph:1645 - INFO - Execution of cephadm --image quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 bootstrap --mon-ip 10.0.66.126 on 10.0.66.126 took 85.522004 seconds
2025-10-03 05:11:38,649 - cephci - test_cephadm:189 - ERROR - cephadm --image quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 bootstrap --mon-ip 10.0.66.126 returned RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 -e NODE_NAME=ceph-tentacle-msaini-dv4qog-node1-installer -v /var/log/ceph/c55bdde0-a038-11f0-8b54-fa163ee64ceb:/var/log/ceph:z -v /tmp/ceph-tmpi9odkrl9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu_oagnp5:/etc/ceph/ceph.conf:z quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)


Version-Release number of selected component (if applicable):
=============================================================

OS : RHEL 9.6
Ceph container images - Based on RHEL 10

# ceph --version
ceph version 20.1.0-36.el10cp (d9dc2f18e0595b2f4d5df751bca2c3351f75e73e) tentacle (rc)

# rpm -qa | grep cephadm
cephadm-20.1.0-36.el9cp.noarch

# cat /etc/redhat-release
Red Hat Enterprise Linux release 9.6 (Plow)

How reproducible:
===============
Everytime


Steps to Reproduce:
===================
1. Bootstrap ceph cluster with Tentacle RHEL 10 based container image


Actual results:
==============
Bootstrap of the Ceph cluster fails with RuntimeError during mgr module enable cephadm.


Expected results:
==============
Ceph cluster bootstrap should succeed on RHEL 9 host with RHEL 10 container images.
The cephadm mgr module should enable successfully without hitting PyO3 limitations.


Additional info:
================
Bootstrap failure logs
-------
2025-10-03 05:10:03,401 - cephci - ceph:1615 - INFO - Execute yum -y install cephadm --nogpgcheck on 10.0.67.35
2025-10-03 05:10:04,407 - cephci - ceph:1187 - DEBUG - Updating Subscription Management repositories.
2025-10-03 05:10:06,395 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:06,396 - cephci - ceph:1187 - DEBUG - This system has release set to 9.6 and it receives updates only for this release.
2025-10-03 05:10:06,396 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:07,483 - cephci - ceph:1187 - DEBUG - ceph-Tools                                      267 kB/s |  32 kB     00:00
2025-10-03 05:10:08,891 - cephci - ceph:1187 - DEBUG - Dependencies resolved.
2025-10-03 05:10:08,892 - cephci - ceph:1187 - DEBUG - ================================================================================
2025-10-03 05:10:08,892 - cephci - ceph:1187 - DEBUG -  Package        Architecture  Version                   Repository         Size
2025-10-03 05:10:08,893 - cephci - ceph:1187 - DEBUG - ================================================================================
2025-10-03 05:10:08,893 - cephci - ceph:1187 - DEBUG - Installing:
2025-10-03 05:10:08,893 - cephci - ceph:1187 - DEBUG -  cephadm        noarch        2:20.1.0-36.el9cp         ceph-Tools        1.1 M
2025-10-03 05:10:08,893 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:08,894 - cephci - ceph:1187 - DEBUG - Transaction Summary
2025-10-03 05:10:08,894 - cephci - ceph:1187 - DEBUG - ================================================================================
2025-10-03 05:10:08,894 - cephci - ceph:1187 - DEBUG - Install  1 Package
2025-10-03 05:10:08,894 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:08,894 - cephci - ceph:1187 - DEBUG - Total download size: 1.1 M
2025-10-03 05:10:08,895 - cephci - ceph:1187 - DEBUG - Installed size: 1.0 M
2025-10-03 05:10:08,895 - cephci - ceph:1187 - DEBUG - Downloading Packages:
2025-10-03 05:10:09,001 - cephci - ceph:1187 - DEBUG - cephadm-20.1.0-36.el9cp.noarch.rpm               10 MB/s | 1.1 MB     00:00
2025-10-03 05:10:09,003 - cephci - ceph:1187 - DEBUG - --------------------------------------------------------------------------------
2025-10-03 05:10:09,005 - cephci - ceph:1187 - DEBUG - Total                                           9.7 MB/s | 1.1 MB     00:00
2025-10-03 05:10:09,005 - cephci - ceph:1187 - DEBUG - Running transaction check
2025-10-03 05:10:09,017 - cephci - ceph:1187 - DEBUG - Transaction check succeeded.
2025-10-03 05:10:09,018 - cephci - ceph:1187 - DEBUG - Running transaction test
2025-10-03 05:10:09,034 - cephci - ceph:1187 - DEBUG - Transaction test succeeded.
2025-10-03 05:10:09,035 - cephci - ceph:1187 - DEBUG - Running transaction
2025-10-03 05:10:09,058 - cephci - ceph:1187 - DEBUG -   Preparing        :                                                        1/1
2025-10-03 05:10:09,098 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:09,099 - cephci - ceph:1187 - DEBUG -   Running scriptlet: cephadm-2:20.1.0-36.el9cp.noarch                       1/1
2025-10-03 05:10:09,301 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:09,301 - cephci - ceph:1187 - DEBUG -   Installing       : cephadm-2:20.1.0-36.el9cp.noarch                       1/1
2025-10-03 05:10:09,347 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:09,348 - cephci - ceph:1187 - DEBUG -   Running scriptlet: cephadm-2:20.1.0-36.el9cp.noarch                       1/1
2025-10-03 05:10:09,565 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:09,566 - cephci - ceph:1187 - DEBUG -   Verifying        : cephadm-2:20.1.0-36.el9cp.noarch                       1/1
2025-10-03 05:10:10,375 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:10,375 - cephci - ceph:1187 - DEBUG - Installed products updated.
2025-10-03 05:10:10,881 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:10,881 - cephci - ceph:1187 - DEBUG - Installed:
2025-10-03 05:10:10,881 - cephci - ceph:1187 - DEBUG -   cephadm-2:20.1.0-36.el9cp.noarch
2025-10-03 05:10:10,882 - cephci - ceph:1187 - DEBUG -
2025-10-03 05:10:10,882 - cephci - ceph:1187 - DEBUG - Complete!
2025-10-03 05:10:12,114 - cephci - ceph:1645 - INFO - Execution of yum -y install cephadm --nogpgcheck on 10.0.67.35 took 8.712316 seconds
2025-10-03 05:10:12,117 - cephci - ceph:1615 - INFO - Execute rpm -qa | grep cephadm on 10.0.67.35
2025-10-03 05:10:13,123 - cephci - ceph:1645 - INFO - Execution of rpm -qa | grep cephadm on 10.0.67.35 took 1.005846 seconds
2025-10-03 05:10:13,126 - cephci - ceph:1615 - INFO - Execute cephadm --image quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 bootstrap --mon-ip 10.0.66.126 on 10.0.66.126
2025-10-03 05:11:38,648 - cephci - ceph:1645 - INFO - Execution of cephadm --image quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 bootstrap --mon-ip 10.0.66.126 on 10.0.66.126 took 85.522004 seconds
2025-10-03 05:11:38,649 - cephci - test_cephadm:189 - ERROR - cephadm --image quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 bootstrap --mon-ip 10.0.66.126 returned RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 -e NODE_NAME=ceph-tentacle-msaini-dv4qog-node1-installer -v /var/log/ceph/c55bdde0-a038-11f0-8b54-fa163ee64ceb:/var/log/ceph:z -v /tmp/ceph-tmpi9odkrl9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu_oagnp5:/etc/ceph/ceph.conf:z quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/sbin/cephadm/__main__.py", line 5627, in <module>
  File "/usr/sbin/cephadm/__main__.py", line 5615, in main
  File "/usr/sbin/cephadm/__main__.py", line 2578, in _rollback
  File "/usr/sbin/cephadm/__main__.py", line 441, in _default_image
  File "/usr/sbin/cephadm/__main__.py", line 2839, in command_bootstrap
  File "/usr/sbin/cephadm/__main__.py", line 2317, in enable_cephadm_mgr_module
  File "/usr/sbin/cephadm/__main__.py", line 2757, in cli
  File "/usr/sbin/cephadm/cephadmlib/container_types.py", line 435, in run
  File "/usr/sbin/cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 -e NODE_NAME=ceph-tentacle-msaini-dv4qog-node1-installer -v /var/log/ceph/c55bdde0-a038-11f0-8b54-fa163ee64ceb:/var/log/ceph:z -v /tmp/ceph-tmpi9odkrl9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu_oagnp5:/etc/ceph/ceph.conf:z quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

 and code 1 on 10.0.66.126
Traceback (most recent call last):
  File "/home/msaini/sept_29/cephci/tests/ceph_installer/test_cephadm.py", line 173, in run
    func(cfg)
  File "/home/msaini/sept_29/cephci/ceph/ceph_admin/bootstrap.py", line 335, in bootstrap
    out, err = self.installer.exec_command(
  File "/home/msaini/sept_29/cephci/ceph/ceph.py", line 2342, in exec_command
    return self.node.exec_command(cmd=cmd, **kw)
  File "/home/msaini/sept_29/cephci/ceph/ceph.py", line 1739, in exec_command
    raise CommandFailed(
ceph.ceph.CommandFailed: cephadm --image quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 bootstrap --mon-ip 10.0.66.126 returned RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 -e NODE_NAME=ceph-tentacle-msaini-dv4qog-node1-installer -v /var/log/ceph/c55bdde0-a038-11f0-8b54-fa163ee64ceb:/var/log/ceph:z -v /tmp/ceph-tmpi9odkrl9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu_oagnp5:/etc/ceph/ceph.conf:z quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/sbin/cephadm/__main__.py", line 5627, in <module>
  File "/usr/sbin/cephadm/__main__.py", line 5615, in main
  File "/usr/sbin/cephadm/__main__.py", line 2578, in _rollback
  File "/usr/sbin/cephadm/__main__.py", line 441, in _default_image
  File "/usr/sbin/cephadm/__main__.py", line 2839, in command_bootstrap
  File "/usr/sbin/cephadm/__main__.py", line 2317, in enable_cephadm_mgr_module
  File "/usr/sbin/cephadm/__main__.py", line 2757, in cli
  File "/usr/sbin/cephadm/cephadmlib/container_types.py", line 435, in run
  File "/usr/sbin/cephadm/cephadmlib/call_wrappers.py", line 307, in call_throws
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 -e NODE_NAME=ceph-tentacle-msaini-dv4qog-node1-installer -v /var/log/ceph/c55bdde0-a038-11f0-8b54-fa163ee64ceb:/var/log/ceph:z -v /tmp/ceph-tmpi9odkrl9:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpu_oagnp5:/etc/ceph/ceph.conf:z quay.io/rhceph-ci/rhceph@sha256:bc0295ef0ce1955eb928329eb4afdc5d566513b21d2cd5033a5a4d0349b80c10 mgr module enable cephadm: Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: PyO3 modules do not yet support subinterpreters, see https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

 and code 1 on 10.0.66.126
2025-10-03 05:11:38,660 - cephci - test_cephadm:190 - DEBUG - Gathering cluster state after running test_cephadm
2025-10-03 05:11:38,662 - cephci - ceph:1615 - INFO - Execute cephadm shell -- ceph status on 10.0.66.126
2025-10-03 05:11:40,667 - cephci - ceph:1645 - INFO - Execution of cephadm shell -- ceph status on 10.0.66.126 took 2.004754 seconds
2025-10-03 05:11:40,668 - cephci - ceph:1721 - INFO -
Command:    cephadm shell -- ceph status
Duration:   2.004754 seconds
Exit Code:  1
Stderr:      Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

2025-10-03 05:11:40,668 - cephci - run:904 - ERROR - cephadm shell -- ceph status returned Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Comment 9 errata-xmlrpc 2026-01-29 07:00:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 9.0 Security and Enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2026:1536