+++ This bug was initially created as a clone of Bug #2169767 +++ +++ This bug was initially created as a clone of Bug #2162781 +++ Description of problem: ----------------------- RHCS 4.3z1 installation fails on RHEL 8.7 at TASK [ceph-mgr : wait for all mgr to be up]. The mgr services fails to start because of below error: ``` Jan 19 03:12:25 ceph50 systemd[1]: Started Ceph Manager. Jan 19 03:12:25 ceph50 ceph-mgr-ceph50[677680]: find: '/var/lib/ceph/mgr/ceph-ceph50/keyring': Permission denied Jan 19 03:12:25 ceph50 ceph-mgr-ceph50[677680]: chown: cannot access '/var/lib/ceph/mgr/ceph-ceph50/keyring': Permission denied Jan 19 03:12:26 ceph50 systemd[1]: ceph-mgr: Main process exited, code=exited, status=1/FAILURE Jan 19 03:12:26 ceph50 systemd[1]: ceph-mgr: Failed with result 'exit-code'. Jan 19 03:12:36 ceph50 systemd[1]: ceph-mgr: Service RestartSec=10s expired, scheduling restart. Jan 19 03:12:36 ceph50 systemd[1]: ceph-mgr: Scheduled restart job, restart counter is at 5198. Jan 19 03:12:36 ceph50 systemd[1]: Stopped Ceph Manager. ``` Version-Release number of selected component (if applicable): -------------------------------------------------------------- RHCS : 4.3z1 ceph-ansible : 4.0.70.18-1.el8cp.noarch RHEL : 8.7 Podman : 4.2 How reproducible: ----------------- Every time Steps to Reproduce: Deploy fresh ceph cluster on 4.3z1 on top of RHEL 8.7. Actual results: --------------- ceph mgr fails to start service because of selinux label and hence the deployment fails. Expected results: ----------------- ceph mgr should start service and the deployment should succeed. --- Additional comment from Lijo Stephen Thomas on 2023-01-20 21:04:10 UTC --- Additional info: ---------------- - RHCS 4.3z1 installation does succeed on RHEL version 8.6 but fails on RHEL 8.7. - Upon checking further I see the selinux label for mgr keyring file is set different on RHEL 8.7: ----> On RHEL 8.7 nodes ``` [root@ceph50 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.7 (Ootpa) [root@ceph50 ~]# podman version Client: Podman Engine Version: 4.2.0 API Version: 4.2.0 Go Version: go1.18.4 Built: Mon Dec 12 17:11:56 2022 OS/Arch: linux/amd64 [root@ceph50 ~]# ll -lZ /var/lib/ceph/mgr/ceph-ceph50/ total 4 -rw-------. 1 167 167 system_u:object_r:var_lib_t:s0 137 Jan 20 07:53 keyring ``` ----> On RHEL 8.6 nodes: ``` [root@ceph80 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.6 (Ootpa) [root@ceph80 ~]# podman version Client: Podman Engine Version: 4.1.1 API Version: 4.1.1 Go Version: go1.17.7 Built: Wed Oct 12 19:12:59 2022 OS/Arch: linux/amd64 [root@ceph80 ~]# ll -lZ /var/lib/ceph/mgr/ceph-ceph80/keyring -rw-------. 1 167 167 system_u:object_r:container_file_t:s0 137 Jan 21 00:59 /var/lib/ceph/mgr/ceph-ceph80/keyring ``` - The correct selinux label should be `container_file_t`. Discussed this issue with Guillaume++ and we believe this is due to the change in podman version that has resulted in the failure. Regards, Lijo --- Additional comment from Lijo Stephen Thomas on 2023-01-20 21:07:44 UTC --- Lab details: [RHCS 4.3z1 + RHEL 8.7] 10.74.250.68 ceph50 10.74.252.92 ceph51 10.74.250.138 ceph52 10.74.250.26 cephadmin1 --> ansible_user=root| pass=redhat123 Logs: /root/ansible/ansible.log --- Additional comment from Teoman ONAY on 2023-02-02 12:58:13 UTC --- So far it does not seem related to the podman version. I have a setup running Centos Stream 8 + podman 4.2.0 and the relabeling of the keyring file to container_file_t when mgr starts works as expected. I still need to figure out why the relabeling -v /var/lib/ceph:/var/lib/ceph:z does not work as expected. I am continuing my investigation and will update the BZ asap --- Additional comment from Manisha Saini on 2023-02-02 12:59:55 UTC --- Hi, we tested this deployment RHCS 4.3z1 with RHEL 8.7, hitting same issue as reported in this BZ [root@ceph-mobisht-4-3z1-aobcda-node1-installer ceph]# rpm -qa | grep ansible ceph-ansible-4.0.70.18-1.el8cp.noarch ansible-2.9.27-1.el8ae.noarch [root@ceph-mobisht-4-3z1-aobcda-node1-installer ceph]# rpm -qa | grep podman podman-4.2.0-6.module+el8.7.0+17498+a7f63b89.x86_64 podman-catatonit-4.2.0-6.module+el8.7.0+17498+a7f63b89.x86_64 [root@ceph-mobisht-4-3z1-aobcda-node1-installer ceph]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.7 (Ootpa) [root@ceph-mobisht-4-3z1-aobcda-node1-installer ceph]# podman version Client: Podman Engine Version: 4.2.0 API Version: 4.2.0 Go Version: go1.18.4 Built: Mon Dec 12 06:41:56 2022 OS/Arch: linux/amd64 Deployment failed with below error ============ Feb 02 07:22:06 ceph-mobisht-4-3z1-aobcda-node1-installer ceph-mgr-ceph-mobisht-4-3z1-aobcda-node1-installer[20023]: find: '/var/lib/ceph/mgr/ceph-ceph-mobisht-4-3z1-aobcda-node1-installer/keyring': Permission > Feb 02 07:22:06 ceph-mobisht-4-3z1-aobcda-node1-installer ceph-mgr-ceph-mobisht-4-3z1-aobcda-node1-installer[20023]: chown: cannot access '/var/lib/ceph/mgr/ceph-ceph-mobisht-4-3z1-aobcda-node1-installer/keyrin> Feb 02 07:22:06 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: ceph-mgr: Main process exited, code=exited, status=1/FAILURE Feb 02 07:22:06 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: ceph-mgr: Failed with result 'exit-code'. Feb 02 07:22:16 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: ceph-mgr: Service RestartSec=10s expired, scheduling restart. Feb 02 07:22:16 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: ceph-mgr: Scheduled restart job, restart counter is at 3. Feb 02 07:22:16 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: Stopped Ceph Manager. Feb 02 07:22:16 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: Starting Ceph Manager... Feb 02 07:22:16 ceph-mobisht-4-3z1-aobcda-node1-installer podman[20266]: Error: no container with name or ID "ceph-mgr-ceph-mobisht-4-3z1-aobcda-node1-installer" found: no such container Feb 02 07:22:16 ceph-mobisht-4-3z1-aobcda-node1-installer podman[20277]: Error: no container with name or ID "ceph-mgr-ceph-mobisht-4-3z1-aobcda-node1-installer" found: no such container Feb 02 07:22:16 ceph-mobisht-4-3z1-aobcda-node1-installer podman[20286]: Feb 02 07:22:16 ceph-mobisht-4-3z1-aobcda-node1-installer podman[20286]: a4fd9d117ca54e869ecc5cb4b9a42290d4a6d51ba38348bee186dba16edc3c08 Feb 02 07:22:16 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: Started Ceph Manager. Feb 02 07:22:17 ceph-mobisht-4-3z1-aobcda-node1-installer ceph-mgr-ceph-mobisht-4-3z1-aobcda-node1-installer[20296]: find: '/var/lib/ceph/mgr/ceph-ceph-mobisht-4-3z1-aobcda-node1-installer/keyring': Permission > Feb 02 07:22:17 ceph-mobisht-4-3z1-aobcda-node1-installer ceph-mgr-ceph-mobisht-4-3z1-aobcda-node1-installer[20296]: chown: cannot access '/var/lib/ceph/mgr/ceph-ceph-mobisht-4-3z1-aobcda-node1-installer/keyrin> Feb 02 07:22:17 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: ceph-mgr: Main process exited, code=exited, status=1/FAILURE Feb 02 07:22:17 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: ceph-mgr: Failed with result 'exit-code'. Feb 02 07:22:27 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: ceph-mgr: Service RestartSec=10s expired, scheduling restart. Feb 02 07:22:27 ceph-mobisht-4-3z1-aobcda-node1-installer systemd[1]: ceph-mgr: Scheduled restart job, restart counter is at 4. ========================= [root@ceph-mobisht-4-3z1-aobcda-node1-installer ceph]# ll -lZ /var/lib/ceph/mgr/ceph-ceph-mobisht-4-3z1-aobcda-node1-installer/keyring -rw-------. 1 167 167 system_u:object_r:var_lib_t:s0 172 Feb 2 07:21 /var/lib/ceph/mgr/ceph-ceph-mobisht-4-3z1-aobcda-node1-installer/keyring ================================ --- Additional comment from Manisha Saini on 2023-02-02 13:00:38 UTC --- Logs - http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-70LC76/ --- Additional comment from Manisha Saini on 2023-02-02 13:26:01 UTC --- [root@ceph-mobisht-4-3z1-aobcda-node1-installer ceph]# rpm -qa | grep selinux container-selinux-2.189.0-1.module+el8.7.0+17498+a7f63b89.noarch rpm-plugin-selinux-4.14.3-24.el8_7.x86_64 libselinux-2.9-6.el8.x86_64 libselinux-utils-2.9-6.el8.x86_64 selinux-policy-3.14.3-108.el8_7.1.noarch python3-libselinux-2.9-6.el8.x86_64 selinux-policy-targeted-3.14.3-108.el8_7.1.noarch --- Additional comment from Teoman ONAY on 2023-02-02 21:28:52 UTC --- My findings so far are that there was a change in podman between these 2 versions: podman-3:4.2.0-1.module+el8.7.0+16772+33343656 <--- works podman-3:4.2.0-4.module+el8.7.0+17064+3b31f55c <--- does not work which broke the selinux relabeling of the objects on the shared volumes (e.g the "z" in -v /var/lib/ceph:/var/lib/ceph:z). When the container start it should relabel all files from system_u:object_r:var_lib_t:s0 -> system_u:object_r:container_file_t:s0. I will now look in podman changelog to find out what could have broken this. --- Additional comment from Jindrich Novy on 2023-02-03 09:29:26 UTC --- Dan, is there anything obvious in comment #7 how it could broke? Podman only contains updates to the latest upstream branch, nothing else. --- Additional comment from Daniel Walsh on 2023-02-04 14:08:54 UTC --- ls -lZd /var/lib/ceph The only change in this area that I am aware of is, if the top level directory is labeled correctly from an SELinux point of view, podman will no longer relabel the contents of the directory. Meaning if you mv'd files into the directory we could have an issue, if /var/lib/ceph is labeled container_file_t:s0. --- Additional comment from Teoman ONAY on 2023-02-14 16:09:13 UTC --- podman behavior changed after podman-3:4.2.0-1.module+el8.7.0+16772+33343656 regarding folder and files selinux relabeling. podman-3:4.2.0-4.module+el8.7.0+17064+3b31f55c and later are affected. The problem occurs when the MON container is started before the MGR one (the other way around it works). Both containers bindmount /var/lib/ceph folder. This is the usual content of that folder: drwxr-x---. 2 ceph ceph system_u:object_r:container_file_t:s0 6 Oct 17 20:40 bootstrap-mds drwxr-x---. 2 ceph ceph system_u:object_r:container_file_t:s0 6 Oct 17 20:40 bootstrap-mgr drwxr-x---. 2 ceph ceph system_u:object_r:container_file_t:s0 6 Oct 17 20:40 bootstrap-osd drwxr-x---. 2 ceph ceph system_u:object_r:container_file_t:s0 6 Oct 17 20:40 bootstrap-rbd drwxr-x---. 2 ceph ceph system_u:object_r:container_file_t:s0 6 Oct 17 20:40 bootstrap-rbd-mirror drwxr-x---. 2 ceph ceph system_u:object_r:container_file_t:s0 6 Oct 17 20:40 bootstrap-rgw drwxr-x---. 3 ceph ceph system_u:object_r:container_file_t:s0 20 Feb 10 19:04 crash drwxr-xr-x. 3 ceph ceph system_u:object_r:container_file_t:s0 23 Feb 10 19:03 mds drwxr-x---. 3 ceph ceph system_u:object_r:container_file_t:s0 23 Oct 17 20:40 mgr drwxr-xr-x. 3 ceph ceph system_u:object_r:container_file_t:s0 23 Feb 10 19:03 mon drwxr-xr-x. 2 ceph ceph system_u:object_r:container_file_t:s0 6 Feb 10 19:03 osd drwxr-xr-x. 3 ceph ceph system_u:object_r:container_file_t:s0 27 Feb 10 19:03 radosgw drwxr-x---. 2 ceph ceph system_u:object_r:container_file_t:s0 6 Oct 17 20:40 tmp the MON process get access to files in /var/lib/ceph/mon while the MGR process get access to /var/lib/ceph/mgr. Starting the MON first (which is started with --security-opt label=disable) does not relabel the /var/lib/ceph folder and its subfolders which are still labeled "var_lib_t" but it looks like it prevents the MGR process to do the relabeling as it should. All folders are relabeled but the file (keyring) contained in /var/lib/ceph/mgr keeps its var_lib_t label which prevent the MGR container from starting. In podman-3:4.2.0-1.module+el8.7.0+16772+33343656 this workflow works (keyring file is relabeled container_file_t) while with podman versions podman-3:4.2.0-4.module+el8.7.0+17064+3b31f55c and later the keyring file relabeling does not work. It seems to be related to "--security-opt label=disable". As a temporary fix (until podman is fixed), I will bind the MON container to "-v /var/lib/ceph/mon:/var/lib/ceph/mon:rshared" instead of "/var/lib/ceph/" which seems to work regardless to what container is started first. Here follows the systemd unit content for both containers: MON --- ExecStart=/usr/bin/podman run --rm --name ceph-mon-%i \ -d --log-driver journald --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \ --pids-limit=0 \ --memory=968m \ --cpus=1 \ --security-opt label=disable \ -v /var/lib/ceph:/var/lib/ceph:rshared \ -v /etc/ceph:/etc/ceph \ -v /var/run/ceph:/var/run/ceph \ -v /etc/localtime:/etc/localtime \ -v /var/log/ceph:/var/log/ceph \ -v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted \ --net=host \ -e IP_VERSION=4 \ -e MON_IP=192.168.1.10 \ -e CLUSTER=ceph \ -e FSID=10d0bcd1-1028-49c3-b30e-fc63c6fafb5c \ -e MON_PORT=3300 \ -e CEPH_PUBLIC_NETWORK=192.168.1.0/24 \ -e CEPH_DAEMON=MON \ -e CONTAINER_IMAGE=quay.io/ceph/daemon:latest-quincy-devel \ -e TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728 \ \ quay.io/ceph/daemon:latest-quincy-devel MGR --- ExecStart=/usr/bin/podman run --rm --net=host \ -d --log-driver journald --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \ --pids-limit=0 \ --memory=968m \ --cpus=1 \ -v /var/lib/ceph:/var/lib/ceph:z,rshared \ -v /etc/ceph:/etc/ceph:z \ -v /var/run/ceph:/var/run/ceph:z \ -v /etc/localtime:/etc/localtime:ro \ -v /var/log/ceph:/var/log/ceph:z \ -e CLUSTER=ceph \ -e CEPH_DAEMON=MGR \ -e CONTAINER_IMAGE=quay.io/ceph/daemon:latest-quincy-devel \ -e TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728 \ \ --name=ceph-mgr-mgr0 \ quay.io/ceph/daemon:latest-quincy-devel As a next step I will assign this BZ to the podman team as the product behavior has changed and needs to be fixed. In the meantime I will clone this BZ to apply the temporary workaround to ceph-ansible. --- Additional comment from RHEL Program Management on 2023-02-14 16:11:38 UTC --- pm_ack is no longer used for this product. The flag has been reset. See https://issues.redhat.com/browse/PTT-1821 for additional details. --- Additional comment from RHEL Program Management on 2023-02-14 16:11:38 UTC --- This bug was reopened or transitioned from a non-RHEL to RHEL product. The stale date has been reset to +6 months. --- Additional comment from Teoman ONAY on 2023-02-20 10:46:28 UTC --- FYI, the fix has been merged upstream (https://github.com/ceph/ceph-ansible/pull/7380) but I still need to backport it to 4.3 and test it. --- Additional comment from Anuchaithra on 2023-06-08 07:17:53 UTC --- Issue of "FAILED - RETRYING: wait for all mgr to be up" seen in upgrade i.e upgrade from 4.3z1 to 5.3 z4 (in both multisite and single site cluster) This issue is seen with rhel 8.7 and 8.8 (upgrade failed since mgr is not comming up) upgrade from ceph version 14.2.22-128.el8cp to ceph Version 16.2.10-174.el8cp http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-dll50/ceph_multisite_upgrade_0.log http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-1xngk/Upgrade_ceph_cluster_to_5.x_latest_0.log tested with rhel 8.5 and 8.6 upgrade is working fine 8.5: http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-ht8zn/ceph_multisite_upgrade_0.log 8.6: http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-i3ib2/Upgrade_ceph_cluster_to_5.x_latest_0.log http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-i3g5k/ceph_multisite_upgrade_0.log