Description of problem: We have deployed Ceph3 containerized env on RHEL 7.4 with selinux enforcing. After " yum update -y; reboot" to RHEL 7.5, the ceph containers are not starting with permission denied ------- Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: mktemp: failed to create directory via template '/var/lib/ceph/tmp/tmp.XXXXXXXXXX': Permission denied Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com docker[28805]: mktemp: failed to create directory via template '/var/lib/ceph/tmp/tmp.XXXXXXXXXX': Permission denied ------- Selinux contex has changed: [root@mons-2 ~]# ls -laZ /var/lib/ceph/tmp/ drwxr-x---. ceph ceph system_u:object_r:ceph_var_lib_t:s0 . drwxr-x---. ceph ceph system_u:object_r:ceph_var_lib_t:s0 .. drwx------. ceph ceph system_u:object_r:ceph_var_lib_t:s0 tmp.EJduARzNRI < not runnig drwx------. ceph ceph system_u:object_r:ceph_var_lib_t:s0 tmp.kQeOR6UmnN < not runnig drwx------. ceph ceph system_u:object_r:container_file_t:s0 tmp.t8zLsF7KjB <-- previously container runnig fine drwx------. ceph ceph system_u:object_r:container_file_t:s0 tmp.U9NNOM2aXC <-- previously container runnig fine [root@mons-2 ~]# ls -la /var/lib/ceph/tmp/ total 0 drwxr-x---. 6 ceph ceph 94 Apr 16 05:16 . drwxr-x---. 13 ceph ceph 181 Mar 8 16:41 .. drwx------. 2 ceph ceph 6 Apr 16 05:16 tmp.EJduARzNRI < not runnig drwx------. 2 ceph ceph 6 Apr 16 05:16 tmp.kQeOR6UmnN < not runnig drwx------. 2 ceph ceph 6 Apr 16 02:07 tmp.t8zLsF7KjB <-- previously container runnig fine drwx------. 2 ceph ceph 6 Apr 16 02:03 tmp.U9NNOM2aXC <-- previously container runnig fine ------- If we set selinux to permissive mode, the containers start ok. Version-Release number of selected component (if applicable): selinux-policy-targeted-3.13.1-192.el7_5.3.noarch libselinux-2.5-12.el7.x86_64 libselinux-utils-2.5-12.el7.x86_64 selinux-policy-3.13.1-192.el7_5.3.noarch libselinux-python-2.5-12.el7.x86_64 ceph-selinux-12.2.1-46.el7cp.x86_64 container-selinux-2.55-1.el7.noarch How reproducible: always Steps to Reproduce: 1. deploy container RHCS 3 + RHEL 7.4, selinux enforcing 2. yum update -y 3. reboot 4. Ceph containers are not starting 5. set selinux to permissive mode, containers start Actual results: Expected results: Additional info:
Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:09.624917374-04:00" level=error msg="Handler for POST /v1.26/containers/ceph-mgr-mons-2/stop returned error: No such container: ceph-mgr-mons-2" Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com docker[28783]: Error response from daemon: No such container: ceph-mgr-mons-2 Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:09.625057488-04:00" level=error msg="Handler for DELETE /v1.26/containers/ceph-mon-mons-2 returned error: No such container: ceph-mon-mons-2" Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:09.625415462-04:00" level=error msg="Handler for POST /v1.26/containers/ceph-mgr-mons-2/stop returned error: No such container: ceph-mgr-mons-2" Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:09.625496562-04:00" level=error msg="Handler for DELETE /v1.26/containers/ceph-mon-mons-2 returned error: No such container: ceph-mon-mons-2" Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com docker[28782]: Error response from daemon: No such container: ceph-mon-mons-2 Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Started Ceph Monitor. Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:09.655240668-04:00" level=error msg="Handler for DELETE /v1.26/containers/ceph-mgr-mons-2 returned error: No such container: ceph-mgr-mons-2" Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:09.655495130-04:00" level=error msg="Handler for DELETE /v1.26/containers/ceph-mgr-mons-2 returned error: No such container: ceph-mgr-mons-2" Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com docker[28792]: Error response from daemon: No such container: ceph-mgr-mons-2 Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Started Ceph Manager. Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Started libcontainer container 22e05c238d0796645f4b1aead31489037ab7b7bad6bb533a3e65af5f2fd1730e. Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Starting libcontainer container 22e05c238d0796645f4b1aead31489037ab7b7bad6bb533a3e65af5f2fd1730e. Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Started libcontainer container dfe00a1dd294334971d65714e1a5d24cc4d473c54ff13f46591168e92d652c6d. Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Starting libcontainer container dfe00a1dd294334971d65714e1a5d24cc4d473c54ff13f46591168e92d652c6d. Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com kernel: SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com kernel: SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com oci-systemd-hook[28868]: systemdhook <debug>: 22e05c238d07: Skipping as container command is /entrypoint.sh, not init or systemd Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com oci-umount[28873]: umounthook <debug>: prestart container_id:22e05c238d07 rootfs:/var/lib/docker/overlay2/cc4667de7e9ce5c64f306c46e374feaff8ac1ba071b3024b3592f34a63f00fca/merged Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com oci-systemd-hook[28875]: systemdhook <debug>: dfe00a1dd294: Skipping as container command is /entrypoint.sh, not init or systemd Apr 16 05:49:09 mons-2.container.quicklab.pnq2.cee.redhat.com oci-umount[28876]: umounthook <debug>: prestart container_id:dfe00a1dd294 rootfs:/var/lib/docker/overlay2/41ad62c1f0bd2812224b8288a7efe717fcf4b12b7eba79ff56dc6b260a0d44e8/merged Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: mktemp: failed to create directory via template '/var/lib/ceph/tmp/tmp.XXXXXXXXXX': Permission denied Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com docker[28805]: mktemp: failed to create directory via template '/var/lib/ceph/tmp/tmp.XXXXXXXXXX': Permission denied Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: mktemp: failed to create directory via template '/var/lib/ceph/tmp/tmp.XXXXXXXXXX': Permission denied Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com docker[28794]: mktemp: failed to create directory via template '/var/lib/ceph/tmp/tmp.XXXXXXXXXX': Permission denied Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com oci-systemd-hook[28946]: systemdhook <debug>: dfe00a1dd294: Skipping as container command is /entrypoint.sh, not init or systemd Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com oci-umount[28947]: umounthook <debug>: dfe00a1dd294: only runs in prestart stage, ignoring Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:10.107175247-04:00" level=error msg="containerd: deleting container" error="exit status 1: \"container dfe00a1dd294334971d65714e1a5d24cc4d473c54ff13f46591168e92d652c6d does not exist\\none or more of the container deletions failed\\n\"" Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com oci-systemd-hook[28958]: systemdhook <debug>: 22e05c238d07: Skipping as container command is /entrypoint.sh, not init or systemd Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com oci-umount[28959]: umounthook <debug>: 22e05c238d07: only runs in prestart stage, ignoring Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:10.119029744-04:00" level=error msg="containerd: deleting container" error="exit status 1: \"container 22e05c238d0796645f4b1aead31489037ab7b7bad6bb533a3e65af5f2fd1730e does not exist\\none or more of the container deletions failed\\n\"" Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:10.149779899-04:00" level=warning msg="dfe00a1dd294334971d65714e1a5d24cc4d473c54ff13f46591168e92d652c6d cleanup: failed to unmount secrets: invalid argument" Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:10.182241158-04:00" level=warning msg="22e05c238d0796645f4b1aead31489037ab7b7bad6bb533a3e65af5f2fd1730e cleanup: failed to unmount secrets: invalid argument" Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: ceph-mgr: main process exited, code=exited, status=1/FAILURE Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:10.219366441-04:00" level=error msg="Handler for POST /v1.26/containers/ceph-mgr-mons-2/stop returned error: No such container: ceph-mgr-mons-2" Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: time="2018-04-16T05:49:10.219649961-04:00" level=error msg="Handler for POST /v1.26/containers/ceph-mgr-mons-2/stop returned error: No such container: ceph-mgr-mons-2" Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com docker[28966]: Error response from daemon: No such container: ceph-mgr-mons-2 Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Unit ceph-mgr entered failed state. Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: ceph-mgr failed. Apr 16 05:49:10 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: ceph-mon: main process exited, code=exited, status=1/FAILURE
SELinux is preventing /usr/bin/mktemp from write access on the directory tmp. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that mktemp should be allowed write access on the tmp directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'mktemp' --raw | audit2allow -M my-mktemp # semodule -i my-mktemp.pp Additional Information: Source Context system_u:system_r:container_t:s0:c384,c657 Target Context system_u:object_r:ceph_var_lib_t:s0 Target Objects tmp [ dir ] Source mktemp Source Path /usr/bin/mktemp Port <Unknown> Host <Unknown> Source RPM Packages coreutils-8.22-21.el7.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-192.el7_5.3.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name mons-2.container.quicklab.pnq2.cee.redhat.com Platform Linux mons-2.container.quicklab.pnq2.cee.redhat.com 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 Alert Count 1 First Seen 2018-04-16 05:35:14 EDT Last Seen 2018-04-16 05:35:14 EDT Local ID 59faa3b1-010e-45ea-884f-50328bc0f65e Raw Audit Messages type=AVC msg=audit(1523871314.212:1560): avc: denied { write } for pid=13477 comm="mktemp" name="tmp" dev="vda1" ino=41952155 scontext=system_u:system_r:container_t:s0:c384,c657 tcontext=system_u:object_r:ceph_var_lib_t:s0 tclass=dir type=SYSCALL msg=audit(1523871314.212:1560): arch=x86_64 syscall=mkdir success=no exit=EACCES a0=1d050b0 a1=1c0 a2=22 a3=7ffdee0816a0 items=0 ppid=13476 pid=13477 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=mktemp exe=/usr/bin/mktemp subj=system_u:system_r:container_t:s0:c384,c657 key=(null) Hash: mktemp,container_t,ceph_var_lib_t,dir,write
Trying to work it around with enforcing selinux: # audit2allow -a #============= container_t ============== allow container_t ceph_var_lib_t:dir { add_name create read setattr write }; ausearch -c 'mktemp' --raw | audit2allow -M my-mktemp semodule -i my-mktemp.pp Then fails with: Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com docker[4771]: Error response from daemon: No such container: ceph-mgr-mons-2 Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Started Ceph Manager. Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Started libcontainer container 2a001c107195ff5d380b7b84b6a4d0480d6d497031a3767e1e623920792bd954. Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Starting libcontainer container 2a001c107195ff5d380b7b84b6a4d0480d6d497031a3767e1e623920792bd954. Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com kernel: SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com oci-systemd-hook[4809]: systemdhook <debug>: 2a001c107195: Skipping as container command is /entrypoint.sh, not init or systemd Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com oci-umount[4810]: umounthook <debug>: prestart container_id:2a001c107195 rootfs:/var/lib/docker/overlay2/d8030f632de9f64667aafadd432271812a1b12af10bce12e71ba144a4c679f35/merged Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com dockerd-current[1125]: find: '/var/lib/ceph/': Permission denied Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com docker[4751]: find: '/var/lib/ceph/': Permission denied Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com oci-systemd-hook[4895]: systemdhook <debug>: 2a001c107195: Skipping as container command is /entrypoint.sh, not init or systemd Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com oci-umount[4896]: umounthook <debug>: 2a001c107195: only runs in prestart stage, ignoring Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Started libcontainer container 7d20529749d35c1340a7e5031f0a183cc393a910599be725a43acbdbb0107428. Apr 16 05:56:36 mons-2.container.quicklab.pnq2.cee.redhat.com systemd[1]: Starting libcontainer container 7d20529749d35c1340a7e5031f0a183cc393a910599be725a43acbdbb0107428. [root@mons-2 ~]# audit2allow -a #============= container_t ============== allow container_t ceph_var_lib_t:dir { read setattr }; #!!!! This avc is allowed in the current policy allow container_t ceph_var_lib_t:dir { add_name create write }; -------------------------------------------------------------------------------- SELinux is preventing /usr/bin/find from read access on the directory ceph. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that find should be allowed read access on the ceph directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'find' --raw | audit2allow -M my-find # semodule -i my-find.pp Additional Information: Source Context system_u:system_r:container_t:s0:c242,c847 Target Context system_u:object_r:ceph_var_lib_t:s0 Target Objects ceph [ dir ] Source find Source Path /usr/bin/find Port <Unknown> Host <Unknown> Source RPM Packages findutils-4.5.11-5.el7.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-192.el7_5.3.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name mons-2.container.quicklab.pnq2.cee.redhat.com Platform Linux mons-2.container.quicklab.pnq2.cee.redhat.com 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 Alert Count 1 First Seen 2018-04-16 06:02:36 EDT Last Seen 2018-04-16 06:02:36 EDT Local ID 1324c97d-b7a4-43ab-b3f4-68007d87e5a3 Raw Audit Messages type=AVC msg=audit(1523872956.294:5799): avc: denied { read } for pid=13824 comm="find" name="ceph" dev="vda1" ino=29360673 scontext=system_u:system_r:container_t:s0:c242,c847 tcontext=system_u:object_r:ceph_var_lib_t:s0 tclass=dir type=SYSCALL msg=audit(1523872956.294:5799): arch=x86_64 syscall=openat success=no exit=EACCES a0=ffffffffffffff9c a1=1e01a40 a2=10900 a3=0 items=0 ppid=13706 pid=13824 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=find exe=/usr/bin/find subj=system_u:system_r:container_t:s0:c242,c847 key=(null) Hash: find,container_t,ceph_var_lib_t,dir,read [root@mons-2 ~]# ausearch -c 'find' --raw | audit2allow -M my-find; semodule -i my-find.pp ******************** IMPORTANT *********************** To make this policy package active, execute: semodule -i my-find.pp Then finally the mon and mgr containers start: [root@mons-2 ~]# ps -ef | egrep "ceph|rados|docker" root 1125 1 1 05:25 ? 00:00:23 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --add-registry registry.access.redhat.com root 1222 1125 0 05:25 ? 00:00:04 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtime-args --systemd-cgroup=true root 1411 1 0 05:25 ? 00:00:01 /usr/libexec/docker/rhel-push-plugin root 15586 1 1 06:03 ? 00:00:00 /usr/bin/docker-current run --rm --name ceph-mon-mons-2 --net=host --memory=1g --cpu-quota=100000 -v /var/lib/ceph:/var/lib/ceph -v /etc/ceph:/etc/ceph -v /etc/localtime:/etc/localtime:ro --net=host -e IP_VERSION=4 -e MON_IP=10.74.157.245 -e CLUSTER=ceph -e FSID=78a165f7-8ec1-4595-9950-a8b7672dc9e9 -e CEPH_PUBLIC_NETWORK=10.74.156.0/22 -e CEPH_DAEMON=MON registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest root 15597 1 1 06:03 ? 00:00:00 /usr/bin/docker-current run --rm --net=host --memory=1g --cpu-quota=100000 -v /var/lib/ceph:/var/lib/ceph -v /etc/ceph:/etc/ceph -v /etc/localtime:/etc/localtime:ro -e CLUSTER=ceph -e CEPH_DAEMON=MGR --name=ceph-mgr-mons-2 registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest root 15618 1222 0 06:03 ? 00:00:00 /usr/bin/docker-containerd-shim-current 371aee859610d7782f25057d8845f30458fb44cf6f752b44b1fbcd05c0446129 /var/run/docker/libcontainerd/371aee859610d7782f25057d8845f30458fb44cf6f752b44b1fbcd05c0446129 /usr/libexec/docker/docker-runc-current root 15631 1222 0 06:03 ? 00:00:00 /usr/bin/docker-containerd-shim-current 3c1c7bc35804975b72e99ffafb73a614304497243162bc538e2a22717719726b /var/run/docker/libcontainerd/3c1c7bc35804975b72e99ffafb73a614304497243162bc538e2a22717719726b /usr/libexec/docker/docker-runc-current root 16003 15637 0 06:03 ? 00:00:00 timeout 7 ceph --cluster ceph mon add mons-2 10.74.157.245:6789 root 16008 16003 16 06:03 ? 00:00:00 /usr/bin/python2.7 /usr/bin/ceph --cluster ceph mon add mons-2 10.74.157.245:6789 ceph 16012 15653 6 06:03 ? 00:00:00 /usr/bin/ceph-mgr --cluster ceph --setuser ceph --setgroup ceph -d -i mons-2 root 16044 2064 0 06:03 pts/0 00:00:00 grep -E --color=auto ceph|rados|docker
Note: problem has showed on RGW, MON+MRG node not showed on OSD node at all, osd started normally, not sealert not tested with MDS and iSCSI
I think I know what this is. Trying a patch for this.
Hopefully with this patch Docker will maintain the SeLinux labels across reboot. The upstream CI does not run on 7.5 since it's not released yet. We can't fully validate this but at least make sure we don't break things.
Tomas, could you please relabel /var/lib/ceph and /etc/ceph directories like this after the reboot: chcon -Rt svirt_sandbox_file_t /var/lib/ceph /etc/ceph That's should be the workaround to that issue. Thanks.
(In reply to leseb from comment #9) > Tomas, could you please relabel /var/lib/ceph and /etc/ceph directories like > this after the reboot: > > chcon -Rt svirt_sandbox_file_t /var/lib/ceph /etc/ceph > > > That's should be the workaround to that issue. > Thanks. Hi Sebastien, thanks, that WA worked as well: # chcon -Rt svirt_sandbox_file_t /var/lib/ceph /etc/ceph [root@mons-1 ~]# ps -ef | grep ceph root 24378 1 0 11:47 ? 00:00:00 /usr/bin/docker-current run --rm --name ceph-mon-mons-1 --net=host --memory=1g --cpu-quota=100000 -v /var/lib/ceph:/var/lib/ceph -v /etc/ceph:/etc/ceph -v /etc/localtime:/etc/localtime:ro --net=host -e IP_VERSION=4 -e MON_IP=10.74.157.247 -e CLUSTER=ceph -e FSID=78a165f7-8ec1-4595-9950-a8b7672dc9e9 -e CEPH_PUBLIC_NETWORK=10.74.156.0/22 -e CEPH_DAEMON=MON registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest root 24407 1 0 11:47 ? 00:00:00 /usr/bin/docker-current run --rm --net=host --memory=1g --cpu-quota=100000 -v /var/lib/ceph:/var/lib/ceph -v /etc/ceph:/etc/ceph -v /etc/localtime:/etc/localtime:ro -e CLUSTER=ceph -e CEPH_DAEMON=MGR --name=ceph-mgr-mons-1 registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest ceph 24627 24508 1 11:47 ? 00:00:00 /usr/bin/ceph-mgr --cluster ceph --setuser ceph --setgroup ceph -d -i mons-1 ceph 24675 24428 1 11:47 ? 00:00:00 /usr/bin/ceph-mon --cluster ceph --setuser ceph --setgroup ceph -d -i mons-1 --mon-data /var/lib/ceph/mon/ceph-mons-1 --public-addr 10.74.157.247:6789 Regards, Tomas
This is annoying though as I don't reproduce this on 7.4. I hope my patch will re-apply the label each time the container starts, otherwise, I'll fix that differently. Tomas, since you have the env could you please, reboot again (I'm expecting you to lose the Selinux labels) then edit one of the systemd unit and change the -v like so: https://github.com/ceph/ceph-ansible/pull/2526/files#diff-9d6f3197018a38e7f772112f9bd62f19R17 Basically add :z at the end of each -v for /var/lib/ceph and /etc/ceph Then reboot again and see if the label get re-applied. If you could help with this that'll really great. I'm thinking of keeping the :z even if the labels don't get re-applied anyways. If that's the case I'll fix the unit file itself by adding a pre-start command (chcon). Thanks in advance!
(In reply to leseb from comment #11) > This is annoying though as I don't reproduce this on 7.4. > I hope my patch will re-apply the label each time the container starts, > otherwise, I'll fix that differently. > > Tomas, since you have the env could you please, reboot again (I'm expecting > you to lose the Selinux labels) then edit one of the systemd unit and change > the -v like so: > > https://github.com/ceph/ceph-ansible/pull/2526/files#diff- > 9d6f3197018a38e7f772112f9bd62f19R17 > > Basically add :z at the end of each -v for /var/lib/ceph and /etc/ceph > > Then reboot again and see if the label get re-applied. If you could help > with this that'll really great. > > I'm thinking of keeping the :z even if the labels don't get re-applied > anyways. If that's the case I'll fix the unit file itself by adding a > pre-start command (chcon). > > Thanks in advance! Seems it worded as well: [root@mons-0 ~]# cat /etc/systemd/system/ceph-mgr@.service [Unit] Description=Ceph Manager After=docker.service [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/docker stop ceph-mgr-mons-0 ExecStartPre=-/usr/bin/docker rm ceph-mgr-mons-0 ExecStart=/usr/bin/docker run --rm --net=host \ --memory=1g \ --cpu-quota=100000 \ -v /var/lib/ceph:/var/lib/ceph:z \ #<-------- -v /etc/ceph:/etc/ceph:z \ #<-------- -v /etc/localtime:/etc/localtime:ro \ -e CLUSTER=ceph \ -e CEPH_DAEMON=MGR \ \ --name=ceph-mgr-mons-0 \ registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest ExecStopPost=-/usr/bin/docker stop ceph-mgr-mons-0 Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi-user.target ---------------------------- [root@mons-0 ~]# reboot [root@mons-0 ~]# ps -ef | egrep "ceph" root 1422 1 0 12:16 ? 00:00:00 /usr/bin/docker-current run --rm --name ceph-mon-mons-0 --net=host --memory=1g --cpu-quota=100000 -v /var/lib/ceph:/var/lib/ceph -v /etc/ceph:/etc/ceph -v /etc/localtime:/etc/localtime:ro --net=host -e IP_VERSION=4 -e MON_IP=10.74.157.243 -e CLUSTER=ceph -e FSID=78a165f7-8ec1-4595-9950-a8b7672dc9e9 -e CEPH_PUBLIC_NETWORK=10.74.156.0/22 -e CEPH_DAEMON=MON registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest root 1434 1 0 12:16 ? 00:00:00 /usr/bin/docker-current run --rm --net=host --memory=1g --cpu-quota=100000 -v /var/lib/ceph:/var/lib/ceph:z -v /etc/ceph:/etc/ceph:z -v /etc/localtime:/etc/localtime:ro -e CLUSTER=ceph -e CEPH_DAEMON=MGR --name=ceph-mgr-mons-0 registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest ceph 1688 1497 2 12:16 ? 00:00:01 /usr/bin/ceph-mgr --cluster ceph --setuser ceph --setgroup ceph -d -i mons-0 ceph 1731 1496 1 12:16 ? 00:00:00 /usr/bin/ceph-mon --cluster ceph --setuser ceph --setgroup ceph -d -i mons-0 --mon-data /var/lib/ceph/mon/ceph-mons-0 --public-addr 10.74.157.243:6789
Thanks a lot Tomas for validating this for me.
Fix already in 3.1.
"selinux: remove chcon calls" has been in stable-3.1 for a long time (since v3.1.0beta8). I'm setting Fixed In Version to the latest available ceph-ansible NVR.
Hi, We are planning to verify the fix by following steps - 1) Using latest available ceph-ansible and container image (that comes with RHCS 3.1), with selinux in enforcing mode, Create a cluster with OSDs of different scenarios ON RHEL- VERSION 7.4 2) Upgrade host OS TO 7.5 and reboot reboot all nodes in rolling fashion and verify that all daemons are up and running once host is back after reboot. Kindly let us know if there are any concerns on the steps we have planned, before EOD tomorrow (Friday). Regards, Vasishta QE, Ceph
lgtm.
providing qa_ack
Followed Steps mentioned planned and as mentioned in Comment 18, Working fine. Verified in - ceph-ansible-3.1.0-0.1.rc17.el7cp.noarch ceph-3.1-rhel-7-containers-candidate-38485-20180810211451 (12.2.5-38.el7cp) Will move BZ to VERIFIED state once BZ comes ON_QA. Thanks.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2820