Description of problem: OSP 13 deployment managing containerized ceph. The service file: /etc/systemd/system/ceph-osd\@.service contains the wrong container name format. ceph-osd-overcloud-cephstorage-0-%i vs actual container name. ceph-osd-%i This is a filestore deployment non-colocated if that makes a difference. ceph-ansible-3.2.30.1-1.el7cp.noarch # cat /etc/systemd/system/ceph-osd\@.service # Please do not change this file directly since it is managed by Ansible and will be overwritten [Unit] Description=Ceph OSD After=docker.service [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/docker stop ceph-osd-overcloud-cephstorage-0-%i ExecStartPre=-/usr/bin/docker rm -f ceph-osd-overcloud-cephstorage-0-%i ExecStart=/usr/share/ceph-osd-run.sh %i ExecStop=-/usr/bin/docker stop ceph-osd-overcloud-cephstorage-0-%i Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi-user.target Where the containers are named: # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a29a9302b27b 172.16.6.1:8787/rhceph/rhceph-3-rhel7:3-37 "/entrypoint.sh" 19 hours ago Up 19 hours ceph-osd-1 f9ce7a071301 172.16.6.1:8787/rhceph/rhceph-3-rhel7:3-37 "/entrypoint.sh" 2 days ago Up 2 days ceph-osd-12 bc7c5ca9157c 172.16.6.1:8787/rhceph/rhceph-3-rhel7:3-37 "/entrypoint.sh" 2 days ago Up 2 days ceph-osd-9 976d58d8387a 172.16.6.1:8787/rhceph/rhceph-3-rhel7:3-37 "/entrypoint.sh" 2 days ago Up 2 days ceph-osd-6 9854a4a629ec 172.16.6.1:8787/rhceph/rhceph-3-rhel7:3-37 "/entrypoint.sh" 2 days ago Up 2 days ceph-osd-3 9967ecd9f026 172.16.6.1:8787/rhosp13/openstack-cron:13.0-102 "dumb-init --singl..." 3 days ago Up 3 days logrotate_crond # grep name /usr/share/ceph-osd-run.sh --name=ceph-osd-"$1" \ Version-Release number of selected component (if applicable): ceph-ansible-3.2.30.1-1.el7cp.noarch OSP 13 deployment How reproducible: 100% Steps to Reproduce: 1. Deploy OSP 13 with ceph and non-colocated filestore OSD disks The OSDs still start up normal but the following are seen in the logs: # systemctl stop ceph-osd # systemctl start ceph-osd # tail -1000 /var/log/messages |grep 'No such container' Dec 6 09:58:07 overcloud-cephstorage-0 dockerd-current: time="2019-12-06T09:58:07.470652767-06:00" level=error msg="Handler for POST /v1.26/containers/ceph-osd-overcloud-cephstorage-0-9/stop returned error: No such container: ceph-osd-overcloud-cephstorage-0-9" Dec 6 09:58:07 overcloud-cephstorage-0 dockerd-current: time="2019-12-06T09:58:07.473639251-06:00" level=error msg="Handler for POST /v1.26/containers/ceph-osd-overcloud-cephstorage-0-9/stop returned error: No such container: ceph-osd-overcloud-cephstorage-0-9" Dec 6 09:58:07 overcloud-cephstorage-0 docker: Error response from daemon: No such container: ceph-osd-overcloud-cephstorage-0-9 Dec 6 09:58:19 overcloud-cephstorage-0 dockerd-current: time="2019-12-06T09:58:19.271731266-06:00" level=error msg="Handler for POST /v1.26/containers/ceph-osd-overcloud-cephstorage-0-9/stop returned error: No such container: ceph-osd-overcloud-cephstorage-0-9" Dec 6 09:58:19 overcloud-cephstorage-0 dockerd-current: time="2019-12-06T09:58:19.273934833-06:00" level=error msg="Handler for POST /v1.26/containers/ceph-osd-overcloud-cephstorage-0-9/stop returned error: No such container: ceph-osd-overcloud-cephstorage-0-9" Dec 6 09:58:19 overcloud-cephstorage-0 docker: Error response from daemon: No such container: ceph-osd-overcloud-cephstorage-0-9 Dec 6 09:58:19 overcloud-cephstorage-0 dockerd-current: time="2019-12-06T09:58:19.314878544-06:00" level=error msg="Handler for DELETE /v1.26/containers/ceph-osd-overcloud-cephstorage-0-9?force=1 returned error: No such container: ceph-osd-overcloud-cephstorage-0-9" Dec 6 09:58:19 overcloud-cephstorage-0 dockerd-current: time="2019-12-06T09:58:19.316382848-06:00" level=error msg="Handler for DELETE /v1.26/containers/ceph-osd-overcloud-cephstorage-0-9 returned error: No such container: ceph-osd-overcloud-cephstorage-0-9" Dec 6 09:58:19 overcloud-cephstorage-0 docker: Error response from daemon: No such container: ceph-osd-overcloud-cephstorage-0-9
I'm doing another deployment to provided the requested information. Note: This seems to only impact filestore deployments. For bluestore, /etc/systemd/system/ceph-osd\@.service was correct.
> Note: This seems to only impact filestore deployments. For bluestore, /etc/systemd/system/ceph-osd\@.service was correct. This should be the same with filestore and bluestore but limited to the collocated/non-colocated osd scenarios. The lvm osd scenario is not impacted.
Verified on ceph-ansible-3.2.38-1.el7cp
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:4353