Bug 1596061 - ceph-osd container does not restart on option changes
Summary: ceph-osd container does not restart on option changes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Ceph-Ansible
Version: 3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z1
: 3.1
Assignee: Sébastien Han
QA Contact: Persona non grata
Bara Ancincova
URL:
Whiteboard:
Depends On:
Blocks: 1578730 1584264
TreeView+ depends on / blocked
 
Reported: 2018-06-28 07:59 UTC by Sébastien Han
Modified: 2018-11-09 01:00 UTC (History)
17 users (show)

Fixed In Version: RHEL: ceph-ansible-3.1.8-1.el7cp Ubuntu: ceph-ansible_3.1.8-2redhat1
Doc Type: Bug Fix
Doc Text:
.Containers are now restarted automatically when changing their options Previously, when changing a container option, for example, `ceph_osd_docker_memory_limit`, the change did not trigger a restart of the container and a manual restart was required. With this update, containers are restarted automatically when changing their options.
Clone Of:
Environment:
Last Closed: 2018-11-09 00:59:17 UTC
Embargoed:


Attachments (Terms of Use)
ansible log (322.44 KB, text/plain)
2018-08-27 06:52 UTC, Persona non grata
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github ceph ceph-ansible pull 2823 0 'None' closed Restart osd container on config change 2020-05-18 06:14:23 UTC
Github ceph ceph-ansible pull 3188 0 'None' closed handler osd 2020-05-18 06:14:23 UTC
Red Hat Product Errata RHBA-2018:3530 0 None None None 2018-11-09 01:00:30 UTC

Description Sébastien Han 2018-06-28 07:59:25 UTC
Description of problem:

When changing an option like ceph_osd_docker_memory_limit the change won't trigger a restart of the container.


Version-Release number of selected component (if applicable):

v3.1.0rc9

How reproducible:

Change ceph_osd_docker_memory_limit to a value different than the default

Steps to Reproduce:
1.
2.
3.

Actual results:

The OSD container(s) won't restart.

Expected results:

The OSD container(s) restart.

Additional info:

Comment 7 Persona non grata 2018-08-16 11:46:17 UTC
Steps followed:

In osd.yaml, changed 'ceph_osd_docker_memory_limit' value to 1g once,6g in next time.  Reran ansible-playbook, container did not restart, moving back to assigned

Comment 8 Sébastien Han 2018-08-21 09:45:23 UTC
logs?

Comment 9 Persona non grata 2018-08-27 06:52:07 UTC
Created attachment 1478874 [details]
ansible log

Comment 11 Gregory Charot 2018-08-31 12:43:40 UTC
Same behaviour when you try to change a value in ceph.conf

With TripleO:

  CephConfigOverrides:
    mon_max_pg_per_osd: 2048

(which uses ceph-ansible ceph_conf_overrides)

ceph.conf is updated on all nodes but OSD daemons are not restarted.

Please note that monitors are properly restarted and thus run with the proper configuration.

Comment 13 Vikhyat Umrao 2018-09-04 18:14:59 UTC
(In reply to Gregory Charot from comment #11)
> Same behaviour when you try to change a value in ceph.conf
> 
> With TripleO:
> 
>   CephConfigOverrides:
>     mon_max_pg_per_osd: 2048
> 
> (which uses ceph-ansible ceph_conf_overrides)
> 
> ceph.conf is updated on all nodes but OSD daemons are not restarted.
> 
> Please note that monitors are properly restarted and thus run with the
> proper configuration.

Maybe because you have chosen mon option *mon_*hence this is the behavior. Please use any option which starts from *osd_* these are OSD options so if you choose one of the osd_ it should help.

Comment 17 Gregory Charot 2018-09-05 09:10:22 UTC
(In reply to Vikhyat Umrao from comment #13)
> (In reply to Gregory Charot from comment #11)
> > Same behaviour when you try to change a value in ceph.conf
> > 
> > With TripleO:
> > 
> >   CephConfigOverrides:
> >     mon_max_pg_per_osd: 2048
> > 
> > (which uses ceph-ansible ceph_conf_overrides)
> > 
> > ceph.conf is updated on all nodes but OSD daemons are not restarted.
> > 
> > Please note that monitors are properly restarted and thus run with the
> > proper configuration.
> 
> Maybe because you have chosen mon option *mon_*hence this is the behavior.
> Please use any option which starts from *osd_* these are OSD options so if
> you choose one of the osd_ it should help.

The option lands in the global ceph.conf section on all nodes so the OSDs should be restarted, i don't think ceph-ansible actually looks at the variable name.

When i applied the changes and tried to create new pools, the ceph osd pool create command worked but pgs creation were stuck. Restarting the OSDs solved the issue.

Comment 18 John Brier 2018-10-02 20:45:46 UTC
Release Notes Doc Text update: replace contraction "won't" with "will not" per IBM Style Guide

Comment 19 John Brier 2018-10-02 21:09:07 UTC
Release Notes Doc Text remove extra space inbetween "example," and "`ceph_osd_docker_memory_limit`"

Comment 20 Sébastien Han 2018-10-03 14:17:25 UTC
Present in https://github.com/ceph/ceph-ansible/releases/tag/v3.1.8

Comment 25 Sébastien Han 2018-10-16 13:51:05 UTC
lgtm thanks

Comment 28 errata-xmlrpc 2018-11-09 00:59:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3530


Note You need to log in before you can comment on or make changes to this bug.