Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1596061

Summary: ceph-osd container does not restart on option changes
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Sébastien Han <shan>
Component: Ceph-AnsibleAssignee: Sébastien Han <shan>
Status: CLOSED ERRATA QA Contact: Persona non grata <nobody+410372>
Severity: high Docs Contact: Bara Ancincova <bancinco>
Priority: high    
Version: 3.0CC: agunn, anharris, aschoen, ceph-eng-bugs, ceph-qe-bugs, gcharot, gfidente, gmeno, hnallurv, jbrier, kdreyer, nobody+410372, nthomas, sankarshan, shan, tserlin, vumrao
Target Milestone: z1   
Target Release: 3.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-ansible-3.1.8-1.el7cp Ubuntu: ceph-ansible_3.1.8-2redhat1 Doc Type: Bug Fix
Doc Text:
.Containers are now restarted automatically when changing their options Previously, when changing a container option, for example, `ceph_osd_docker_memory_limit`, the change did not trigger a restart of the container and a manual restart was required. With this update, containers are restarted automatically when changing their options.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-09 00:59:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1578730, 1584264    
Attachments:
Description Flags
ansible log none

Description Sébastien Han 2018-06-28 07:59:25 UTC 
Description of problem:

When changing an option like ceph_osd_docker_memory_limit the change won't trigger a restart of the container.


Version-Release number of selected component (if applicable):

v3.1.0rc9

How reproducible:

Change ceph_osd_docker_memory_limit to a value different than the default

Steps to Reproduce:
1.
2.
3.

Actual results:

The OSD container(s) won't restart.

Expected results:

The OSD container(s) restart.

Additional info:
Comment 7 Persona non grata 2018-08-16 11:46:17 UTC 
Steps followed:

In osd.yaml, changed 'ceph_osd_docker_memory_limit' value to 1g once,6g in next time.  Reran ansible-playbook, container did not restart, moving back to assigned
Comment 8 Sébastien Han 2018-08-21 09:45:23 UTC 
logs?
Comment 9 Persona non grata 2018-08-27 06:52:07 UTC 
Created attachment 1478874 [details]
ansible log
Comment 11 Gregory Charot 2018-08-31 12:43:40 UTC 
Same behaviour when you try to change a value in ceph.conf

With TripleO:

  CephConfigOverrides:
    mon_max_pg_per_osd: 2048

(which uses ceph-ansible ceph_conf_overrides)

ceph.conf is updated on all nodes but OSD daemons are not restarted.

Please note that monitors are properly restarted and thus run with the proper configuration.
Comment 13 Vikhyat Umrao 2018-09-04 18:14:59 UTC 
(In reply to Gregory Charot from comment #11)
> Same behaviour when you try to change a value in ceph.conf
> 
> With TripleO:
> 
>   CephConfigOverrides:
>     mon_max_pg_per_osd: 2048
> 
> (which uses ceph-ansible ceph_conf_overrides)
> 
> ceph.conf is updated on all nodes but OSD daemons are not restarted.
> 
> Please note that monitors are properly restarted and thus run with the
> proper configuration.

Maybe because you have chosen mon option *mon_*hence this is the behavior. Please use any option which starts from *osd_* these are OSD options so if you choose one of the osd_ it should help.
Comment 17 Gregory Charot 2018-09-05 09:10:22 UTC 
(In reply to Vikhyat Umrao from comment #13)
> (In reply to Gregory Charot from comment #11)
> > Same behaviour when you try to change a value in ceph.conf
> > 
> > With TripleO:
> > 
> >   CephConfigOverrides:
> >     mon_max_pg_per_osd: 2048
> > 
> > (which uses ceph-ansible ceph_conf_overrides)
> > 
> > ceph.conf is updated on all nodes but OSD daemons are not restarted.
> > 
> > Please note that monitors are properly restarted and thus run with the
> > proper configuration.
> 
> Maybe because you have chosen mon option *mon_*hence this is the behavior.
> Please use any option which starts from *osd_* these are OSD options so if
> you choose one of the osd_ it should help.

The option lands in the global ceph.conf section on all nodes so the OSDs should be restarted, i don't think ceph-ansible actually looks at the variable name.

When i applied the changes and tried to create new pools, the ceph osd pool create command worked but pgs creation were stuck. Restarting the OSDs solved the issue.
Comment 18 John Brier 2018-10-02 20:45:46 UTC 
Release Notes Doc Text update: replace contraction "won't" with "will not" per IBM Style Guide
Comment 19 John Brier 2018-10-02 21:09:07 UTC 
Release Notes Doc Text remove extra space inbetween "example," and "`ceph_osd_docker_memory_limit`"
Comment 20 Sébastien Han 2018-10-03 14:17:25 UTC 
Present in https://github.com/ceph/ceph-ansible/releases/tag/v3.1.8
Comment 25 Sébastien Han 2018-10-16 13:51:05 UTC 
lgtm thanks
Comment 28 errata-xmlrpc 2018-11-09 00:59:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3530