Bug 1596061 – ceph-osd container does not restart on option changes

Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.

Bug 1596061 - ceph-osd container does not restart on option changes

Summary:	ceph-osd container does not restart on option changes

Status:	VERIFIED

Aliases:	None

Product:	Red Hat Ceph Storage
Classification:	Red Hat
Component:	Ceph-Ansible (Show other bugs)
Sub Component:
Version:	3.0
Hardware:	Unspecified Unspecified

Priority	high Severity high
Target Milestone:	z1
Target Release:	3.1
Assigned To:	leseb
QA Contact:	Shreekar
Docs Contact:	Bara Ancincova

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:	1578730 1584264
	Show dependency tree / graph

Reported:	2018-06-28 03:59 EDT by leseb
Modified:	2018-10-23 12:59 EDT (History)
CC List:	17 users (show)

See Also:
Fixed In Version:	RHEL: ceph-ansible-3.1.8-1.el7cp Ubuntu: ceph-ansible_3.1.8-2redhat1
Doc Type:	Bug Fix
Doc Text:	.Containers are now restarted automatically when changing their options _[fixed by 3.1z1]_ Previously, when changing a container option, for example, `ceph_osd_docker_memory_limit`, the change did not trigger a restart of the container and a manual restart was required. With this update, containers are restarted automatically when changing their options.
Story Points:	---
Clone Of:
Environment:
Last Closed:
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
ansible log (322.44 KB, text/plain) 2018-08-27 02:52 EDT, Shreekar	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Github	ceph/ceph-ansible/pull/2823	None	None	None	2018-06-28 04:02 EDT
Github	ceph/ceph-ansible/pull/3188	None	None	None	2018-10-03 08:00 EDT

Groups: None (edit)

Description leseb 2018-06-28 03:59:25 EDT

Description of problem:

When changing an option like ceph_osd_docker_memory_limit the change won't trigger a restart of the container.


Version-Release number of selected component (if applicable):

v3.1.0rc9

How reproducible:

Change ceph_osd_docker_memory_limit to a value different than the default

Steps to Reproduce:
1.
2.
3.

Actual results:

The OSD container(s) won't restart.

Expected results:

The OSD container(s) restart.

Additional info:

Comment 7 Shreekar 2018-08-16 07:46:17 EDT

Steps followed:

In osd.yaml, changed 'ceph_osd_docker_memory_limit' value to 1g once,6g in next time.  Reran ansible-playbook, container did not restart, moving back to assigned

Comment 8 leseb 2018-08-21 05:45:23 EDT

logs?

Comment 9 Shreekar 2018-08-27 02:52 EDT

Created attachment 1478874 [details]
ansible log

Comment 11 Gregory Charot 2018-08-31 08:43:40 EDT

Same behaviour when you try to change a value in ceph.conf

With TripleO:

  CephConfigOverrides:
    mon_max_pg_per_osd: 2048

(which uses ceph-ansible ceph_conf_overrides)

ceph.conf is updated on all nodes but OSD daemons are not restarted.

Please note that monitors are properly restarted and thus run with the proper configuration.

Comment 13 Vikhyat Umrao 2018-09-04 14:14:59 EDT

(In reply to Gregory Charot from comment #11)
> Same behaviour when you try to change a value in ceph.conf
> 
> With TripleO:
> 
>   CephConfigOverrides:
>     mon_max_pg_per_osd: 2048
> 
> (which uses ceph-ansible ceph_conf_overrides)
> 
> ceph.conf is updated on all nodes but OSD daemons are not restarted.
> 
> Please note that monitors are properly restarted and thus run with the
> proper configuration.

Maybe because you have chosen mon option *mon_*hence this is the behavior. Please use any option which starts from *osd_* these are OSD options so if you choose one of the osd_ it should help.

Comment 17 Gregory Charot 2018-09-05 05:10:22 EDT

(In reply to Vikhyat Umrao from comment #13)
> (In reply to Gregory Charot from comment #11)
> > Same behaviour when you try to change a value in ceph.conf
> > 
> > With TripleO:
> > 
> >   CephConfigOverrides:
> >     mon_max_pg_per_osd: 2048
> > 
> > (which uses ceph-ansible ceph_conf_overrides)
> > 
> > ceph.conf is updated on all nodes but OSD daemons are not restarted.
> > 
> > Please note that monitors are properly restarted and thus run with the
> > proper configuration.
> 
> Maybe because you have chosen mon option *mon_*hence this is the behavior.
> Please use any option which starts from *osd_* these are OSD options so if
> you choose one of the osd_ it should help.

The option lands in the global ceph.conf section on all nodes so the OSDs should be restarted, i don't think ceph-ansible actually looks at the variable name.

When i applied the changes and tried to create new pools, the ceph osd pool create command worked but pgs creation were stuck. Restarting the OSDs solved the issue.

Comment 18 John Brier 2018-10-02 16:45:46 EDT

Release Notes Doc Text update: replace contraction "won't" with "will not" per IBM Style Guide

Comment 19 John Brier 2018-10-02 17:09:07 EDT

Release Notes Doc Text remove extra space inbetween "example," and "`ceph_osd_docker_memory_limit`"

Comment 20 leseb 2018-10-03 10:17:25 EDT

Present in https://github.com/ceph/ceph-ansible/releases/tag/v3.1.8

Comment 25 leseb 2018-10-16 09:51:05 EDT

lgtm thanks

Note You need to log in before you can comment on or make changes to this bug.