Bug 1593866 - ceph-ansible fails to restart services after ceph_conf_overrides is changed and rerun
Summary: ceph-ansible fails to restart services after ceph_conf_overrides is changed a...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Ceph-Ansible
Version: 3.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 3.1
Assignee: Sébastien Han
QA Contact: ceph-qe-bugs
URL:
Whiteboard:
Depends On:
Blocks: 1581350
TreeView+ depends on / blocked
 
Reported: 2018-06-21 17:28 UTC by Vasu Kulkarni
Modified: 2023-09-14 04:30 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-11 14:14:41 UTC
Embargoed:


Attachments (Terms of Use)

Description Vasu Kulkarni 2018-06-21 17:28:27 UTC
Description of problem:

This issue was discovered by John during DFG workload, after applying config changes to ceph_conf_overrides and rerunning ceph-ansible, the services were not restarted and the setting didn't apply properly

From John's email thread.

1) verify current settings

[root@c04-h01-6048r ~]# ceph daemon osd.100 config get filestore_merge_threshold (10)

[root@c04-h01-6048r ~]# ceph daemon osd.100 config get objecter_inflight_ops (1024)

[root@c04-h01-6048r ~]# ceph daemon ceph-rgw.c04-h01-6048r config rgw_thread_pool_size  ← FAILS

2) Edit all.yml with recommended settings

  # ln -s all.ymlRECOMMENDED all.yml

  # vi group_vars/all.yml

# JTH apply RECOMMENDED settings

  global:

    objecter_inflight_ops: 24576         # default 1024

    rgw_thread_pool_size: 512            # default 50 (unable to verify)

  osd:

    filestore_merge_threshold: -1        # default 10

3) run ceph-ansible

  # ansible-playbook site.yml 2>&1 | tee recommendedJun17.txt


4) view settings

  # cat /etc/ceph/ceph.conf

        objecter_inflight_ops = 24576

        rgw_thread_pool_size = 512
        filestore_merge_threshold = -1


5) probe with 'ceph daemon' (run on OSD node)

  # ceph daemon osd.100 config get filestore_merge_threshold

      <-- still 10

  # ceph daemon osd.100 config get objecter_inflight_ops

      <-- still 1024

Comment 3 John Harrigan 2018-06-26 17:02:01 UTC
I ran the sequence of commands again and now see that one of the values
(rgw_thread_pool_size) gets applied while the other (objecter_inflight_ops)
does not:

1) edit 'group_vars/all.yml' as such:
# JTH apply RECOMMENDED settings
  global:
    objecter_inflight_ops: 24576         # default 1024
    rgw_thread_pool_size: 512            # default 100

2) run 'ansible-playbook site.yml'

3) look at ceph.conf confirming that new values are in there
   [global]
   objecter_inflight_ops = 24576
   rgw_thread_pool_size = 512

3) probe for values
[OSDnode1]# ceph daemon osd.105 config get objecter_inflight_ops
{
    "objecter_inflight_ops": "1024"  <-- NOT CHANGED
}
[OSDnode1]# ceph daemon client.rgw.c04-h01-6048r config get rgw_thread_pool_size
{
    "rgw_thread_pool_size": "512"    <-- CHANGED
}

4) restart daemons
# ansible all -a “systemctl restart ceph.target”

5) probe again for objecter_inflight_ops value
# ceph daemon osd.105 config get objecter_inflight_ops{
    "objecter_inflight_ops": "24576"   <-- CHANGED
}
# ceph daemon client.rgw.c04-h01-6048r config get rgw_thread_pool_size
{
    "rgw_thread_pool_size": "512"    <-- persists as expected
}

Any ideas on one value is applied while the other is not?

Comment 4 John Harrigan 2018-06-26 17:13:28 UTC
here is a transcript provided Guillaume Abrioux which indicates that this
works correctly
https://docs.google.com/document/d/1Nas4b-ZgmI4pGFtYwaMvCpWY_zrlFesL_3PPol7ERcc/edit?usp=sharing

My experience differs.

Comment 5 Christina Meno 2018-07-11 14:14:41 UTC
Vasu, what do you think about Johns comment in #c4? I plan to close this as WORKSFORME tomorrow lacking further detail from you.

Comment 6 Red Hat Bugzilla 2023-09-14 04:30:10 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.