Bug 1856981

Summary:

9GB of RAM are wasted for TASK [ceph-facts : set_fact ceph_current_status (convert to json)]

Product:

[Red Hat Storage] Red Hat Ceph Storage

Reporter:

John Fulton <johfulto>

Component:

Ceph-Ansible

Assignee:

Guillaume Abrioux <gabrioux>

Status:

CLOSED ERRATA

QA Contact:

Vasishta <vashastr>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

4.1

CC:

aschoen, bdobreli, bengland, ceph-eng-bugs, dsavinea, gabrioux, gmeno, nthomas, smalleni, tserlin, vereddy, ykaul

Target Milestone:

---

Target Release:

4.2

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

ceph-ansible-4.0.32-1.el8cp, ceph-ansible-4.0.32-1.el7cp

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2021-01-12 14:56:02 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1760354

Attachments:

Description	Flags
Output of per ansible task memory profile	none

Description John Fulton 2020-07-14 20:29:15 UTC

Created attachment 1701114 [details]
Output of per ansible task memory profile

While profiling memory usage of a ceph-ansible run [1] with hundreds of ceph clients it was discovered that the task "set_fact ceph_current_status (convert to json)" [2] consumed over 9 GB of RAM on the host running Ansible [3].

This code should be optimized so that it is more memory efficient.


Additional Details to consider: The task seems to store output from_json [4]. Is this data being stored on every node? Does it really need to be? Could we change the task to a module to make it more efficient?


[1] https://thesaitech.wordpress.com/2019/11/17/profiling-ansible-memory-usage-per-task/

[2] https://github.com/ceph/ceph-ansible/blob/bcc673f66c22364766beb4b5ebb971bd3f517693/roles/ceph-facts/tasks/facts.yml#L147-L152

[3] See line 1 of the attached 
2020-07-10 04:17:22,720 p=784384 u=root n=ansible | ceph-facts : set_fact ceph_current_status (convert to json) (ac1f6bc3-28e2-ef2d-49c7-000000050628): 9786.91MB

[4] https://github.com/ceph/ceph-ansible/blob/bcc673f66c22364766beb4b5ebb971bd3f517693/roles/ceph-facts/tasks/facts.yml#L108

Comment 1 RHEL Program Management 2020-07-14 20:29:23 UTC

Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Comment 5 Sai Sindhur Malleni 2020-07-16 15:34:58 UTC

This is when scaling the overcloud compute count from 472 to 474 on an already deployed ceph cluster. Also, I don't believe we were using -e ceph_ansible_limit so the client role was probably running across all the compute nodes.

I can give you that data, but note that, we've scaled that cluster further and are now at 660+ compute nodes, so this might not exactly be the output when the profiling was done at the 472 node scale.

Current output from the cluster:
https://gist.githubusercontent.com/smalleni/e8828ade26ce679df799d02847ef5035/raw/372ff53d0b6e49c8c83c932d691d8a5fd5d63f05/gistfile1.txt

Comment 9 Ben England 2020-07-16 16:53:24 UTC

what is the problem with using 9 GB of RAM for deploying a *huge* cluster?   It's only needed during the deployment, right? 
  A typical midrange server has > 100 GB of RAM.   If that's the only problem, why don't we just recommend additional memory in such cases?

Also, I thought fact gathering was only done on the ansible-playbook host and we weren't asking each host for facts about all the other hosts (which would be O(N^2)).  Is that so?

What is the --fork parameter? I saw this in e-mail chain:

"FTR In my testing, the memory consumption is likely related to the number of forks and the async waits that we perform.  Example in my basic reproducer (with a whole 3 tasks) was forks=80 against 20 hosts used ~1.5G whereas as default forks vs 20 hosts used ~900M according to cgroup_memory_recap."

so why not use --forks=40 instead?   What are the requirements on overall deploy or scale-up time and how is this impacted by --forks?  Is elapsed time proportional to 1/forks, as we might guess, or are there other factors?

RHCS 5 will be based on cephadm, not ceph-ansible, so I don't think we want to invest a ton of time in fixing ceph-ansible - we need data on cephadm scalability, right?

Comment 10 Sai Sindhur Malleni 2020-07-16 17:03:03 UTC

There is a related issue about the default ansble.cfg forks we use, which is calculated as 10*CPU_COUNT which is problematic on a 64 core undercloud, as 640 ansible forks can end up actually consuming ALL of the memory in some cases. I opened: https://bugzilla.redhat.com/show_bug.cgi?id=1857451

Comment 11 Sai Sindhur Malleni 2020-07-16 17:04:15 UTC

Actually ignore my previous comment, the forks used by ceph-ansible are different from the forks used by tripleo-ansible. Looks like my ceph_ansible_command.sh has ANSIBLE_FORKS=25

Comment 18 John Fulton 2020-08-25 15:43:18 UTC

Now that https://github.com/ceph/ceph-ansible/pull/5663 has merged into the 4 stable branch, can it into the coming v4.0.30 release and the bug moved to POST?

Comment 32 Veera Raghava Reddy 2020-11-24 05:09:11 UTC

Based on comment 30 moving to Verified.

Comment 34 errata-xmlrpc 2021-01-12 14:56:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0081