Bug 1977380

This sounds fair enough, though it is purely an optimization. My hunch is that this would not be a trivial implementation, but not untenable. Still, moving this to ODF 4.9.

We also need more Ceph-focused discussion to determine the right path here. Travis, Orit, could either of you weigh in here?

I'm thinking we should do the following:
- The OCS operator should not be setting osd_memory_target_cgroup_limit_ratio in the config overrides
- Rook's storageClassDeviceSets should have a setting, let's call it osdMemoryLimitRatio to correspond to ceph's osd_memory_target_cgroup_limit_ratio. If not set, Rook would default this to the suggested ratio depending on the memory limits, 0.5 for smaller OSDs, 0.9 for larger OSDs, etc.
- The OCS StorageCluster CR could also expose the setting if the default computation needs to be overridden.

Taking another look at this, Rook really doesn't seem like the right place to calculate a smart default. Exposing this detailed of a setting in Rook should also be avoided if possible since it's getting into the internals of ceph.

What about the following?
1. The OCS operator should allow the value of osd_memory_target_cgroup_limit_ratio to be overridden, whether this is by allowing the configmap values to not be reconciled (preferred), or by exposing a new setting in the StorageCluster CR. This would give flexibility for perf testing and advanced customers.
2. The OSDs themselves would compute a smart default themselves if the osd_memory_target_cgroup_limit_ratio value is not explicitly set. Ben is there any reason Ceph can't do this computation itself?

Let's ask them, cc'ing Josh Durgin and Sebastien Han.   rook is imposing the CGroup limit, but RHCS 5 (container-based) does this too, so seems like in this time frame we could make Ceph do it when it detects a CGroup limit for memory?  I don't care who does it as long as it gets done, that's for the engineering groups to decide.

(In reply to Travis Nielsen from comment #5)
> Taking another look at this, Rook really doesn't seem like the right place
> to calculate a smart default. Exposing this detailed of a setting in Rook
> should also be avoided if possible since it's getting into the internals of
> ceph.
> 
> What about the following?
> 1. The OCS operator should allow the value of
> osd_memory_target_cgroup_limit_ratio to be overridden, whether this is by
> allowing the configmap values to not be reconciled (preferred), or by
> exposing a new setting in the StorageCluster CR. This would give flexibility
> for perf testing and advanced customers.

+1

> 2. The OSDs themselves would compute a smart default themselves if the
> osd_memory_target_cgroup_limit_ratio value is not explicitly set. Ben is
> there any reason Ceph can't do this computation itself?

@Josh?

cgroup limits are set via pod resources from the ocs-op.
Ceph detects the variables passed by Rook, respectively POD_MEMORY_LIMIT and POD_MEMORY_REQUEST.

Ben when you said:

>However, osd_memory_target is nowhere near what I would expect:
>
>sh-4.4# for n in `seq 0 11` ; do ceph config show osd.$n osd_memory_target ; done
>42949672960

It's expected. Ceph currently looks up the value of osd_memory_target_cgroup_limit_ratio then computes it by the POD_MEMORY_LIMIT value. So in your case:
osd_memory_target_cgroup_limit_ratio (0.5) * POD_MEMORY_LIMIT (85899345920) = 42949672960

Now for "big" configuration, the default value for osd_memory_target_cgroup_limit_ratio is already 0.8 which seems high enough.

Now for "small" configuration, if I read you correctly, Ceph should lower the default value of osd_memory_target_cgroup_limit_ratio when POD_MEMORY_LIMIT is around 5GB which seems acceptable to me if this is what the performance results are showing.
When it's done we can remove ocs-op default's value for osd_memory_target_cgroup_limit_ratio.
Now ocs-op probably sets osd_memory_target_cgroup_limit_ratio to 0.5 because most envs we deploy on don't have that much memory.
Recently, in https://bugzilla.redhat.com/show_bug.cgi?id=1914475 ocs-op has allowed overriding the CM rook-config-override when a specific deployment strategy is used.
So I believe you can already set osd_memory_target_cgroup_limit_ratio to 0.9 if you want.


Ben, does that help?

Created a clone for 4.8.z https://bugzilla.redhat.com/show_bug.cgi?id=1991458

If you want IBM cloud to utilize ODF at large scale, we had better get our efficiency act together.  IBM Cloud offers 16-TB volumes as OSDs, ask Elvir Kuric, and a Ceph OSD requires more than 2.5 GB of RAM to service a volume that big, just to cache metadata!  Ask Mark Nelson or Josh Durgin.  You can have all the functionality in the world, but if ODF runs really slowly no one will want it for anything more than a toy configuration.

As for bz 1914475, this is a unsupported, undocumented backdoor.  So if your primary way to support large ODF sites is this, that's defeating the whole point of ODF, which was to make it easy to deploy Ceph. 

We need to adjust osd_memory_target in an opinionated way, and I'm suggesting above how you could do that safely and effectively.  Perhaps the initial post wasn't clear enough.  I want to ensure that there is at least 2 GB separating osd_memory_target and CGroup limit = K8S memory:limit, to prevent short-term memory spikes from triggering an OSD OOM.   With this formula, for a CGroup limit of 5 GB (current default), osd_memory_target should be 3 GB, not that different from current default of 2.5 GB.  For a CGroup limit ratio of 10 GB, osd_memory_target should be 8 GB.   So by doubling your memory:limit, you get 3x the amount of *usable* memory for each OSD, which should have a very beneficial effect on metadata caching in the OSD.   That's why this is important for sites with large OSD volumes.

sorry, posted prematurely ;-)  I meant "ensure that there is 2.5 GB separating OSD memory target and CGroup limit"  So for CGroup limit of 5 GB, osd_memory_target should be 2.5 GB, same as current default.   For doubled CGroup limit of 10 GB, osd_memory_target would be 7.5 GB, 3 times what it was before, and 75% efficient.  As Cgroup limit climbs, efficiency climbs.  The admin can adjust the CGroup limit today via the CR in the StorageDeviceSet.

Sorry for the late response.

We do agree that this optimization would be very beneficial tot he product. That said, we certainly don't have time to do so for ODF 4.9, so moving it to ODF 4.10 for further consideration.

I'm also renaming this BZ as an RFE. Eran, please weigh in on how much priority this should take in our planning.

lowering the queue depth from 16 to 2 for my 9-pod test results in stable numbers for 16-KiB randwrites, while getting reasonably good throughput.   We need to retest across the whole range of parameters to verify, but am getting excellent stability from the tests this way.

Now for the actual regression numbers, from here:

https://docs.google.com/spreadsheets/d/1hnq-N4eEEvpxg82JYAFTsRSHMSMHn3OGLAWTAqrEQsU/edit#gid=1049516327

It shows that for 16-KiB size (drum roll....):

randread throughput increased by 10% (yay!)
randwrite throughput decreased by 30% (boo!)

These numbers are very consistent and the percent deviation of the samples is VERY LOW.  But why?   I see that more reads are happening during randwrite test with ODF 4.9:

with OCS 4.8:  http://nfs-storage01.scalelab.redhat.com/bengland/public/openshift/ripsaw/fio-logs/2021-11-16-14-32/Screenshot%20from%202021-11-16%2017-13-20.png
with ODF 4.9:  http://nfs-storage01.scalelab.redhat.com/bengland/public/openshift/ripsaw/fio-logs/2021-11-17-17-45/iops-over-time.png

To get these PromQL graphs, go to the openshift console and replace everything to the right of the hostname in the URL with:

monitoring/query-browser?query0=irate%28node_disk_writes_completed_total%7Bdevice%3D~"nvme%5B0-9%5Dn1"%7D%5B1m%5D%29&query1=irate%28node_disk_reads_completed_total%7Bdevice%3D~"nvme%5B0-9%5Dn1"%7D%5B1m%5D%29

I see that more reads are happening with ODF 4.9 during randwrite test than happened with 4.8.  If so, this could explain the regression.

next step is to run this test with higher mem limits and see if this difference goes away.   If so, the solution might be to just give OSDs a little more memory to account for changes happening in RHCS 5 = Ceph Pacific, which wouldn't be so bad.  We can investigate why this might have happened, but the important thing for now is to stop the regression so we can  release 4.9.  Right now I've altered the ocs-storagecluster CR as shown here:

[bengland@localhost ripsaw]$ ocos get storagecluster ocs-storagecluster -o json | jq .spec.storageDeviceSets[0].resources
{
  "limits": {
    "cpu": "2",
    "memory": "10Gi"
  },
  "requests": {
    "cpu": "2",
    "memory": "10Gi"
  }
}

So now I get OSDs with more memory and CPU:

[bengland@localhost ripsaw]$ oc -n openshift-storage get pod rook-ceph-osd-2-86dcbc5fbb-wfvnp -o json | jq .spec.containers[0].resources
{
  "limits": {
    "cpu": "2",
    "memory": "10Gi"
  },
  "requests": {
    "cpu": "2",
    "memory": "10Gi"
  }
}

rerunning test now.   If this works, I'll experiment with different amounts of memory to see how much is actually needed.   Shekhar Berry has already shed light on this with his bz, 

https://bugzilla.redhat.com/show_bug.cgi?id=2012082

ignore last reply to this bz, it was intended for a different bz!

We can start with setting osd_memory_target_cgroup_limit_ratio to 0 to disable the feature. We set the memory requests and limits to the same value in OCS/ODF. It should be the simplest fix

@Ben, can you try with setting osd_memory_target_cgroup_limit_ratio to 0?
In ODF there is no reason to use it as we set the same memory requests and limits on the OSD pod.

The reason we have left some buffer in OSD by setting osd_memory_target_cgroup_limit_ratio to 0.5 is to prevent OSDs getting OOM kill. This can happen when there is burst of high IO causing transitory memory spikes in OSD memory usage causing K8s to kill the OSD pod. 

So, IMO if we set osd_memory_target_cgroup_limit_ratio to 0 the possibility of OSD getting OOM kill is significantly increased. 

On the other hand for large OSD memory size (especially in baremetal environments) we would waste a lot of memory if osd_memory_target_cgroup_limit_ratio is set to 0.5 as explained in comment 0 by Ben. 

I am ok with the solution proposed by Ben in comment 0, that would work well for both large and small memory size.

Just to repeat what Ben says:

Set osd_memory_target_cgroup_limit_ratio to 0.9

osd_memory_target = (OSD_limit * osd_memory_target_cgroup_limit_ratio) - 1.5GB

What do you think of this solution Orit?

After consulting with Josh Durgin (Rados). 
Using the Ceph default of 0.8 with OCS default of 5G will get a 1G of buffer that would be more than enough and there will be no risk of OOM.
I recommend just using Ceph default as it will most likely be update if needed and it will be the simplest.

@owasserm A 20% buffer is a good default starting point, but it in no way guarantees no risk of OOM.  Rather it was enough to prevent OOM in the cases we tested upstream with a default 4GB osd_memory_target and also using the default tcmalloc memory allocator.  It's possible that there are corner cases that could cause extreme memory fragmentation that could still push a given OSD over that limit (even temporarily) and a hard cap at 20% per-OSD could result in OOM.  It may be rare, but it could still happen.

This is one area where containers with hard memory limits suffer vs contains with no limits (or non-container installs).  Ceph's memory autotuning works well to keep daemons within a reasonable average memory limit over time, but it's too CPU intensive to have it react to extreme transient memory spikes  (this is all configurable via config parameters btw).  Having it react too quickly also means we thrash the caches every time something not controlled by the priority cache uses too much memory.  When memory limits are shared across OSDs, these transient memory spikes are much less likely to cause a global OOM event than when implementing per-container or per-osd quotas.

To illustrate the point, here's a look at mapped osd memory from Ceph master from 20210923 during interleaved RBD 4K random read/write and RGW 4K object puts/gets/deletes.


https://docs.google.com/spreadsheets/d/1lSp2cLzYmRfPILDCyLMXciIfdf0OvSFngwXukQFXIqQ/edit#gid=1834043199


In most cases we keep the mapped* osd memory overage well below 20%.  There is a single point during a transition from RGW deletes to RBD 4k random reads (around the 1680 second mark) where the overage spikes to nearly the full 20%.  Very small RGW object workloads tend to stress memory allocation far more than RBD so this isn't totally unexpected.  That kind of extreme workload transition especially combined with very high IO depth are when we might see the overage potentially go beyond 20%.  Having said that, this test was designed specifically to stress memory allocation behavior so in practice even large spikes may be rarer on properly configured** systems than this test indicates.

* mapped memory is heap memory - unmapped memory, but it's not the same as RSS memory used when viewed with say top.  Even if we tell the kernel we don't want some region of memory anymore there's no guarantee that the kernel will (or even can) reclaim it.

** meaning huge pages are not being improperly used, tcmalloc is used with enough thread cache, The OSD memory target is 4GB+, rocksdb is not throttling, etc.

We always use tcmalloc in ODF so that is good news. Another thing is that the workload used on ODF are not heavy on I/O and each the disk size we support per OSD is up to 4T.
All those make OOM very unlikely.

I feel comfortable taking this in for ODF 4.9.z, the development work should be fairly trivial. It would come down to QE how much they're willing to take on the validation.

(In reply to Jose A. Rivera from comment #26)
> I feel comfortable taking this in for ODF 4.9.z, the development work should
> be fairly trivial. It would come down to QE how much they're willing to take
> on the validation.

QE?

Clarification on status of the bug:

- Jose subscribed (comment 26) to implement changes summarized in comment 21, does that still hold?
- Do we plant to have performance testing done by a perf team for this change?
- Are we comfortable to not having this change covered by a performance tests?

QE team would cover this by standard regressions run, but on top of that, I assume that we should
try additional standard memory sizes of storage node, making sure that nothing is OOM killed
by mistake.

Fixed as part of this dev preview epic.
https://issues.redhat.com/browse/RHSTOR-2517

The original request was to automatically tune the configuration.
But instead of that we have provided a way to configure it manually as per https://bugzilla.redhat.com/show_bug.cgi?id=1977380#c30

Requesting qa_ack based on the same as the above feature is already part of 4.12