Describe the issue:
With RHCS 4, we don't have a clear documentation for NUMA configurations.
For example : Pinning an OSD to specific CPU.
From the ceph OSD configs, auto-affinity is enabled but it couldn't be validated.
# ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep numa
"osd_numa_auto_affinity": "true",
"osd_numa_node": "-1",
"osd_numa_prefer_iface": "true",
This article helped to understand the highlights in Nautilus,
https://ceph.io/community/new-in-nautilus-rados-highlights/
However the `ceph osd numa-status` comes empty.
# ceph osd numa-status
OSD HOST NETWORK STORAGE AFFINITY CPUS
0 ceph81 - - - -
1 ceph80 - - - -
2 ceph82 - - - -
3 ceph81 - - - -
4 ceph80 - - - -
5 ceph82 - - - -
Also a runtime change is not possible.
# ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set osd_numa_node 0
{
"error": "error setting 'osd_numa_node' to '0': (38) Function not implemented"
}
# ceph tell osd.1 config set osd_numa_node 0
Error ENOSYS: Configuration option 'osd_numa_node' may not be modified at runtime
We manually added the osd numa node in ceph.conf and restarted osd, which does apply the changes. But not able to validate it properly.
Document URL:
RHCS 4 Administration Guide
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/administration_guide/index
Chapter/Section Number and Title:
Product Version: RHCS 4.1
We need a document for configuring and validation of NUMA settings for OSD's.
Describe the issue: With RHCS 4, we don't have a clear documentation for NUMA configurations. For example : Pinning an OSD to specific CPU. From the ceph OSD configs, auto-affinity is enabled but it couldn't be validated. # ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep numa "osd_numa_auto_affinity": "true", "osd_numa_node": "-1", "osd_numa_prefer_iface": "true", This article helped to understand the highlights in Nautilus, https://ceph.io/community/new-in-nautilus-rados-highlights/ However the `ceph osd numa-status` comes empty. # ceph osd numa-status OSD HOST NETWORK STORAGE AFFINITY CPUS 0 ceph81 - - - - 1 ceph80 - - - - 2 ceph82 - - - - 3 ceph81 - - - - 4 ceph80 - - - - 5 ceph82 - - - - Also a runtime change is not possible. # ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set osd_numa_node 0 { "error": "error setting 'osd_numa_node' to '0': (38) Function not implemented" } # ceph tell osd.1 config set osd_numa_node 0 Error ENOSYS: Configuration option 'osd_numa_node' may not be modified at runtime We manually added the osd numa node in ceph.conf and restarted osd, which does apply the changes. But not able to validate it properly. Document URL: RHCS 4 Administration Guide https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/administration_guide/index Chapter/Section Number and Title: Product Version: RHCS 4.1 We need a document for configuring and validation of NUMA settings for OSD's.