Bug 2214499

Description of problem (please be detailed as possible and provide log
snippests):
===========================================================================
On an ODF cluster with the following cluster-level parameters enabled:
FIPS
Hugepages
KMS - vault
Cluster-wide encryption
Encryption in transit

The ceph-client.admin crashed in the ceph-exporter thread with the error message "throw_invalid_argument(char const*, boost::source_location const&)+0x37) [0x557c40cab267]" while filling the cluster to the near full ratio.

Please note that the cluster had not reached the near full ratio at the time of the crash. The OSDs were filled up to 62.2%.

backtrace:

2023-06-12T20:38:44.709+0000 7f9898b80e80 -1 *** Caught signal (Aborted) **
 in thread 7f9898b80e80 thread_name:ceph-exporter

 ceph version 17.2.6-70.el9cp (fe62dcdbb2c6e05782a3e2b67d025b84ff5047cc) quincy (stable)
 1: /lib64/libc.so.6(+0x54df0) [0x7f9899274df0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f98992c154c]
 3: raise()
 4: abort()
 5: /lib64/libstdc++.so.6(+0xa1a01) [0x7f98994e5a01]
 6: /lib64/libstdc++.so.6(+0xad37c) [0x7f98994f137c]
 7: /lib64/libstdc++.so.6(+0xad3e7) [0x7f98994f13e7]
 8: /lib64/libstdc++.so.6(+0xad649) [0x7f98994f1649]
 9: ceph-exporter(+0x2a218) [0x557c40c98218]
 10: (boost::json::detail::throw_invalid_argument(char const*, boost::source_location const&)+0x37) [0x557c40cab267]
 11: ceph-exporter(+0x683c7) [0x557c40cd63c7]
 12: (DaemonMetricCollector::dump_asok_metrics()+0x22c8) [0x557c40cbae98]
 13: ceph-exporter(+0x41230) [0x557c40caf230]
 14: ceph-exporter(+0x60b7d) [0x557c40cceb7d]
 15: ceph-exporter(+0xace0f) [0x557c40d1ae0f]
 16: (DaemonMetricCollector::main()+0x212) [0x557c40c9dc32]
 17: main()
 18: /lib64/libc.so.6(+0x3feb0) [0x7f989925feb0]
 19: __libc_start_main()
 20: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.


Version of all relevant components (if applicable):
OCP: 4.13.0-0.nightly-2023-06-09-152551
ODF: 4.13.0-218

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
ceph health went to warning state, other than that I did not observe any functionality impact, atleast from the positive scenarios.

Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Reporting at the first hit

Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
===================
1) Create an ODF cluster with the cluster level parms 1) FIPS 2) Hugepages 3) KMS - vault 4) cluster wide encrption 5) encryption in transit enabled 

2) Run automated system test - https://github.com/red-hat-storage/ocs-ci/blob/master/tests/e2e/system_test/test_cluster_full_and_recovery.py

The automated test executes below steps,
a) Create PVC1 [FS + RBD]
b) Verify new PVC1 [FS + RBD] on Bound state
c) Run FIO on PVC1_FS + PVC1_RBD
d) Calculate Checksum PVC1_FS + PVC1_RBD
e) Fill the cluster to “Full ratio” (usually 85%) using the benchmark-operator

Actual results:
===============
ceph-client.admin crashed in ceph-exporter thread with "throw_invalid_argument(char const*, boost::source_location const&)+0x37) [0x557c40cab267]" while filling the cluster to the near full ratio.


Expected results:
=================
No crashes should be observed during IO write operations.