Bug 2250995

+++ This bug was initially created as a clone of Bug #2248850 +++

Description of problem (please be detailed as possible and provide log
snippests):
rook-ceph-exporter restarts multiple times


rook-ceph-exporter logs:

2023-11-09T08:29:35.751+0000 7f75b4165e80 -1 asok(0x55d8a71f94d0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.asok': (17) File exists
system:0
*** Caught signal (Segmentation fault) **
 in thread 7f75b4165e80 thread_name:ceph-exporter
 ceph version 17.2.6-148.el9cp (badc1d27cb07762bea48f6554ad4f92b9d3fbb6b) quincy (stable)
 1: /lib64/libc.so.6(+0x54df0) [0x7f75b4859df0]
 2: (std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x24) [0x7f75b4aef034]
 3: (DaemonMetricCollector::dump_asok_metrics()+0x1fe) [0x55d8a6329e2e]
 4: ceph-exporter(+0x45eb0) [0x55d8a632beb0]
 5: ceph-exporter(+0x5cb1d) [0x55d8a6342b1d]
 6: ceph-exporter(+0xacb9f) [0x55d8a6392b9f]
 7: (DaemonMetricCollector::main()+0x212) [0x55d8a6315c22]
 8: main()
 9: /lib64/libc.so.6(+0x3feb0) [0x7f75b4844eb0]
 10: __libc_start_main()
 11: _start()
2023-11-09T08:29:35.752+0000 7f75b4165e80 -1 *** Caught signal (Segmentation fault) **
 in thread 7f75b4165e80 thread_name:ceph-exporter

 ceph version 17.2.6-148.el9cp (badc1d27cb07762bea48f6554ad4f92b9d3fbb6b) quincy (stable)
 1: /lib64/libc.so.6(+0x54df0) [0x7f75b4859df0]
 2: (std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x24) [0x7f75b4aef034]
 3: (DaemonMetricCollector::dump_asok_metrics()+0x1fe) [0x55d8a6329e2e]
 4: ceph-exporter(+0x45eb0) [0x55d8a632beb0]
 5: ceph-exporter(+0x5cb1d) [0x55d8a6342b1d]
 6: ceph-exporter(+0xacb9f) [0x55d8a6392b9f]
 7: (DaemonMetricCollector::main()+0x212) [0x55d8a6315c22]
 8: main()
 9: /lib64/libc.so.6(+0x3feb0) [0x7f75b4844eb0]
 10: __libc_start_main()
 11: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

    -3> 2023-11-09T08:29:35.751+0000 7f75b4165e80 -1 asok(0x55d8a71f94d0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.asok': (17) File exists


$ ceph status
  cluster:
    id:     f112b846-0527-4ef9-ac6e-519e7011a676
    health: HEALTH_WARN
            2 daemons have recently crashed

  services:
    mon: 3 daemons, quorum a,b,c (age 2h)
    mgr: a(active, since 2h)
    mds: 1/1 daemons up, 1 hot standby
    osd: 15 osds: 15 up (since 2h), 15 in (since 2h)

  data:
    volumes: 1/1 healthy
    pools:   4 pools, 642 pgs
    objects: 201 objects, 346 MiB
    usage:   800 MiB used, 105 TiB / 105 TiB avail
    pgs:     642 active+clean

  io:
    client:   852 B/s rd, 8.3 KiB/s wr, 1 op/s rd, 1 op/s wr




$ ceph crash ls
ID                                                                ENTITY        NEW
2023-11-09T06:46:08.690038Z_ceec941c-4c1c-4b60-b037-efc3cc25905b  client.admin   *
2023-11-09T06:46:09.706085Z_30028a56-be4f-45c5-bf0c-1000706ce9b9  client.admin   *

$ ceph crash info 2023-11-09T06:46:08.690038Z_ceec941c-4c1c-4b60-b037-efc3cc25905b
{
    "backtrace": [
        "/lib64/libc.so.6(+0x54df0) [0x7f4c6a384df0]",
        "(std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x24) [0x7f4c6a61a034]",
        "(DaemonMetricCollector::dump_asok_metrics()+0x1fe) [0x559c0f26ae2e]",
        "ceph-exporter(+0x45eb0) [0x559c0f26ceb0]",
        "ceph-exporter(+0x5cb1d) [0x559c0f283b1d]",
        "ceph-exporter(+0xacb9f) [0x559c0f2d3b9f]",
        "(DaemonMetricCollector::main()+0x212) [0x559c0f256c22]",
        "main()",
        "/lib64/libc.so.6(+0x3feb0) [0x7f4c6a36feb0]",
        "__libc_start_main()",
        "_start()"
    ],
    "ceph_version": "17.2.6-148.el9cp",
    "crash_id": "2023-11-09T06:46:08.690038Z_ceec941c-4c1c-4b60-b037-efc3cc25905b",
    "entity_name": "client.admin",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.2 (Plow)",
    "os_version_id": "9.2",
    "process_name": "ceph-exporter",
    "stack_sig": "445f7e928870d7f3a4ac83dd88c42c1ea1b453f27da54f8999b3570b25614589",
    "timestamp": "2023-11-09T06:46:08.690038Z",
    "utsname_hostname": "compute-1-ru5.rackm01.rtp.raleigh.ibm.com",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.36.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Oct 5 08:11:31 EDT 2023"
}




Version of all relevant components (if applicable):
ODF: 4.14.0-162


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
No

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
Yes, occurring on all fresh installed HCI clusters

Can this issue reproduce from the UI?



Steps to Reproduce:
1.Install ODF
2.Create storagecluster


Actual results:
Storage Cluster created and Ceph Cluster is in Health_Warn
Expected results:
Storage Cluster created and Ceph Cluster is in Health_OK

--- Additional comment from RHEL Program Management on 2023-11-09 09:33:38 UTC ---

This bug having no release flag set previously, is now set with release flag 'odf‑4.15.0' to '?', and so is being proposed to be fixed at the ODF 4.15.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.

--- Additional comment from avan on 2023-11-10 10:05:05 UTC ---

 Can you please confirm with Ken dreyer the Ceph image you using has Ceph v6.1z2?

--- Additional comment from Rohan Gupta on 2023-11-10 14:19:47 UTC ---

@athakkar 
ceph version 17.2.6-148.el9cp (badc1d27cb07762bea48f6554ad4f92b9d3fbb6b) quincy (stable)
Image is cp.icr.io/cp/ibm-ceph/ceph-6-rhel9@sha256:162ce3abf5e4adc81e6b4957b22caddbb7a87ae30340348a06e21b7c8123b86e 
will check with Ken and update here

--- Additional comment from Juan Miguel Olmo on 2023-11-13 11:58:54 UTC ---

I do not know what could be the root cause for having a socket file already created when Ceph exporter starts, But apart of solve what seems to be the root cause of the problem (why we are trying to start ceph exporter twice or more times). 
I would suggest to manage the error properly in the Ceph exporter. To have the socket file in use must no cause a seg. fault error, probably just an error line in the log.

--- Additional comment from Leela Venkaiah Gangavarapu on 2023-11-17 03:58:39 UTC ---

tracker for https://github.ibm.com/ProjectAbell/abell-tracking/issues/29079

--- Additional comment from avan on 2023-11-20 12:02:43 UTC ---

(In reply to Juan Miguel Olmo from comment #4)
> I do not know what could be the root cause for having a socket file already
> created when Ceph exporter starts, But apart of solve what seems to be the
> root cause of the problem (why we are trying to start ceph exporter twice or
> more times). 
> I would suggest to manage the error properly in the Ceph exporter. To have
> the socket file in use must no cause a seg. fault error, probably just an
> error line in the log.

The error isn't coming from ceph exporter, its the admin socket file. Debugging why this fails only on this env & works fine on others.

--- Additional comment from Divyansh Kamboj on 2023-11-20 17:16:57 UTC ---

update on the bug, the issue is that ceph-exporter crashes when it tries to create the admin socket file when it restarts, but the file is already created by either already existing exporter or some other daemon. @athakkar is working on renaming the socket file to avoid the conflict.

--- Additional comment from Juan Miguel Olmo on 2023-11-21 09:17:24 UTC ---

(In reply to avan from comment #6)
> (In reply to Juan Miguel Olmo from comment #4)
> > I do not know what could be the root cause for having a socket file already
> > created when Ceph exporter starts, But apart of solve what seems to be the
> > root cause of the problem (why we are trying to start ceph exporter twice or
> > more times). 
> > I would suggest to manage the error properly in the Ceph exporter. To have
> > the socket file in use must no cause a seg. fault error, probably just an
> > error line in the log.
> 
> The error isn't coming from ceph exporter, its the admin socket file.
> Debugging why this fails only on this env & works fine on others.

The exporter explodes with a "segmentation fault" error because the socket cannot be used. Probably a better error management can replace the "explosion" by log lines and a standby behavior.
Do you think that the ceph exporter behavior is robust in this specific case?

--- Additional comment from avan on 2023-11-21 09:51:30 UTC ---

I've raised the fix for the issue but seems like still some issues with caps for exporter https://github.com/rook/rook/pull/13239

--- Additional comment from Juan Miguel Olmo on 2023-11-21 10:00:11 UTC ---

Is good to have a fix in OCS to avoid happen the issue again, but it would be awesome to have a more robust behavior in the Ceph exporter and unit tests to avoid explosions like this:

2023-11-09T08:29:35.751+0000 7f75b4165e80 -1 asok(0x55d8a71f94d0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.asok': (17) File exists
system:0
*** Caught signal (Segmentation fault) **
 in thread 7f75b4165e80 thread_name:ceph-exporter
 ceph version 17.2.6-148.el9cp (badc1d27cb07762bea48f6554ad4f92b9d3fbb6b) quincy (stable)
 1: /lib64/libc.so.6(+0x54df0) [0x7f75b4859df0]
 2: (std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x24) [0x7f75b4aef034]
 3: (DaemonMetricCollector::dump_asok_metrics()+0x1fe) [0x55d8a6329e2e]
 4: ceph-exporter(+0x45eb0) [0x55d8a632beb0]
 5: ceph-exporter(+0x5cb1d) [0x55d8a6342b1d]
 6: ceph-exporter(+0xacb9f) [0x55d8a6392b9f]
 7: (DaemonMetricCollector::main()+0x212) [0x55d8a6315c22]
 8: main()
 9: /lib64/libc.so.6(+0x3feb0) [0x7f75b4844eb0]
 10: __libc_start_main()
 11: _start()
2023-11-09T08:29:35.752+0000 7f75b4165e80 -1 *** Caught signal (Segmentation fault) **
 in thread 7f75b4165e80 thread_name:ceph-exporter

--- Additional comment from avan on 2023-11-21 10:10:37 UTC ---

(In reply to Juan Miguel Olmo from comment #10)
> Is good to have a fix in OCS to avoid happen the issue again, but it would
> be awesome to have a more robust behavior in the Ceph exporter and unit
> tests to avoid explosions like this:
> 
> 2023-11-09T08:29:35.751+0000 7f75b4165e80 -1 asok(0x55d8a71f94d0)
> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
> bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.asok': (17)
> File exists
> system:0
> *** Caught signal (Segmentation fault) **
>  in thread 7f75b4165e80 thread_name:ceph-exporter
>  ceph version 17.2.6-148.el9cp (badc1d27cb07762bea48f6554ad4f92b9d3fbb6b)
> quincy (stable)
>  1: /lib64/libc.so.6(+0x54df0) [0x7f75b4859df0]
>  2: (std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x24) [0x7f75b4aef034]
>  3: (DaemonMetricCollector::dump_asok_metrics()+0x1fe) [0x55d8a6329e2e]
>  4: ceph-exporter(+0x45eb0) [0x55d8a632beb0]
>  5: ceph-exporter(+0x5cb1d) [0x55d8a6342b1d]
>  6: ceph-exporter(+0xacb9f) [0x55d8a6392b9f]
>  7: (DaemonMetricCollector::main()+0x212) [0x55d8a6315c22]
>  8: main()
>  9: /lib64/libc.so.6(+0x3feb0) [0x7f75b4844eb0]
>  10: __libc_start_main()
>  11: _start()
> 2023-11-09T08:29:35.752+0000 7f75b4165e80 -1 *** Caught signal (Segmentation
> fault) **
>  in thread 7f75b4165e80 thread_name:ceph-exporter

The issue isn;t coming from ceph-exporter here as it seems to work fine on other env for ODF & also in standalone Ceph. This is very specific case where it seems fix should be in Rook only. 
I don't follow how exporter unit test would have help given traceback is already coming from AdminSocket

--- Additional comment from avan on 2023-11-21 10:42:14 UTC ---

(In reply to Juan Miguel Olmo from comment #8)
> (In reply to avan from comment #6)
> > (In reply to Juan Miguel Olmo from comment #4)
> > > I do not know what could be the root cause for having a socket file already
> > > created when Ceph exporter starts, But apart of solve what seems to be the
> > > root cause of the problem (why we are trying to start ceph exporter twice or
> > > more times). 
> > > I would suggest to manage the error properly in the Ceph exporter. To have
> > > the socket file in use must no cause a seg. fault error, probably just an
> > > error line in the log.
> > 
> > The error isn't coming from ceph exporter, its the admin socket file.
> > Debugging why this fails only on this env & works fine on others.
> 
> The exporter explodes with a "segmentation fault" error because the socket
> cannot be used. Probably a better error management can replace the
> "explosion" by log lines and a standby behavior.
> Do you think that the ceph exporter behavior is robust in this specific case?

Well exporter crashes because socket is in use already, not by exporter itself because it works fine in vstart & mstart standalone Ceph env which will generate same socket file name for exporter (client.admin.asok).
So I'd put this way: The issue seems more about deployment of exporter for mentioned env and not of exporter itself. Simply having specific user-keyring pair for exporter in Rook like we have for Cephadm[1] should avoid crashes we facing. 

[1] https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/services/cephadmservice.py#L1168C11-L1168C11.

--- Additional comment from avan on 2023-11-21 19:18:41 UTC ---

Upstream fix is merged. Created the backport for 4.14 https://github.com/red-hat-storage/rook/pull/541