Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2102485

Summary: prometheus can't rm successfully and keep status <deleting>
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: shiqi <qshi>
Component: CephadmAssignee: Adam King <adking>
Status: CLOSED DUPLICATE QA Contact: Manasa <mgowri>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 5.1CC: cephqe-warriors, gjose, saraut
Target Milestone: ---   
Target Release: 6.1   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-03-31 19:14:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description shiqi 2022-06-30 04:17:58 UTC 
Description of problem:
rm prometheus failed
the prometheus status keep showing <deleting> in the cephadm(orch ls)
Restarting all ceph nodes does not solve the problem.prometheus status keep showing <deleting>

Version-Release number of selected component (if applicable):
ceph version 16.2.7-98.el8cp (b20d33c3b301e005bed203d3cad7245da3549f80) pacific (stable)

How reproducible:
the prometheus status keep showing <deleting> in the customer environment.
But I haven't reproduced in my customer environment

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 shiqi 2022-07-07 04:53:28 UTC 
the customer reboot all the ceph node. But the prometheus status in cephadm still show <deleting>

[PROD] [root@tr15elceph101] ~ # uptime
 10:10:25 up 1 min,  1 user,  load average: 0.20, 0.09, 0.03

[PROD] [root@tr15elceph102] ~ # uptime
 10:10:30 up 1 min,  1 user,  load average: 0.59, 0.31, 0.12

[PROD] [root@tr15elceph103] ~ # uptime
 10:10:32 up 1 min,  1 user,  load average: 0.68, 0.40, 0.16

[PROD] [root@tr15elceph101] ~ # ceph -s
  cluster:
    id:     8df43a2c-e65b-11ec-aca4-005056bc3473
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum tr15elceph101,tr15elceph102,tr15elceph103 (age 84s)
    mgr: tr15elceph101.uiqgjk(active, since 79s), standbys: tr15elceph102.maxxqs, tr15elceph103.sztndc
    osd: 9 osds: 9 up (since 73s), 9 in (since 2w)
    rgw: 3 daemons active (3 hosts, 1 zones)

  data:
    pools:   7 pools, 145 pgs
    objects: 240 objects, 27 KiB
    usage:   638 MiB used, 2.6 TiB / 2.6 TiB avail
    pgs:     145 active+clean

[PROD] [root@tr15elceph101] ~ # systemctl list-units | grep prom
[PROD] [root@tr15elceph101] ~ # systemctl list-units -a | grep prom
[PROD] [root@tr15elceph101] ~ # 

[PROD] [root@tr15elceph101] ~ # ceph orch ls
NAME           PORTS        RUNNING  REFRESHED   AGE  PLACEMENT
alertmanager   ?:9093,9094      1/1  2w ago      4w   count:1
crash                           2/3  2w ago      4w   *
grafana        ?:3000           1/1  2w ago      4w   count:1
mgr                             3/3  2w ago      4w   label:mgr
mon                             3/3  2w ago      4w   label:mon
node-exporter  ?:9100           3/3  2w ago      4w   *
osd.hdd                           9  2w ago      2w   tr15elceph101;tr15elceph102;tr15elceph103
prometheus     ?:9095           0/1  <deleting>  2d   count:1
rgw.rgw        ?:8000           3/3  2w ago      4w   label:rgw

[PROD] [root@tr15elceph101] ~ # ceph orch ps
NAME                          HOST           PORTS        STATUS        REFRESHED  AGE  MEM USE  MEM LIM  VERSION           IMAGE ID      CONTAINER ID
alertmanager.tr15elceph101    tr15elceph101  *:9093,9094  running (2w)     2w ago   4w    44.4M        -  0.21.0            7f3265d3cb44  7156895286ff
crash.tr15elceph101           tr15elceph101               unknown          2w ago   4w        -        -  <unknown>         <unknown>     <unknown>
crash.tr15elceph102           tr15elceph102               running (4w)     2w ago   4w    6937k        -  16.2.7-112.el8cp  854c5a170572  8e23c367b059
crash.tr15elceph103           tr15elceph103               running (4w)     2w ago   4w    7516k        -  16.2.7-112.el8cp  854c5a170572  a47d838d1aea
grafana.tr15elceph101         tr15elceph101  *:3000       running (2w)     2w ago   4w    89.5M        -  7.2.0             4a9624f3a08c  89c0243b2c1e
mgr.tr15elceph101.uiqgjk      tr15elceph101  *:9283       running (2w)     2w ago   4w     528M        -  16.2.7-112.el8cp  854c5a170572  dcf9ff20074f
mgr.tr15elceph102.maxxqs      tr15elceph102  *:8443,9283  running (2w)     2w ago   4w     401M        -  16.2.7-112.el8cp  854c5a170572  1a905de8eada
mgr.tr15elceph103.sztndc      tr15elceph103  *:8443,9283  running (3w)     2w ago   4w     477M        -  16.2.7-112.el8cp  854c5a170572  ed432d041912
mon.tr15elceph101             tr15elceph101               running (2w)     2w ago   4w     214M    2048M  16.2.7-112.el8cp  854c5a170572  39e2f7d0c436
mon.tr15elceph102             tr15elceph102               running (4w)     2w ago   4w    2526M    2048M  16.2.7-112.el8cp  854c5a170572  4777ff01db17
mon.tr15elceph103             tr15elceph103               running (4w)     2w ago   4w    3088M    2048M  16.2.7-112.el8cp  854c5a170572  3f4aafe75156
node-exporter.tr15elceph101   tr15elceph101  *:9100       running (2w)     2w ago   4w    23.2M        -  1.0.1             d3ae0bfa2913  cd28ecff24f9
node-exporter.tr15elceph102   tr15elceph102  *:9100       running (4w)     2w ago   4w    29.3M        -  1.0.1             d3ae0bfa2913  cb1fbe8a3cb6
node-exporter.tr15elceph103   tr15elceph103  *:9100       running (4w)     2w ago   4w    32.9M        -  1.0.1             d3ae0bfa2913  3d03bb7e7a1d
osd.0                         tr15elceph101               running (2w)     2w ago   2w     232M    4096M  16.2.7-112.el8cp  854c5a170572  3c7ef3b23c4a
osd.1                         tr15elceph101               running (2w)     2w ago   2w     271M    4096M  16.2.7-112.el8cp  854c5a170572  5e093c08a9e9
osd.2                         tr15elceph101               running (2w)     2w ago   2w     161M    4096M  16.2.7-112.el8cp  854c5a170572  0b3d45ce2807
osd.3                         tr15elceph102               running (2w)     2w ago   4w    2140M    4096M  16.2.7-112.el8cp  854c5a170572  0440827e87ce
osd.4                         tr15elceph102               running (2w)     2w ago   4w    2058M    4096M  16.2.7-112.el8cp  854c5a170572  c7fb9070fa7a
osd.5                         tr15elceph102               running (2w)     2w ago   4w    2037M    4096M  16.2.7-112.el8cp  854c5a170572  73b3332aa625
osd.6                         tr15elceph103               running (4w)     2w ago   4w    13.7G    4096M  16.2.7-112.el8cp  854c5a170572  bf748195358c
osd.7                         tr15elceph103               running (4w)     2w ago   4w    9145M    4096M  16.2.7-112.el8cp  854c5a170572  35d6df7a71d9
osd.8                         tr15elceph103               running (4w)     2w ago   4w    12.1G    4096M  16.2.7-112.el8cp  854c5a170572  78e39851a220
rgw.rgw.tr15elceph101.qupihw  tr15elceph101  *:8000       running (2w)     2w ago   4w    91.3M        -  16.2.7-112.el8cp  854c5a170572  5d3deabf2e78
rgw.rgw.tr15elceph102.fsxuid  tr15elceph102  *:8000       running (4w)     2w ago   4w     100M        -  16.2.7-112.el8cp  854c5a170572  552554df8938
rgw.rgw.tr15elceph103.lpsgpy  tr15elceph103  *:8000       running (4w)     2w ago   4w    86.3M        -  16.2.7-112.el8cp  854c5a170572  ab3dfaaeb934