Bug 2270375 - [CephFS-Mirror] - After destroying the mirror configuration, the metrics still reporting the "mirrored_filesystems" as true.
Summary: [CephFS-Mirror] - After destroying the mirror configuration, the metrics stil...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: CephFS
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 7.1
Assignee: Jos Collin
QA Contact: Hemanth Kumar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-03-20 02:42 UTC by Hemanth Kumar
Modified: 2024-03-27 02:59 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-03-26 14:18:36 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 64998 0 None None None 2024-03-20 05:29:08 UTC
Red Hat Issue Tracker RHCEPH-8568 0 None None None 2024-03-20 02:43:09 UTC

Description Hemanth Kumar 2024-03-20 02:42:21 UTC
Description of problem:
-----------------------
After destroying the cephfs mirroring configuration, the metrics still reports the mirrored_filesystem.

[root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph fs snapshot mirror daemon status
Error ENOTSUP: Module 'mirroring' is not enabled/loaded (required by command 'fs snapshot mirror daemon status'): use `ceph mgr module enable mirroring` to enable it


=========

Executed the automated Tests to Configure and Destroy the Mirroring - http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-H6MTU0/

[root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph fs ls
name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ]

[root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph mgr module enable mirroring


[root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph fs snapshot mirror disable cephfs
{}
[root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph fs snapshot mirror peer_list
Error EINVAL: filesystem None is not mirrored

[root@ceph1-hk-m-zy7rnm-node6 7abd8508-e505-11ee-bbf4-fa163e6ab61d]# ceph --admin-daemon ceph-client.cephfs-mirror.ceph1-hk-m-zy7rnm-node6.yvtajb.2.94028663044328.asok help
{
    "config diff": "dump diff of current config and default config",
    "config diff get": "dump diff get <field>: dump diff of current and default config setting <field>",
    "config get": "config get <field>: get the config value",
    "config help": "get config setting schema and descriptions",
    "config set": "config set <field> <val> [<val> ...]: set a config variable",
    "config show": "dump current config settings",
    "config unset": "config unset <field>: unset a config variable",
    "counter dump": "dump all labeled and non-labeled counters and their values",
    "counter schema": "dump all labeled and non-labeled counters schemas",
    "dump_mempools": "get mempool stats",
    "get_command_descriptions": "list available commands",
    "git_version": "get git sha1",
    "help": "list available commands",
    "injectargs": "inject configuration arguments into running daemon",
    "log dump": "dump recent log entries to log file",
    "log flush": "flush log entries to log file",
    "log reopen": "reopen log file",
    "objecter_requests": "show in-progress osd requests",
    "perf dump": "dump non-labeled counters and their values",
    "perf histogram dump": "dump perf histogram values",
    "perf histogram schema": "dump perf histogram schema",
    "perf reset": "perf reset <name>: perf reset all or one perfcounter name",
    "perf schema": "dump non-labeled counters schemas",
    "rotate-key": "rotate live authentication key",
    "version": "get ceph version"
}
[root@ceph1-hk-m-zy7rnm-node6 7abd8508-e505-11ee-bbf4-fa163e6ab61d]#

[root@ceph1-hk-m-zy7rnm-node6 7abd8508-e505-11ee-bbf4-fa163e6ab61d]# ceph --admin-daemon ceph-client.cephfs-mirror.ceph1-hk-m-zy7rnm-node6.yvtajb.2.94028663044328.asok counter dump | jq -r '. | to_entries | map(select(.key | match("cephfs_mirror")))'
[
  {
    "key": "cephfs_mirror",
    "value": [
      {
        "labels": {},
        "counters": {
          "mirrored_filesystems": 1,
          "mirror_enable_failures": 0
        }
      }
    ]
  }
]

Comment 2 Venky Shankar 2024-03-20 05:03:38 UTC
(In reply to Hemanth Kumar from comment #0)
> Description of problem:
> -----------------------
> After destroying the cephfs mirroring configuration, the metrics still
> reports the mirrored_filesystem.
> 
> [root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph fs snapshot mirror daemon
> status
> Error ENOTSUP: Module 'mirroring' is not enabled/loaded (required by command
> 'fs snapshot mirror daemon status'): use `ceph mgr module enable mirroring`
> to enable it
> 
> 
> =========
> 
> Executed the automated Tests to Configure and Destroy the Mirroring -
> http://magna002.ceph.redhat.com/cephci-jenkins/cephci-run-H6MTU0/
> 
> [root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph fs ls
> name: cephfs, metadata pool: cephfs.cephfs.meta, data pools:
> [cephfs.cephfs.data ]
> 
> [root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph mgr module enable
> mirroring
> 
> 
> [root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph fs snapshot mirror
> disable cephfs
> {}
> [root@ceph1-hk-m-zy7rnm-node8 subvolgroup_1]# ceph fs snapshot mirror
> peer_list
> Error EINVAL: filesystem None is not mirrored
> 
> [root@ceph1-hk-m-zy7rnm-node6 7abd8508-e505-11ee-bbf4-fa163e6ab61d]# ceph
> --admin-daemon
> ceph-client.cephfs-mirror.ceph1-hk-m-zy7rnm-node6.yvtajb.2.94028663044328.
> asok help
> {
>     "config diff": "dump diff of current config and default config",
>     "config diff get": "dump diff get <field>: dump diff of current and
> default config setting <field>",
>     "config get": "config get <field>: get the config value",
>     "config help": "get config setting schema and descriptions",
>     "config set": "config set <field> <val> [<val> ...]: set a config
> variable",
>     "config show": "dump current config settings",
>     "config unset": "config unset <field>: unset a config variable",
>     "counter dump": "dump all labeled and non-labeled counters and their
> values",
>     "counter schema": "dump all labeled and non-labeled counters schemas",
>     "dump_mempools": "get mempool stats",
>     "get_command_descriptions": "list available commands",
>     "git_version": "get git sha1",
>     "help": "list available commands",
>     "injectargs": "inject configuration arguments into running daemon",
>     "log dump": "dump recent log entries to log file",
>     "log flush": "flush log entries to log file",
>     "log reopen": "reopen log file",
>     "objecter_requests": "show in-progress osd requests",
>     "perf dump": "dump non-labeled counters and their values",
>     "perf histogram dump": "dump perf histogram values",
>     "perf histogram schema": "dump perf histogram schema",
>     "perf reset": "perf reset <name>: perf reset all or one perfcounter
> name",
>     "perf schema": "dump non-labeled counters schemas",
>     "rotate-key": "rotate live authentication key",
>     "version": "get ceph version"
> }
> [root@ceph1-hk-m-zy7rnm-node6 7abd8508-e505-11ee-bbf4-fa163e6ab61d]#
> 
> [root@ceph1-hk-m-zy7rnm-node6 7abd8508-e505-11ee-bbf4-fa163e6ab61d]# ceph
> --admin-daemon
> ceph-client.cephfs-mirror.ceph1-hk-m-zy7rnm-node6.yvtajb.2.94028663044328.
> asok counter dump | jq -r '. | to_entries | map(select(.key |
> match("cephfs_mirror")))'
> [
>   {
>     "key": "cephfs_mirror",
>     "value": [
>       {
>         "labels": {},
>         "counters": {
>           "mirrored_filesystems": 1,
>           "mirror_enable_failures": 0
>         }
>       }
>     ]
>   }
> ]

This should be a bug.

Jos, please create a tracker and link it in this BZ.

Comment 3 Jos Collin 2024-03-20 13:10:02 UTC
@Hemanth

This doesn't exist in the upstream. We've backported all metrics patches downstream, as discussed with Venky.
After disabling, wait for sometime and issue the 'counter dump' command and check if 'mirrored_filesystems' shows correct value (asynchronous)?

Comment 4 Jos Collin 2024-03-20 13:18:10 UTC
Btw, your peer_list command is wrong.


Note You need to log in before you can comment on or make changes to this bug.