Created attachment 1769526 [details] client info wrong Created attachment 1769526 [details] client info wrong Description of problem: cephfs-top command showing kernal mount as libcephfs client in the description cephfs-top - Tue Apr 6 05:54:25 2021 Client(s): 2 - 1 FUSE, 0 kclient, 1 libcephfs CLIENT_ID MOUNT_ROOT CAP_HIT(%) READ_LATENCY(s) WRITE_LATENCY(s) METADATA_LATENCY(s) DENTRY_LEASE(%) OPENED_FILES PINNED_ICAPS OPENED_INODES MOUNT_POINT@HOST/ADDR 34206 / 93.46 0.0 0.0 0.02 0.0 0 4 0 /mnt/fuse_mount@ceph-amk-1617277523462-node7-client/10. 0.2034489 / N/A N/A N/A N/A N/A N/A N/A N/A N/A@ceph-amk-1617277523462-node7-client/v1:10.0.208.72 Version-Release number of selected component (if applicable): ceph version 16.1.0-1323.el8cp How reproducible: Fuse and Kernal mount on the same client. cephfs-top command is listing kernal client as libcephfs client cephfs-top - Tue Apr 6 05:17:09 2021 Client(s): 2 - 1 FUSE, 0 kclient, 1 libcephfs CLIENT_ID MOUNT_ROOT CAP_HIT(%) READ_LATENCY(s) WRITE_LATENCY(s) METADATA_LATENCY(s) DENTRY_LEASE(%) OPENED_FILES PINNED_ICAPS OPENED_INODES MOUNT_POINT@HOST/ADDR 34206 / 93.46 0.0 0.0 0.02 0.0 0 4 0 /mnt/fuse_mount@ceph-amk-1617277523462-node7-client/10. 0.2034489 / N/A N/A N/A N/A N/A N/A N/A N/A N/A@ceph-amk-1617277523462-node7-client/v1:10.0.208.72 Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Please specify the severity of this bug. Severity is defined here: https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
What kernel version is the kernel mount here? Those patches only started going in to upstream kernels fairly recently (v5.8 for the first tranche, with some follow-on patches later).
As per my cephfs-top debug and also from Jeff's comment above, the issue of missing metrics and the wrong count are because of the older kernel. So please test this with the 8.4 kernel and let us know. I'm closing this bug for now.
I am facing problem for getting ceph fs perf stats with latest kernel :4.18.0-305.el8.x86_64 [cephuser@ceph-amar-42-84-1620129830591-node1-mon-mgr-installer ~]$ sudo ceph fs status cephfs - 0 clients ====== +------+--------+-----------------------------------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+-----------------------------------------+---------------+-------+-------+ | 0 | active | ceph-amar-42-84-1620129830591-node3-mds | Reqs: 0 /s | 10 | 13 | +------+--------+-----------------------------------------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 1536k | 109G | | cephfs_data | data | 0 | 109G | +-----------------+----------+-------+-------+ +------------------------------------------+ | Standby MDS | +------------------------------------------+ | ceph-amar-42-84-1620129830591-node12-mds | | ceph-amar-42-84-1620129830591-node4-mds | | ceph-amar-42-84-1620129830591-node6-mds | +------------------------------------------+ MDS version: ceph version 14.2.11-154.el8cp (214891cf4af753adb7301d7180650d79dd6d7550) nautilus (stable) [cephuser@ceph-amar-42-84-1620129830591-node1-mon-mgr-installer ~]$ sudo ceph mgr module enable stats Error ENOENT: all mgr daemons do not support module 'stats', pass --force to force enablement [cephuser@ceph-amar-42-84-1620129830591-node1-mon-mgr-installer ~]$ sudo ceph mgr module enable stats --force module 'stats' is already enabled [cephuser@ceph-amar-42-84-1620129830591-node1-mon-mgr-installer ~]$ sudo ceph fs perf stats no valid command found; 10 closest matches: fs status {<fs>} fs volume ls fs volume create <name> fs volume rm <vol_name> {<yes-i-really-mean-it>} fs subvolumegroup ls <vol_name> fs subvolumegroup create <vol_name> <group_name> {<pool_layout>} {<int>} {<int>} {<mode>} fs subvolumegroup rm <vol_name> <group_name> {--force} fs subvolume ls <vol_name> {<group_name>} fs subvolume create <vol_name> <sub_name> {<int>} {<group_name>} {<pool_layout>} {<int>} {<int>} {<mode>} {--namespace-isolated} fs subvolume rm <vol_name> <sub_name> {<group_name>} {--force} {--retain-snapshots} Error EINVAL: invalid command [cephuser@ceph-amar-42-84-1620129830591-node1-mon-mgr-installer ~]$ uname -a Linux ceph-amar-42-84-1620129830591-node1-mon-mgr-installer 4.18.0-305.el8.x86_64 #1 SMP Thu Apr 29 08:54:30 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux [cephuser@ceph-amar-42-84-1620129830591-node1-mon-mgr-installer ~]$ cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.4 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.4" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
The 8.4 kernel also does not have these patches. They are still a WIP upstream and have not been backported to RHEL yet.
The complete set of 'perf stats' patches would probably make Kernel v5.14. So you need those patches to resolve the issue mentioned in the Description. The downstream doc (if there is one) needs an update about this.
*** Bug 1946520 has been marked as a duplicate of this bug. ***