Bug 2029832
| Summary: | [CephFS] MDS is getting Crashed when we try directory pinning on multiple clients | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Amarnath <amk> | ||||
| Component: | CephFS | Assignee: | Patrick Donnelly <pdonnell> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Hemanth Kumar <hyelloji> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 5.0 | CC: | ceph-eng-bugs | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 5.0z1 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-12-09 16:43:10 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
MDS first went to active(laggy) state and then to stopped(laggy)
[root@ceph-ci-crash-l5yue2-node11 ~]# ceph fs status cephfs
cephfs - 18 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.ceph-ci-crash-l5yue2-node7.rbktqx Reqs: 0 /s 1564 1540 1029 4003
1 active(laggy) cephfs.ceph-ci-crash-l5yue2-node5.dpjndg 0 0 0 0
POOL TYPE USED AVAIL
cephfs.cephfs.meta metadata 388M 40.6G
cephfs.cephfs.data data 33.1G 40.6G
MDS version: ceph version 16.2.0-146.el8cp (56f5e9cfe88a08b6899327eca5166ca1c4a392aa) pacific (stable)
[root@ceph-ci-crash-l5yue2-node11 ~]# ceph fs status cephfs
cephfs - 18 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.ceph-ci-crash-l5yue2-node7.rbktqx Reqs: 0 /s 1563 1538 1027 4003
1 stopping(laggy) cephfs.ceph-ci-crash-l5yue2-node5.dpjndg 0 0 0 0
POOL TYPE USED AVAIL
cephfs.cephfs.meta metadata 388M 40.6G
cephfs.cephfs.data data 33.1G 40.6G
MDS version: ceph version 16.2.0-146.el8cp (56f5e9cfe88a08b6899327eca5166ca1c4a392aa) pacific (stable)
MDS first went to active(laggy) state and then to stopped(laggy)
[root@ceph-ci-crash-l5yue2-node11 ~]# ceph fs status cephfs
cephfs - 18 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.ceph-ci-crash-l5yue2-node7.rbktqx Reqs: 0 /s 1564 1540 1029 4003
1 active(laggy) cephfs.ceph-ci-crash-l5yue2-node5.dpjndg 0 0 0 0
POOL TYPE USED AVAIL
cephfs.cephfs.meta metadata 388M 40.6G
cephfs.cephfs.data data 33.1G 40.6G
MDS version: ceph version 16.2.0-146.el8cp (56f5e9cfe88a08b6899327eca5166ca1c4a392aa) pacific (stable)
[root@ceph-ci-crash-l5yue2-node11 ~]# ceph fs status cephfs
cephfs - 18 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active cephfs.ceph-ci-crash-l5yue2-node7.rbktqx Reqs: 0 /s 1563 1538 1027 4003
1 stopping(laggy) cephfs.ceph-ci-crash-l5yue2-node5.dpjndg 0 0 0 0
POOL TYPE USED AVAIL
cephfs.cephfs.meta metadata 388M 40.6G
cephfs.cephfs.data data 33.1G 40.6G
MDS version: ceph version 16.2.0-146.el8cp (56f5e9cfe88a08b6899327eca5166ca1c4a392aa) pacific (stable)
|
Created attachment 1845063 [details] MDS and MGR logs Description of problem: MDS is getting crashed when we try pinning directory on multiple clients and run IOs on it This was encountered in the Automation run Version-Release number of selected component (if applicable): MDS version: ceph version 16.2.0-146.el8cp (56f5e9cfe88a08b6899327eca5166ca1c4a392aa) pacific (stable) How reproducible: 2/2 Steps to Reproduce: We are doing the below steps as part of the automation 1. we are having 4 clients configured 2. On 2 clients we are mounting using fuse and on 3 clients we are using kernel 3. We are activating multiple mds on cephfs ceph fs set cephfs max_mds 2 4. We are doing read and write operations on all the clients using fio 5. After this we are creating bulk directories on client 1 and client 2 6. We are pinning first 10 directories in client1 and running IOs on other clients 7. on client1 we are failing MDS ceph mds fail 1 8. step 6 and 7 is performed on all the clients After this test all the MDS are going to error state [ceph: root@ceph-ci-crash-l5yue2-node1-installer /]# ceph orch ps NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID mds.cephfs.ceph-ci-crash-l5yue2-node3.zjwvvg ceph-ci-crash-l5yue2-node3 error 5m ago 3h - <unknown> <unknown> <unknown> mds.cephfs.ceph-ci-crash-l5yue2-node4.swlqwe ceph-ci-crash-l5yue2-node4 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d f3c62e2190f6 mds.cephfs.ceph-ci-crash-l5yue2-node5.dpjndg ceph-ci-crash-l5yue2-node5 error 5m ago 3h - <unknown> <unknown> <unknown> mds.cephfs.ceph-ci-crash-l5yue2-node6.hsazfq ceph-ci-crash-l5yue2-node6 error 9m ago 3h - <unknown> <unknown> <unknown> mds.cephfs.ceph-ci-crash-l5yue2-node7.rbktqx ceph-ci-crash-l5yue2-node7 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d 249ab44a9b12 mgr.ceph-ci-crash-l5yue2-node1-installer.dboxxr ceph-ci-crash-l5yue2-node1-installer running (3h) 9m ago 3h *:9283 16.2.0-146.el8cp f5dcd499576d be7eaefbc445 mgr.ceph-ci-crash-l5yue2-node2.goqlbq ceph-ci-crash-l5yue2-node2 running (3h) 5m ago 3h *:8443 16.2.0-146.el8cp f5dcd499576d ce8ede2b6366 mon.ceph-ci-crash-l5yue2-node1-installer ceph-ci-crash-l5yue2-node1-installer running (3h) 9m ago 3h - 16.2.0-146.el8cp f5dcd499576d baa38df82604 mon.ceph-ci-crash-l5yue2-node2 ceph-ci-crash-l5yue2-node2 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d c1adee376d1b mon.ceph-ci-crash-l5yue2-node3 ceph-ci-crash-l5yue2-node3 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d 8ab8f1dfab5c osd.0 ceph-ci-crash-l5yue2-node4 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d a9800132ceab osd.1 ceph-ci-crash-l5yue2-node5 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d e6170808b885 osd.10 ceph-ci-crash-l5yue2-node5 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d 02fe2e1b6d02 osd.11 ceph-ci-crash-l5yue2-node6 running (3h) 9m ago 3h - 16.2.0-146.el8cp f5dcd499576d b5b4ee1237cd osd.2 ceph-ci-crash-l5yue2-node6 running (3h) 9m ago 3h - 16.2.0-146.el8cp f5dcd499576d b9bcc0713216 osd.3 ceph-ci-crash-l5yue2-node4 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d 4d664c08b0e3 osd.4 ceph-ci-crash-l5yue2-node5 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d aa89c56e540b osd.5 ceph-ci-crash-l5yue2-node6 running (3h) 9m ago 3h - 16.2.0-146.el8cp f5dcd499576d 8e074a60ffe2 osd.6 ceph-ci-crash-l5yue2-node4 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d d240b55975e4 osd.7 ceph-ci-crash-l5yue2-node5 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d 40fe656fb7e2 osd.8 ceph-ci-crash-l5yue2-node6 running (3h) 9m ago 3h - 16.2.0-146.el8cp f5dcd499576d b65084f39380 osd.9 ceph-ci-crash-l5yue2-node4 running (3h) 5m ago 3h - 16.2.0-146.el8cp f5dcd499576d cb30c1ba2796 Actual results: Expected results: Additional info: Attached mgr and mds logs