Bug 1623264
| Summary: | Ceph MDS hang with smallfile workload | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Ben England <bengland> |
| Component: | CephFS | Assignee: | Yan, Zheng <zyan> |
| Status: | CLOSED ERRATA | QA Contact: | Ramakrishnan Periyasamy <rperiyas> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.0 | CC: | bengland, ceph-eng-bugs, john.spray, pdonnell, rperiyas, tchandra, vshankar, zyan |
| Target Milestone: | z1 | ||
| Target Release: | 3.1 | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | RHEL: ceph-12.2.5-47.el7cp Ubuntu: ceph_12.2.5-32redhat1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-11-09 00:59:32 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Ben England
2018-08-28 20:52:41 UTC
I'll take a look... Here are 2 other forced core dumps where the workload hung, but when I forced the core dump, the smallfile workload started right up again as if nothing had happened. This is becoming a pattern - no clear reason why it is hanging. The test run is: http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/public/ceph/linode/dallas.logs/smf-2018-08-29-10-42/ cores/mds-000/crash/ceph-mds-li108-47.members.linode.com-23551-1535563501.core.gz cores/mds-001/crash/ceph-mds-li496-6.members.linode.com-23340-1535563707.core.gz This script tells you when the core dump was generated - just feed it the pathname of the uncompressed core file. http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/public/ceph/linode/time-of-core-dump.py [root@li35-54 tmp]# for n in *.core ; do ls -l $n ; ~/time-of-core-dump.py $n ; done -rw-------. 1 root root 13488488448 Aug 30 18:22 ceph-mds-li108-47.members.linode.com-23551-1535563501.core Wed Aug 29 17:25:01 2018 -rw-------. 1 root root 4716032000 Aug 30 18:22 ceph-mds-li496-6.members.linode.com-23340-1535563707.core Wed Aug 29 17:28:27 2018 and it happened during this smallfile append test: http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/public/ceph/linode/dallas.logs/smf-2018-08-29-10-42/sample-1/ops/append/smf.log It's just a gut feel thing but my hypothesis is that Filestore is causing MDS to hang, because it causes huge latencies due to directory splits and fssync() taking a long time, am going to try Bluestore and see if the same workload completes, Bluestore has lower tail latency due to direct I/O. Changed name of bz because I'm not convinced of Patrick's explanation, that will come from analysis. Main thing is that i can frequently reproduce a behavior that seems to consist of 3 phases: - MDS is having super-long latencies on writes to cephfs_metadata - MDS evicts clients for failure to respond - cluster-wide throughput drops to zero I can restore throughput by restarting MDS processes, so its not the OSDs that are hung, or the clients, or smallfile. Forget my Filestore hypothesis. I made MDS hang with Bluestore. Several times. this ceph.log shows one of them. http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/public/ceph/linode/dallas.logs/smf-2018-09-03-20-56/ceph.log Thanks, Ben. I'll look into this since it is not filestore/bluestore or OSD latencies. Lamba function kill_mds_session (in MDSRank::evict_client) waited for session to be evicted. The function blocked finisher. So finisher can't execute later contexts (including the context of session eviction). I updated ceph-ci/zyan-luminous-integration *** Bug 1628308 has been marked as a duplicate of this bug. *** the test should do client eviction frequently. test should be something like: 1. mount cephfs 2. run smallfile on cephfs 3. wait random time and evict the client 4. goto step 1 above steps are be executed by multiple mount point parallelly Thanks Zheng, will try this steps and update the bug. yes. this bug can be marked as verified Thanks Zheng. Moving this bug to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3530 |