Bug 1346170
Summary: | Nested directory creation performance degrade after 100+ iterations 3*3 vol | ||
---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Karan Sandha <ksandha> |
Component: | distribute | Assignee: | Xavi Hernandez <jahernan> |
Status: | CLOSED WORKSFORME | QA Contact: | |
Severity: | high | Docs Contact: | |
Priority: | medium | ||
Version: | mainline | CC: | bugs, jahernan, mpillai, nbalacha |
Target Milestone: | --- | Keywords: | Performance, Triaged |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-11-20 12:09:04 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Karan Sandha
2016-06-14 07:34:14 UTC
The same test was reproduced in 1) Plain distribute volume 2) Single brick volume **correction** it was not reproducible in Single brick volume We have noticed that backend brick performance matters (and amplifies the performance issues) a lot for how gluster performs. Considering in this test, we are ending up creating fragments in backend filesystem, I see that this may be expected behavior. I recommend running the tests with latest glusterfs-6.x releases, and checking what is the behavior. If not happening, close it as WORKSFORME. Based on profile information, the problem seems related to the number of lookups processed by bricks (they take 99% of the time). Kernel sends a lookup for each path component before every operation. Given that this tests creates a nested directory structure, the number of lookups grows linearly as the depth is greater. The total number of lookups is quadratic on the depth of the directory. I see two possible issues here: 1. md-stat should prevent most of the lookup requests to reach the bricks. 2. There are many more lookups than expected (even considering a quadratic growth). I'll run a test with latest version and see if any of the previous problems persist. I've just repeated the test with version 7.0 and I've got much better performance. I still see a small slowdown when on depth 300, but it's less than a second. The number of lookups have been drastrically reduced. With depth 300, I only see 3710 lookups. So it seems that caching is doing its job. Based on these results I'll close the bug. |