| Summary: | Swift proxy-server leaks connections to memcached | ||
|---|---|---|---|
| Product: | Red Hat Gluster Storage | Reporter: | Nick Dokos <ndokos> |
| Component: | gluster-swift | Assignee: | Luis Pabón <lpabon> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | SATHEESARAN <sasundar> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.1 | CC: | madam, ndokos, ppai, rhs-bugs, sasundar |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-11-20 06:09:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Hi Nick, (Great bug by the way) It seems that the fix is in Havana version of OpenStack Swift, and we are including it in RHS 2.1u2. Do we need to include anything else? Actually, Havana only includes the initial version of the leak fix but not the final version and not the other fixes. Those are in Icehouse and they will need to be backported. At least that's my understanding at the moment. This has been fixed by: https://review.openstack.org/#/c/45134/ and https://review.openstack.org/#/c/45134/ These are present in icehouse release. As 2.1 is EOL, closing this bug. |
Description of problem: The OpenStack Swift proxy-server service will leak connections to memcached over time. This behavior was found and fixed during the Havana release cycle, exists in Grizzly and likely Folsom as well. No upstream bug report exists that covers this leak, but the following commit does describe the problem and the fix: https://review.openstack.org/45134 However, even with that fix, the proxy-server does not obey the configured maximum connection limits to memcached. There are three separate problems in addition to the fix above: o There is still a leak when using eventlet-0.9.16 which is fixed by eventlet-0.9.17. The bug report on launchpad is: https://bugs.launchpad.net/swift/+bug/1254119 Using either Eventlet 0.9.17 or the patch suggested in the bug will address this additional leak. o The current proxy-server code does not obey any max connection limits to memcached with the above leak fixed. The bug report on launchpad is: https://bugs.launchpad.net/swift/+bug/1235027 During a catalyst run GETting one million files, we monitored the connections between proxy-server workers and memcached (these are local connections). Despite the fact that the default number of such connections is 2, the proxy-server workers routinely opened more (potentially many more, although the current tests show only modest increases). Here are the results of all the workers on one of the six servers in the setup (the first column contains the pids of the 12 proxy worker processes): | pid | Min | Max | |-------+-----+-----| | 11478 | 3 | 4 | | 11479 | 3 | 5 | | 11480 | 4 | 5 | | 11481 | 3 | 4 | | 11482 | 3 | 4 | | 11483 | 2 | 5 | | 11484 | 3 | 4 | | 11485 | 3 | 4 | | 11486 | 5 | 6 | | 11487 | 4 | 5 | | 11491 | 2 | 4 | | 11493 | 5 | 6 | With the original leak patch, and the patch from 1235027 applied (https://review.openstack.org/49739), the number of connections stays bounded: | pid | Min | Max | |-------+-----+-----| | 13530 | 2 | 2 | | 13531 | 1 | 2 | | 13532 | 1 | 2 | | 13533 | 1 | 2 | | 13534 | 1 | 2 | | 13535 | 1 | 2 | | 13536 | 2 | 2 | | 13537 | 1 | 2 | | 13538 | 1 | 2 | | 13539 | 1 | 2 | | 13540 | 1 | 2 | | 13541 | 1 | 2 | o The documented method of changing the number of such connections (which by default is supposed to be 2) is to set the option memcache_max_connections in /etc/swift/proxy-server.conf with a fallback to memcache.conf. However the code only checks for "max_connections" and only in proxy-server.conf. The bug report on launchpad is: https://bugs.launchpad.net/swift/+bug/1252893 We recommend that these changes be backported and applied to the gluster-swift component of RHS. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: