Cause:
When a C++ client is trying to reconnect over an unreliable network that drops a lot of packets, the client library sometimes fails to close the TCP connection on its end. The connection can get stuck in CLOSE_WAIT status and the associated resources are held indefinitely.
Consequence:
A series of these leaks can cause resources issues, such as FDs running out and preventing new client connections to the broker.
Fix:
In cases where the connection is aborted due to things like a faulty network, we now explicitly close the socket to avoid the leak.
Result:
This allows the client to reconnect without leaking any associated resources.
Description of problem:
When a C++ client is trying to reconnect over unreliable network that drops a lot of packets, the client library sometimes forgets to close TCP connection on its end. New one is established later on, server properly closed the semi-hung, this one remains in CLOSE_WAIT status forever - a CLOSE_WAIT leak.
Next CLOSE_WAIT leaks can subsequently happen, causing ulimit in FDs hit after some time and preventing the client to open a new connection to the broker.
A simple patch proposed by the customer++ , I acknowledge its logic though some developer more familiar with that code should do proper review.
Version-Release number of selected component (if applicable):
qpid-cpp-client 1.36.0-20 or older
How reproducible:
100% over some time
Steps to Reproduce:
1. Update in attached client program IP address to yours qpid broker one
2. Compile attached client program:
g++ client.cpp -lqpidmessaging -I/usr/include -o leak_repro -std=gnu++0x
3. Run it:
./leak_repro
4. Add iptables rule to drop 4/5 of packets towards qpidd:
iptables -A OUTPUT -p tcp --destination-port 5672 -m statistic --mode random --probability 0.80 -j DROP
5. monitor CLOSE_WAITs on outgoing connections
Actual results:
5. after few minutes, netstat shows a growing list of CLOSE_WAITs on client side only
Expected results:
5. no CLOSE_WAIT (other than very short-living)
Additional info:
Created attachment 1518640[details]
proposed patch
Proposed patch (pre-verified against the reproducer on my system).
Rationale:
- per tcpdumps, CLOSE_WAITs happen when TCP connection is established and AMQP connection is opening via this scenario:
- "AMQP" frame sent from client
- "AMQP" and connection.start replied from broker
- then silence due to all packets dropped
- then broker sends FIN+ACK
- This line of code is executed: https://github.com/apache/qpid-cpp/blob/master/src/qpid/client/TCPConnector.cpp#L164
- but it does not close the socket
- patch executes the same methods + calls socket->close();
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2019:0390