Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
While pcsd is running it leaves stalled connections in CLOSE_WAIT state which indicates the other end of the connection has been closed while the local end is still waiting for the application to close. THis doesn't affect pcsd nor cluster operation as such but might have external effects i.e. flooding system log with notifications about INVALID packets if iptables/firewalld is configured to log ctstate INVALID packets [there is situation when one side sends closure request (FIN packet), remote side (client) acks that FIN request, but does not send its own FIN (which would be expected for proper connection closure process) for long time, and keeps the socket on the client side in CLOSE_WAIT state.
After some time the client finally sends the FIN, but as it takes too long time, the remote end already closed the connection based on timeout, and as such the last (late send) packets are considered as not legitimate, and therefore logged.]
# netstat -laputen | grep 2224
tcp 32 0 10.20.30.40:37372 10.20.30.40:2224 CLOSE_WAIT 0 318652188 1781/ruby
tcp 32 0 10.20.30.40:31713 10.20.30.41:2224 CLOSE_WAIT 0 318659641 1781/ruby
tcp 32 0 10.20.30.40:30813 10.20.30.41:2224 CLOSE_WAIT 0 318546897 1781/ruby
tcp 32 0 10.20.30.40:36472 10.20.30.40:2224 CLOSE_WAIT 0 318542722 1781/ruby
tcp 32 0 10.20.30.40:29931 10.20.30.41:2224 CLOSE_WAIT 0 318441750 1781/ruby
tcp 32 0 10.20.30.40:35594 10.20.30.40:2224 CLOSE_WAIT 0 318438254 1781/ruby
tcp6 0 0 :::2224 :::* LISTEN 0 23943 1781/ruby
Version-Release number of selected component (if applicable):
pcs-0.9.168-4.el7.x86_64
How reproducible:
always
Steps to Reproduce:
1. pcsd.service running
2. netstat -laputen | grep 2224
3. few minutes after pcsd started the stalled connections start to pile up
Actual results:
stalled connections in CLOSE_WAIT on port 2224
Expected results:
connections are closed properly
Additional info:
After further investigation (together with Tomas, thank you) we've been able to understand the issue.
Pcsd is using libcurl for communication. Libcurl is designed to reuse open sockets, therefore there might be some tcp connections in ESTABLISHED state even thought there is no communication happening at the moment. We assumed that those connection will be eventually closed by GC mechanism (but it seems that is not the case)
To solve this issue, we have to explicitly close all connections right after we use them rather than rely on GC to do so.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (pcs bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2020:5449