1907941 – Connection from tun0 subnet gateway to console 8443 keeping haproxy alive

Bug 1907941 - Connection from tun0 subnet gateway to console 8443 keeping haproxy alive

Summary: Connection from tun0 subnet gateway to console 8443 keeping haproxy alive

Keywords:
Status:	CLOSED DUPLICATE of bug 1905100
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.5
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Andrew McDermott
QA Contact:	Hongan Li
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-12-15 14:47 UTC by Matthew Robson
Modified:	2024-06-13 23:44 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-12-17 19:00:38 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Matthew Robson 2020-12-15 14:47:17 UTC

Description of problem:

Seeing 100+ haproxy instances in the signal finish pid list going back to in excess of 24 hours.

Looking at netstat on the node, we see a number of connections from the tun0 gateway address to the console on port 8443:

./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:44858      100.126.4.60:8443       ESTABLISHED 1000520000 300961268  134480/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:57228      100.126.3.77:8443       ESTABLISHED 1000520000 301401966  178199/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:36124      100.126.4.60:8443       ESTABLISHED 1000520000 301471094  183503/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:42634      100.126.11.80:8443      ESTABLISHED 1000520000 301515811  187579/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:47382      100.126.4.60:8443       ESTABLISHED 1000520000 302278604  257063/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:51542      100.126.3.77:8443       ESTABLISHED 1000520000 302930008  315916/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:34934      100.126.3.77:8443       ESTABLISHED 1000520000 303502867  370969/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:51772      100.126.4.60:8443       ESTABLISHED 1000520000 303659380  385705/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:32934      100.126.3.77:8443       ESTABLISHED 1000520000 303960079  414273/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:41078      100.126.3.77:8443       ESTABLISHED 1000520000 304153111  432096/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:40962      100.126.4.60:8443       ESTABLISHED 1000520000 304166754  433512/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:38744      100.126.3.77:8443       ESTABLISHED 1000520000 304194909  433870/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:55358      100.126.4.60:8443       ESTABLISHED 1000520000 305178294  526268/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:41348      100.126.4.60:8443       ESTABLISHED 1000520000 305963376  596792/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:42028      100.126.4.60:8443       ESTABLISHED 1000520000 306463976  644003/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:35576      100.126.3.77:8443       ESTABLISHED 1000520000 307087370  701936/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:59986      100.126.3.77:8443       ESTABLISHED 1000520000 307268535  717979/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:58716      100.126.4.60:8443       ESTABLISHED 1000520000 308001737  785968/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:39692      100.126.3.77:8443       ESTABLISHED 1000520000 308526953  830776/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:39408      100.126.3.77:8443       ESTABLISHED 1000520000 308536994  830776/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:35572      100.126.4.60:8443       ESTABLISHED 1000520000 308906367  869874/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:57576      100.126.3.77:8443       ESTABLISHED 1000520000 309514985  927315/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:51360      100.126.4.60:8443       ESTABLISHED 1000520000 309725128  947919/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:53234      100.126.3.77:8443       ESTABLISHED 1000520000 309759536  951195/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:40868      100.126.3.77:8443       ESTABLISHED 1000520000 309823000  956470/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:59184      100.126.3.77:8443       ESTABLISHED 1000520000 309846505  957602/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:46802      100.126.3.77:8443       ESTABLISHED 1000520000 309972719  971141/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:41496      100.126.3.77:8443       ESTABLISHED 1000520000 310011338  974905/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:48090      100.126.3.77:8443       ESTABLISHED 1000520000 310098349  981440/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:59254      100.126.4.60:8443       ESTABLISHED 1000520000 310114179  984555/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:43826      100.126.3.77:8443       ESTABLISHED 1000520000 310135977  985985/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:59054      100.126.3.77:8443       ESTABLISHED 1000520000 310154505  987213/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:36858      100.126.3.77:8443       ESTABLISHED 1000520000 310188776  992137/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:38364      100.126.3.77:8443       ESTABLISHED 1000520000 310273644  999133/haproxy       off (0.00/0/0)
./networking/netstat_-W_-neopa:tcp        0      0 100.126.41.1:42744      100.126.3.77:8443       ESTABLISHED 1000520000 310390340  1010328/haproxy      off (0.00/0/0)

Restarting the console pod lead to all of the old haproxy instances being released (see attached graph).



Version-Release number of selected component (if applicable):

4.5.22


How reproducible:

Keep reoccurring after haproxy restarts. In extreme cases, the router is crashing and restarting on its own.


Steps to Reproduce:
1. TBD
2.
3.

Actual results:

Lots of haproxy instances are kept around


Expected results:

Should be closing the connections and letting haproxy exit,


Additional info:

Comment 3 Andrew McDermott 2020-12-17 17:35:25 UTC

Related bug: https://bugzilla.redhat.com/show_bug.cgi?id=1905100

Comment 4 Andrew McDermott 2020-12-17 19:00:38 UTC

This looks to be identical to https://bugzilla.redhat.com/show_bug.cgi?id=1905100. Debugging 1905100 yesterday there were many many occasions where the only remaining connections were all console related.

*** This bug has been marked as a duplicate of bug 1905100 ***

Note You need to log in before you can comment on or make changes to this bug.