Description of problem: goferd is not closing network sockets resulting in a lot of CLOSE_WAIT sockets 213 close_wait sockets tcp 54 0 10.7.44.118:50152 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49666 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44174 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46506 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48587 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49963 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43728 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50032 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44603 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48470 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44747 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43125 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48552 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45105 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49112 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51452 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50326 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49852 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51599 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49468 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45780 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48196 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44408 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45624 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46466 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:42526 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51831 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49435 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52153 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49220 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46615 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47281 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:42549 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46867 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50765 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43627 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44886 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52761 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50842 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46234 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51526 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52677 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51493 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48329 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:53196 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43453 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51098 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50877 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46347 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51191 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51899 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47318 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45829 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44563 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 46 0 10.7.44.118:51230 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 0 0 10.7.44.118:53421 10.1.43.204:5647 ESTABLISHED 4277/python tcp 0 0 10.7.44.118:47439 10.1.43.204:5646 ESTABLISHED 57753/qdrouterd tcp 54 0 10.7.44.118:47728 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50287 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44250 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47442 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43922 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52184 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52873 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47483 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48436 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46948 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43558 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45369 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45554 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43660 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51061 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46794 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49780 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44283 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43761 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45703 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52501 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45867 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43484 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47902 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52801 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46149 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51371 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44213 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50066 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52301 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44491 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47975 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:42499 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52084 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:42486 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44445 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47943 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46312 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48052 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46985 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52834 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48365 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45593 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46077 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52470 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50251 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46111 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50196 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43048 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47090 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45744 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43346 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52722 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48844 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44672 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50364 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43151 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49733 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49998 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49183 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50611 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43694 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47232 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48285 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51133 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48013 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48403 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46901 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49819 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43520 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49078 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:53231 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47654 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:42539 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49885 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43088 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46834 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47166 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46414 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50113 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50576 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44528 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48614 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49510 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50805 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43877 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47865 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46382 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49924 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45519 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52119 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 46 0 10.7.44.118:44820 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52610 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45022 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45664 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49595 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:53149 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48965 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46647 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47021 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46679 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51303 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44918 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48092 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51263 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48804 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:42516 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43955 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45480 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45059 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51411 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51028 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52918 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48160 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:46270 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47053 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43312 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50912 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49035 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44636 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48765 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50995 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48127 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47130 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52046 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44957 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50955 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47617 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51562 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44851 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47406 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52642 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49627 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45297 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43382 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43419 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43821 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48230 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44711 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43271 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49699 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43596 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52013 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:51866 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47694 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:43233 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 0 0 10.7.44.118:53890 10.1.43.204:5647 ESTABLISHED 4277/python tcp 54 0 10.7.44.118:51334 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47831 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:44784 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:52268 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47198 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:48999 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:50646 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:49396 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:47351 10.1.43.204:5647 CLOSE_WAIT 4277/python tcp 54 0 10.7.44.118:45331 10.1.43.204:5647 CLOSE_WAIT 4277/python 214 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Hi Marcello, do you have a reproducer machine I could play with? The reproducer steps above are quite generic and doe not work for me (to see the close waits).
This was fixed by https://issues.apache.org/jira/browse/PROTON-1010 which has been back ported to proton 0.9-13 (and brewed). This bug can probably be marked as MODIFIED.
Moving to POST based on comment 4
Verified! @ Sat 6.2 Snap 9.3 I had RHEL67 and Capsule repo that I synced/pushed to capsule. I see lot of TIME_WAIT connections but not CLOSE_WAIT connections during push or capsule-sync. Please feel free to reopen if I misunderstood the steps. Currently moving to Verified.
/me verified this as well. Also installing an errata/package to a Content Host (that uses very similar call flows wrt. goferd and proton reactor library) does not cause CLOSE_WAITs (just a memory leak, subject of https://bugzilla.redhat.com/show_bug.cgi?id=1272758)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501