Bug 1569046

Summary:	Katello agent does not reconnect to the server once QPID and Qdrouter are restarted
Product:	Red Hat Satellite	Reporter:	sbadhwar
Component:	katello-agent	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED DUPLICATE	QA Contact:	Jan Hutař <jhutar>
Severity:	medium	Docs Contact:
Priority:	high
Version:	6.3.0	CC:	jhutar, mmccune, pmoravec, psuriset, sthirugn
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-03-25 20:16:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description sbadhwar 2018-04-18 14:31:22 UTC

Description of problem:
Opening this bug about a problem we discovered while running katello-agent at scale, for tracking purpose.

We have observed that katello-agent(10K clients were running katello-agent) does not re-establish a connection to the server once qdrouter and qpid are restarted to Satellite and Capsules even after quite a long time of waiting period.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Increase the Max open files limit for QPID and Qdrouter on Satellite and Capsules.
2. Restart Qdouter and QPID on both Satellite and capsules.
3. Wait for 10-15 minutes before executing some operation.
4. Run an errata install on hosts

Actual results:
Errata install job times out for the hosts even after having the accept action timeout limit set to 120 seconds.

Expected results:
Errata install works fine for clients

Additional info:
After our errata install job failed, we tried to manually restart the katello-agent on the clients. After this was done, we tried to do an errata install again and the process went fine.
One message that we noted on the client side after manually restarting the katello-agent was that katello-agent logged a message about A task being dropped.