Bug 1494280

Summary:	Supercolony seems to be overloaded on a regular basis
Product:	[Community] GlusterFS	Reporter:	M. Scherer <mscherer>
Component:	project-infrastructure	Assignee:	bugs <bugs>
Status:	CLOSED WORKSFORME	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, gluster-infra, nigelb
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-10-08 03:15:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description M. Scherer 2017-09-21 22:11:45 UTC

Description of problem:

Not sure why, but supercolony seems to take a long time to let me connect with ssh since a few days. I did look and can't see anything overly wrong.

Munin show some mail activity, but this should result into slowdown for login.

Looking at postfix log, it seems to not do g suspicious (even if we have postgrey message, so I did restart postgrey). I see nothing weird either on mailman log, no specific acitivty on I/O, network, or CPU. 

Will look later

Comment 1 M. Scherer 2017-09-22 13:02:47 UTC

So log also stopped, so it might be related to that. (the syslog server seems to also be full...)

Comment 2 M. Scherer 2017-09-22 13:20:56 UTC

There is also a ton of errors: 
Sep 22 12:59:19 supercolony postfix/smtp[28860]: certificate verification failed for aspmx.l.google.com[74.125.69.27]:25: untrusted issuer /C=US/O=Equifax/OU=Equifax Secure Certificate Authority


But the problem seems to be linked to rsyslog, I removed the logs export and now, everything is faster.

Comment 3 M. Scherer 2017-09-22 13:29:07 UTC

Ok so the log server is full, and it seems to slow down all servers sending something there.

Comment 4 M. Scherer 2017-10-09 12:06:00 UTC

So, ansible did reinstall the logs export and login on supercolony was slow again. I have cleaned (or rather) compressed the log on the syslog server. Seems for some reason to have generated lots of logs by itself in the past.

Comment 5 Nigel Babu 2018-10-08 03:15:10 UTC

This is now fixed and I'm going to close the bug.