Bug 1520502

Summary:	Candlepin keeps writing to rotated files and logrotate is unable to do its job
Product:	[Community] Candlepin	Reporter:	Barnaby Court <bcourt>
Component:	candlepin	Assignee:	ojanus
Status:	CLOSED CURRENTRELEASE	QA Contact:	Katello QA List <katello-qa-list>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.1	CC:	candlepin-bugs, katello-qa-list, lzap, nmoumoul, redakkan, riehecky, skallesh
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	candlepin-2.6.2-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1520499	Environment:
Last Closed:	2019-03-04 15:14:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1473349, 1520499
Bug Blocks:

Description Barnaby Court 2017-12-04 15:23:41 UTC

+++ This bug was initially created as a clone of Bug #1520499 +++
Cloning to fix in master

+++ This bug was initially created as a clone of Bug #1520357 +++

Hello,

logratating does not work for candlepin in Satellite 6.3. Although it might look like it's doing something, it is not working correctly. Log rotation must happen everyday on midnight, instead logs are only created when application is restarted. What's the problem is inability of reopening files when they are rotated - Candlepin carries on writing logs into deleted/renamed file until it is restarted. Then the correct file is appended again. This is a big problem when debugging customer calls as we end up with files of big sizes in our debug report or sos report.

Currently (Satellite 6.3) candlepin uses log4j/logstash with simple FileAppender and there is no way of telling it to reopen logs.

TRIAGE NOTES: Please do backport for us, create some simple way of telling the log4j stack to reopen logs (SIGHUP or HTTP endpoint) and use that in /etc/logrotate.d/candlepin file. Make sure it works under SELinux (logrotate runs confined and often cause problems).

SIDENOTE: Long term I recommend to consider using syslog/journad for all logs. This way, rotation is solved problem as it is provided by logging daemon "for free" (plus it enables sending logs over network - planned feature for future Satellite version). If you choose to use syslog for