Bug 1750248

Summary:	Rotated and deleted log files kept open until smart-proxy is restarted
Product:	Red Hat Satellite	Reporter:	Lukas Zapletal <lzap>
Component:	Foreman Proxy	Assignee:	Lukas Zapletal <lzap>
Status:	CLOSED ERRATA	QA Contact:	Lukas Pramuk <lpramuk>
Severity:	medium	Docs Contact:
Priority:	high
Version:	6.6.0	CC:	aruzicka, bkearney, inecas, mawerner, mschibli, pcreech, pdudley, spetrosi, trichard
Target Milestone:	6.7.0	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Known Issue
Doc Text:	Rotated log files are not reopened even when a HUP signal is sent to the foreman-proxy daemon. As a result, deleted files are reported as still open, logging does not work correctly, and log records can be lost. You can check for deleted files by running the following command: + ---- lsof \| grep deleted \| grep "\/var\/log" ---- + To work around this issue, restart the `foreman-proxy` service after logs are rotated.	Story Points:	---
Clone Of:
Clones:	1769894 (view as bug list)		Environment:
Last Closed:	2020-04-14 13:25:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Lukas Zapletal 2019-09-09 07:01:47 UTC

After refactoring to logging gem the USR1 "reopen" UNIX signal no longer works as reported at:

https://community.theforeman.org/t/foreman-proxy-open-fd-of-deleted-log-files-hdd-out-of-space/14481/4

This can cause smart-proxy process to consume ALL space on /var eventually. This is important bug to fix for 6.6, thus raising this. Fixed upstream.

https://projects.theforeman.org/issues/27509

QA: To test this, rotate logs (or wait few days) and then:

# lsof | grep deleted | grep "\/var\/log"
ruby      16887       foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby-time 16887 16888 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16891 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16893 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16894 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16984 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16985 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16986 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)

There should be no deleted files reported by smart-proxy service via lsof.

TEST THIS WITH PERMISSIVE MODE because another bug prevents logrotate from sending UNIX signals: https://bugzilla.redhat.com/show_bug.cgi?id=1750030 and this bug is really about signal handing rather than signal not being passed through.

Comment 5 Lukas Pramuk 2019-11-27 10:44:44 UTC

VERIFIED.

@satellite-6.7.0-4.beta.el7sat.noarch
foreman-proxy-1.24.0-0.4.RC2.el7sat.noarch

by the following manual reproducer:

1) Check that foreman-proxy is running

# ps -efH | grep foreman\\-proxy
foreman+ 18267     1  0 Nov26 ?        00:00:07   ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize
foreman+ 18556     1  0 Nov26 ?        00:00:13   ruby /usr/bin/smart_proxy_dynflow_core -d -p /var/run/foreman-proxy/smart_proxy_dynflow_core.pid

2) Check that there are already logrotated files (as Satellite is running for couple of days)

# ll /var/log/foreman-proxy/proxy*.gz
-rw-r--r--. 1 foreman-proxy foreman-proxy  222 Nov 25 23:51 /var/log/foreman-proxy/proxy.log-20191126.gz
-rw-r--r--. 1 foreman-proxy foreman-proxy 2532 Nov 26 23:51 /var/log/foreman-proxy/proxy.log-20191127.gz

3) Check for presence of logrotated (deleted) log files being kept open

# lsof | grep '/var/log.* (deleted)'
<empty>

>>> the foreman-proxy log files are logrotated properly without keeping deleted files open

Comment 8 errata-xmlrpc 2020-04-14 13:25:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454