Bug 1750248 - Rotated and deleted log files kept open until smart-proxy is restarted
Summary: Rotated and deleted log files kept open until smart-proxy is restarted
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Capsule
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: 6.7.0
Assignee: Lukas Zapletal
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-09 07:01 UTC by Lukas Zapletal
Modified: 2020-04-14 13:25 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Rotated log files are not reopened even when a HUP signal is sent to the foreman-proxy daemon. As a result, deleted files are reported as still open, logging does not work correctly, and log records can be lost. You can check for deleted files by running the following command: + ---- lsof | grep deleted | grep "\/var\/log" ---- + To work around this issue, restart the `foreman-proxy` service after logs are rotated.
Clone Of:
: 1769894 (view as bug list)
Environment:
Last Closed: 2020-04-14 13:25:37 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 27509 Normal Closed Smart-proxy USR1 reopen does not work 2020-07-07 14:46:14 UTC
Red Hat Product Errata RHSA-2020:1454 None None None 2020-04-14 13:25:49 UTC

Description Lukas Zapletal 2019-09-09 07:01:47 UTC
After refactoring to logging gem the USR1 "reopen" UNIX signal no longer works as reported at:

https://community.theforeman.org/t/foreman-proxy-open-fd-of-deleted-log-files-hdd-out-of-space/14481/4

This can cause smart-proxy process to consume ALL space on /var eventually. This is important bug to fix for 6.6, thus raising this. Fixed upstream.

https://projects.theforeman.org/issues/27509

QA: To test this, rotate logs (or wait few days) and then:

# lsof | grep deleted | grep "\/var\/log"
ruby      16887       foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby-time 16887 16888 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16891 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16893 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16894 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16984 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16985 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)
ruby      16887 16986 foreman-proxy    7u      REG              253,0      6647  135126473 /var/log/foreman-proxy/proxy.log-20190907 (deleted)

There should be no deleted files reported by smart-proxy service via lsof.

TEST THIS WITH PERMISSIVE MODE because another bug prevents logrotate from sending UNIX signals: https://bugzilla.redhat.com/show_bug.cgi?id=1750030 and this bug is really about signal handing rather than signal not being passed through.

Comment 5 Lukas Pramuk 2019-11-27 10:44:44 UTC
VERIFIED.

@satellite-6.7.0-4.beta.el7sat.noarch
foreman-proxy-1.24.0-0.4.RC2.el7sat.noarch

by the following manual reproducer:

1) Check that foreman-proxy is running

# ps -efH | grep foreman\\-proxy
foreman+ 18267     1  0 Nov26 ?        00:00:07   ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize
foreman+ 18556     1  0 Nov26 ?        00:00:13   ruby /usr/bin/smart_proxy_dynflow_core -d -p /var/run/foreman-proxy/smart_proxy_dynflow_core.pid

2) Check that there are already logrotated files (as Satellite is running for couple of days)

# ll /var/log/foreman-proxy/proxy*.gz
-rw-r--r--. 1 foreman-proxy foreman-proxy  222 Nov 25 23:51 /var/log/foreman-proxy/proxy.log-20191126.gz
-rw-r--r--. 1 foreman-proxy foreman-proxy 2532 Nov 26 23:51 /var/log/foreman-proxy/proxy.log-20191127.gz

3) Check for presence of logrotated (deleted) log files being kept open

# lsof | grep '/var/log.* (deleted)'
<empty>

>>> the foreman-proxy log files are logrotated properly without keeping deleted files open

Comment 8 errata-xmlrpc 2020-04-14 13:25:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454


Note You need to log in before you can comment on or make changes to this bug.