RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1803783 - [RHEL/Bug] log rotation of rhsm.log not correctly handled
Summary: [RHEL/Bug] log rotation of rhsm.log not correctly handled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: subscription-manager
Version: 7.6
Hardware: All
OS: Linux
high
medium
Target Milestone: rc
: ---
Assignee: Jiri Hnidek
QA Contact: Red Hat subscription-manager QE Team
URL:
Whiteboard:
Depends On:
Blocks: 1122832 1827103
TreeView+ depends on / blocked
 
Reported: 2020-02-17 12:54 UTC by Oliver Falk
Modified: 2023-09-07 21:53 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1827103 (view as bug list)
Environment:
Last Closed: 2020-09-29 19:22:47 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github candlepin subscription-manager pull 2225 0 None closed 1803783: Added copytruncate option to logrotate conf file; ENT-2114 2021-01-22 09:43:22 UTC
Red Hat Knowledge Base (Solution) 5088661 0 None None None 2020-05-19 13:15:23 UTC
Red Hat Product Errata RHBA-2020:3866 0 None None None 2020-09-29 19:23:17 UTC

Description Oliver Falk 2020-02-17 12:54:47 UTC 
Description of problem:

rhsm.log is rotated by '/etc/logrotate.d/subscription-manager'. However, this logrotate config doesn't correctly force the processes to reopen log files or otherwise take care about processes having opened the /var/log/rhsm/rhsm.log.


Version-Release number of selected component (if applicable): 1.24.13 (but probably all other versions as well).


How reproducible: Always


Steps to Reproduce:
1. Have subscription-manager installed/running (rhsmd)
2. # logrotate -f /etc/logrotate.conf
3. # lsof | grep deleted | grep rhsm

Actual results:

    # System w/ virt-who
    # lsof | grep deleted | grep rhsm
    rhsmd      1639                           root    4w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    rhsmd      1639                           root    8u      REG             253,11       4096         76 /tmp/ffi4TtlUS (deleted)
    virt-who   1964                           root    6w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964                           root    7w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964                           root    9w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964  2473                     root    6w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964  2473                     root    7w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964  2473                     root    9w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964  2481                     root    6w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964  2481                     root    7w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964  2481                     root    9w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964  2483                     root    6w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964  2483                     root    7w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)
    virt-who   1964  2483                     root    9w      REG              253,0   17020466   52066240 /var/log/rhsm/rhsm.log-20200216 (deleted)

    # System w/o virt-who
    # lsof | grep deleted | grep rhsm
    rhsmd      1167                           root    4w      REG              253,0  11598316      87353 /var/log/rhsm/rhsm.log-20200119 (deleted)
    rhsmd      1167                           root    6u      REG              253,5      4096         97 /tmp/ffiUFtUfE (deleted)
    gmain      1167  1270                     root    4w      REG              253,0  11598316      87353 /var/log/rhsm/rhsm.log-20200119 (deleted)


Expected results:
logrotate should handle the rotation of logfiles either in a way that the processes do not need to reopen the file handle (eg. copytruncate) or send the appropriate signal to the processes that have an open file handle.


Additional info:
Customer case will be linked.
Comment 12 Rehana 2020-05-12 12:15:21 UTC 
Verifying on : 

[root@ibm-x3650m4-01-vm-08 rhsm]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 2.9.27-1
subscription management rules: 5.37
subscription-manager: 1.24.37

[root@ibm-x3650m4-01-vm-08 rhsm]# rpm -qa subscription-manager* --changelog | grep 1803783
- 1803783: Added copytruncate option to logrotate conf file; ENT-2114
- 1803783: Added copytruncate option to logrotate conf file; ENT-2114
- 1803783: Added copytruncate option to logrotate conf file; ENT-2114

[root@ibm-x3650m4-01-vm-08 rhsm]# cat /etc/logrotate.d/subscription-manager 
/var/log/rhsm/*.log {
    missingok
    notifempty
    sharedscripts
    copytruncate
}

Make sure the services are running 

[root@ibm-x3650m4-01-vm-08 rhsm]# ps -aux | grep rhsmd
root     21025 19.0  0.7 478180 28384 pts/1    Sl+  08:05   0:00 /usr/bin/python /usr/libexec/rhsmd -k
root     21038  0.0  0.0 112808   936 pts/0    S+   08:05   0:00 grep --color=auto rhsmd

[root@ibm-x3650m4-01-vm-08 rhsm]# ps -aux | grep virt-who
root      9934  0.0  0.7 819536 30032 ?        Ssl  06:39   0:01 /usr/bin/python /usr/bin/virt-who
root     21040  0.0  0.0 112808   936 pts/0    S+   08:05   0:00 grep --color=auto virt-who

Execute any subscription manager commands to generate new logs 

[root@ibm-x3650m4-01-vm-08 rhsm]# subscription-manager  attach --auto
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

Execute logroate -f /etc/logrotate.conf

[root@ibm-x3650m4-01-vm-08 rhsm]# logrotate -f /etc/logrotate.conf

[root@ibm-x3650m4-01-vm-08 rhsm]# ll
total 16
-rw-r--r--. 1 root root    0 May 12 06:50 rhsmcertd.log
-rw-r--r--. 1 root root  491 May 12 06:50 rhsmcertd.log-20200512
-rw-r--r--. 1 root root    0 May 12 08:07 rhsm.log
-rw-r--r--. 1 root root 8416 May 12 08:07 rhsm.log-20200512

Delete the file rhsm.log-20200512 

[root@ibm-x3650m4-01-vm-08 rhsm]# rm -rf rhsm.log-20200512 

Notice that the service are still running 

[root@ibm-x3650m4-01-vm-08 rhsm]# ps -aux | grep rhsmd
root     21025  0.2  0.8 564540 31104 pts/1    Sl+  08:05   0:00 /usr/bin/python /usr/libexec/rhsmd -k
root     21119  0.0  0.0 112808   932 pts/0    S+   08:08   0:00 grep --color=auto rhsmd
[root@ibm-x3650m4-01-vm-08 rhsm]#  ps -aux | grep virt-who
root      9934  0.0  0.7 819536 30032 ?        Ssl  06:39   0:01 /usr/bin/python /usr/bin/virt-who
root     21121  0.0  0.0 112808   936 pts/0    S+   08:08   0:00 grep --color=auto virt-who

Lets the lsof results for rhsm file 

[root@ibm-x3650m4-01-vm-08 rhsm]# lsof | grep deleted | grep rhsm*

[root@ibm-x3650m4-01-vm-08 rhsm]# lsof | grep deleted | grep rhsm
rhsmd     21025          root    6u      REG              253,0      4096  100688378 /tmp/#100688378 (deleted)

[root@ibm-x3650m4-01-vm-08 rhsm]# ll
total 4
-rw-r--r--. 1 root root   0 May 12 06:50 rhsmcertd.log
-rw-r--r--. 1 root root 491 May 12 06:50 rhsmcertd.log-20200512
-rw-r--r--. 1 root root   0 May 12 08:07 rhsm.log

^^ Notice there are no reference to the deleted rhsm log file. 

Based on the above observation moving the bug to verified
Comment 14 errata-xmlrpc 2020-09-29 19:22:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3866
Note You need to log in before you can comment on or make changes to this bug.