Bug 1827103

Summary: [RHEL/Bug] log rotation of rhsm.log not correctly handled [rhel-7.8.z]
Product: Red Hat Enterprise Linux 7 Reporter: RAD team bot copy to z-stream <autobot-eus-copy>
Component: subscription-managerAssignee: candlepin-bugs
Status: CLOSED ERRATA QA Contact: Red Hat subscription-manager QE Team <rhsm-qe>
Severity: medium Docs Contact:
Priority: high    
Version: 7.6CC: csnyder, jhnidek, jsefler, kagarwal, ofalk, peter.vreman, redakkan, wpoteat
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1803783 Environment:
Last Closed: 2020-05-12 18:40:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1803783    
Bug Blocks:    

Description RAD team bot copy to z-stream 2020-04-23 09:45:04 UTC
This bug has been copied from bug #1803783 and has been proposed to be backported to 7.8 z-stream (EUS).

Comment 4 Rehana 2020-04-28 07:11:08 UTC
Reproducing the issue : 
========================
Register and subscribe the system 
with virt-who /rhsmd running 

Steps 1: Make virt-who /rhsmd running ;
============================
#service virt-who restart and
# /usr/libexec/rhsmd -k in terminal

Step 2 : execute some sub-man commands to create log entries
=============================================
[root@kvm-06-guest10 rhsm]# subscription-manager remove --all
1 local certificate has been deleted.
1 subscription removed at the server.
 
Step 3 : force logrotate to run
======================
[root@kvm-06-guest10 rhsm]# logrotate -f /etc/logrotate.conf
 
[root@kvm-06-guest10 rhsm]# ls
rhsmcertd.log  rhsmcertd.log-20200422  rhsm.log  rhsm.log-20200422
 
^^ notice "rhsm.log-20200422" is now created
 
Step 4: Grep for open files
===================
 
[root@kvm-06-guest10 rhsm]# sudo lsof | grep deleted | grep rhsm
rhsmd     21406          root    6u      REG              253,0      4096  100709773 /tmp/ffib6dPFX (deleted)
[root@kvm-06-guest10 rhsm]# rm -rf rhsm.log-20200422

^^ Delete the file 

[root@kvm-06-guest10 rhsm]# sudo lsof | grep deleted | grep rhsm
rhsmd     21406          root    4w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
rhsmd     21406          root    6u      REG              253,0      4096  100709773 /tmp/ffib6dPFX (deleted)
gmain     21406 21417    root    4w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871          root    5w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871          root    6w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871          root    7w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21876    root    5w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21876    root    6w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21876    root    7w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21877    root    5w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21877    root    6w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21877    root    7w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21878    root    5w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21878    root    6w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21878    root    7w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21879    root    5w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21879    root    6w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
virt-who  21871 21879    root    7w      REG              253,0     20908   34568341 /var/log/rhsm/rhsm.log-20200422 (deleted)
 
[root@kvm-06-guest10 rhsm]# cat /etc/logrotate.d/subscription-manager
/var/log/rhsm/*.log {
    missingok
    notifempty
    sharedscripts
}


Verifying on RHEL 78 with updating build from internal 7.8z brew build ( not shown ) 
==================================================================================
subscription management server: 2.9.21-1
subscription management rules: 5.37
subscription-manager: 1.24.26-3.el7_8

[root@kvm-01-guest27 rhsm]# cat /etc/logrotate.d/subscription-manager 
/var/log/rhsm/*.log {
    missingok
    notifempty
    sharedscripts
    copytruncate  ---> Notice the new "copytruncate" option added to the logrotate subscription manager file 
}

Register the system 

[root@kvm-01-guest27 rhsm]# subscription-manager register --username=**** --password=****
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
The system has been registered with ID: d8d53e95-4c8a-43fc-ab13-af28b5e88399
The registered system name is: kvm-01-guest27.lab.eng.rdu2.redhat.com

Make sure the services are running 

[root@kvm-01-guest27 rhsm]# ps -aux | grep virt-who
root     20829  0.0  0.7 965288 30292 ?        Ssl  02:52   0:00 /usr/bin/python /usr/bin/virt-who
root     21313  0.0  0.0 112808   968 pts/0    R+   03:05   0:00 grep --color=auto virt-who
[root@kvm-01-guest27 rhsm]# ps -aux | grep rhsmd
root     20801  0.0  0.8 564560 32672 pts/1    Sl+  02:52   0:00 /usr/bin/python /usr/libexec/rhsmd -k
root     21315  0.0  0.0 112808   968 pts/0    R+   03:05   0:00 grep --color=auto rhsmd
[root@kvm-01-guest27 rhsm]# 

Execute logroate -f /etc/logrotate.conf

[root@kvm-01-guest27 rhsm]# logrotate -f /etc/logrotate.conf
[root@kvm-01-guest27 rhsm]# ll
total 8
-rw-r--r--. 1 root root    0 Apr 28 02:53 rhsmcertd.log
-rw-r--r--. 1 root root    0 Apr 28 03:06 rhsm.log
-rw-r--r--. 1 root root 4622 Apr 28 03:06 rhsm.log-20200428


Delete the file rhsm.log-20200428


[root@kvm-01-guest27 rhsm]# rm -rf rhsm.log-20200428 

Notice that the service are still running 

[root@kvm-01-guest27 rhsm]# ps -aux | grep rhsmd
root     20801  0.0  0.8 564560 32672 pts/1    Sl+  02:52   0:00 /usr/bin/python /usr/libexec/rhsmd -k
root     21320  0.0  0.0 112808   968 pts/0    R+   03:07   0:00 grep --color=auto rhsmd
[root@kvm-01-guest27 rhsm]# ps -aux | grep virt-who
root     20829  0.0  0.8 965288 32316 ?        Ssl  02:52   0:00 /usr/bin/python /usr/bin/virt-who
root     21322  0.0  0.0 112808   968 pts/0    R+   03:07   0:00 grep --color=auto virt-who


Lets the lsof results for rhsm file 

[root@kvm-01-guest27 rhsm]# lsof | grep deleted | grep rhsm*
[root@kvm-01-guest27 rhsm]# lsof | grep deleted | grep rhsm
rhsmd     20801          root    6u      REG              253,0      4096  100664085 /tmp/#100664085 (deleted)
[root@kvm-01-guest27 rhsm]# ll
total 0
-rw-r--r--. 1 root root 0 Apr 28 02:53 rhsmcertd.log
-rw-r--r--. 1 root root 0 Apr 28 03:06 rhsm.log


^^ Notice there are no reference to the deleted rhsm log file. 

Based on the above observation moving the bug to verified

Comment 6 errata-xmlrpc 2020-05-12 18:40:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2101