Bug 1939326

Summary: ATOP is not collecting logs at mentioned interval
Product: [Fedora] Fedora EPEL Reporter: SASIKUMAR K <sasi1212>
Component: atopAssignee: Gwyn Ciesla <gwync>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: epel7CC: andy, gerlof.langeveld, gwync, sasi1212
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: atop-2.6.0-5.fc34 atop-2.6.0-5.fc33 atop-2.6.0-5.fc32 atop-2.6.0-5.el7 atop-2.6.0-5.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-28 00:15:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
SOSreport - Part A
none
SOSreport - Part B none

Description SASIKUMAR K 2021-03-16 05:04:47 UTC
Description of problem:
ATOP RPM provided by EPEL is not working. It is not collecting logs at the mentioned interval, it collects logs only one that it during the atop process start-up.


Version-Release number of selected component (if applicable):
# rpm -qi atop
Name        : atop
Version     : 2.6.0
Release     : 3.el7
Architecture: x86_64
Install Date: Tue 16 Mar 2021 04:53:02 AM UTC
Group       : Unspecified
Size        : 365765
License     : GPLv2+
Signature   : RSA/SHA256, Mon 22 Feb 2021 02:44:12 PM UTC, Key ID 6a2faea2352c64e5
Source RPM  : atop-2.6.0-3.el7.src.rpm
Build Date  : Mon 22 Feb 2021 02:38:04 PM UTC
Build Host  : buildhw-x86-16.iad2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.atoptool.nl
Bug URL     : https://bugz.fedoraproject.org/atop
Summary     : An advanced interactive monitor to view the load on system and process level
Description :
An advanced interactive monitor for Linux-systems to view the load on
system-level and process-level.
The command atop has some major advantages compared to other
performance-monitors:
   - Resource consumption by all processes
   - Utilization of all relevant resources
   - Permanent logging of resource utilization
   - Highlight critical resources
   - Watch activity only
   - Watch deviations only
   - Accumulated process activity per user
   - Accumulated process activity per program
For more informations: http://www.atcomputing.nl/Tools/atop
The package does not make use of the patches available at
http://www.atcomputing.nl/Tools/atop/kernpatch.html

How reproducible:
Always.

Steps to Reproduce:
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum -y install atop --enablerepo=epel
sudo sed -i 's/^LOGINTERVAL=600.*/LOGINTERVAL=60/' /etc/sysconfig/atop
sudo systemctl enable atop.service
sudo systemctl restart atop.service

Actual results:
The logs are not getting generated every minute.

Expected results:
The logs should be generated every minute.


Additional info:
Tried the replication steps few times in different RHEL7.7 instances, and I was able to reproduce the same.
I use the RPM from https://www.atoptool.nl/downloadatop.php and it works fine. The problem is very specific to RPM available in EPEL repository.

Hence, I request Red Hat team to validate the problem statement and investigate the same.

Comment 1 Gwyn Ciesla 2021-03-16 13:49:36 UTC
Does sudo systemctl daemon-reload, followed by sudo systemctl restart atop, help?

Comment 2 SASIKUMAR K 2021-03-18 02:37:05 UTC
It neither helps. Attached a sosreport just in case.

$ sudo systemctl status atop.service
● atop.service - Atop advanced performance monitor
   Loaded: loaded (/usr/lib/systemd/system/atop.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-03-18 02:31:13 UTC; 45s ago
     Docs: man:atop(1)
  Process: 1433 ExecStartPost=/usr/bin/find ${LOGPATH} -name atop_* -mtime +${LOGGENERATIONS} -exec rm -v {} ; (code=exited, status=0/SUCCESS)
  Process: 1430 ExecStartPre=/bin/sh -c test -n "$LOGGENERATIONS" -a "$LOGGENERATIONS" -eq "$LOGGENERATIONS" (code=exit
ed, status=0/SUCCESS)
  Process: 1429 ExecStartPre=/bin/sh -c test -n "$LOGINTERVAL" -a "$LOGINTERVAL" -eq "$LOGINTERVAL" (code=exited, status=0/SUCCESS)
 Main PID: 1432 (atop)
   CGroup: /system.slice/atop.service
           └─1432 /usr/bin/atop -w /var/log/atop/atop_20210318 60

Mar 18 02:31:13 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Starting Atop advanced performance monitor...
Mar 18 02:31:13 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Started Atop advanced performance monitor.
Hint: Some lines were ellipsized, use -l to show in full.

$ sudo systemctl daemon-reload
$ sudo systemctl restart atop
$ sudo systemctl status atop.service
● atop.service - Atop advanced performance monitor
   Loaded: loaded (/usr/lib/systemd/system/atop.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-03-18 02:32:18 UTC; 3s ago
     Docs: man:atop(1)
  Process: 1482 ExecStartPost=/usr/bin/find ${LOGPATH} -name atop_* -mtime +${LOGGENERATIONS} -exec rm -v {} ; (code=exited, status=0/SUCCESS)
  Process: 1479 ExecStartPre=/bin/sh -c test -n "$LOGGENERATIONS" -a "$LOGGENERATIONS" -eq "$LOGGENERATIONS" (code=exited, status=0/SUCCESS)
  Process: 1478 ExecStartPre=/bin/sh -c test -n "$LOGINTERVAL" -a "$LOGINTERVAL" -eq "$LOGINTERVAL" (code=exited, status=0/SUCCESS)
 Main PID: 1481 (atop)
   CGroup: /system.slice/atop.service
           └─1481 /usr/bin/atop -w /var/log/atop/atop_20210318 60

Mar 18 02:32:18 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Stopped Atop advanced performance monitor.
Mar 18 02:32:18 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Starting Atop advanced performance monitor...
Mar 18 02:32:18 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Started Atop advanced performance monitor.
Hint: Some lines were ellipsized, use -l to show in full.

$ watch -n1 "date; ls -l /var/log/atop"
$ date; ls -l /var/log/atop
Thu Mar 18 02:34:11 UTC 2021
total 24
-rw-r--r--. 1 root root 20654 Mar 18 02:32 atop_20210318

$ sudo ps -ef | grep atop
root      1481     1  0 02:32 ?        00:00:00 /usr/bin/atop  -w /var/log/atop/atop_20210318 60
ec2-user  1762  1303  0 02:34 pts/0    00:00:00 grep --color=auto atop

$ date; ls -l /var/log/atop
Thu Mar 18 02:36:04 UTC 2021
total 24
-rw-r--r--. 1 root root 20654 Mar 18 02:32 atop_20210318

Comment 3 SASIKUMAR K 2021-03-18 03:04:38 UTC
Created attachment 1764261 [details]
SOSreport - Part A

$ md5sum sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTaa
bc489a5e84b0711cd48a07d0ddc06cc2  sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTaa

Comment 4 SASIKUMAR K 2021-03-18 03:18:39 UTC
Created attachment 1764262 [details]
SOSreport - Part B

$ md5sum sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTab
506e8b34bac50cacd226554c9fe5317f  sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTab

Merge steps:
$ cat sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTa* > sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz

$ md5sum sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz

Comment 5 SASIKUMAR K 2021-03-18 03:21:21 UTC
$ md5sum sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz
0d85c3b52b8523a08d9879fba4b14de3  sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz

Comment 6 Gwyn Ciesla 2021-03-18 13:48:28 UTC
There seem to be some differences between the upstream atop rpm's /etc/default/atop and our /etc/sysconfig/atop. If you use there file in our location, do you see a change?

Comment 7 SASIKUMAR K 2021-03-22 18:32:15 UTC
Good guess. But it does not help.

$ cat /etc/default/atop
LOGOPTS=""
LOGINTERVAL=60
LOGGENERATIONS=28
LOGPATH=/var/log/atop

$ ls -lZ /etc/default/atop
-rw-r--r--. root root system_u:object_r:etc_t:s0       /etc/default/atop

$ sudo systemctl restart atop

--- Some moments later ---

$ date; ls -l /var/log/atop
Mon Mar 22 18:26:18 UTC 2021
total 256
-rw-r--r--. 1 root root 213606 Mar 18 13:00 atop_20210318
-rw-r--r--. 1 root root  43804 Mar 22 18:24 atop_20210322

As you made me to compare ATOP RPM from EPEL and the one from upstream, I noted Python3 gets installed along with the upstream version, whereas it is not getting installed in EPEL version. I installed Python3 and restarted atop. But my guess is also not correct.

Comment 8 Gwyn Ciesla 2021-03-22 21:13:27 UTC
Interesting. The python difference is because in EPEL we don't ship atopgpud because the python3-nvml module isn't available.

Comment 9 SASIKUMAR K 2021-03-23 06:18:26 UTC
Thanks for your update.

Let me know in case if you need any further information from my lab.

Comment 10 Gwyn Ciesla 2021-03-23 19:45:44 UTC
I've confirmed that 100% replacement of the file we ship as /etc/sysconfig/atop with the file upstream ships as /etc/sysconfig/atop fixes the issue. I'm pushing an update today. When you have it installed, replace your /etc/sysconfig/atop with /etc/sysconfig/atop.rpmnew, and customize.

Comment 11 Fedora Update System 2021-03-23 20:03:58 UTC
FEDORA-2021-f238ed0ea7 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-f238ed0ea7

Comment 12 Fedora Update System 2021-03-23 20:03:59 UTC
FEDORA-EPEL-2021-6b39871d3f has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-6b39871d3f

Comment 13 Fedora Update System 2021-03-23 20:04:00 UTC
FEDORA-2021-0f367a5b54 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-0f367a5b54

Comment 14 Fedora Update System 2021-03-23 20:04:00 UTC
FEDORA-2021-956bbc2742 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2021-956bbc2742

Comment 15 Fedora Update System 2021-03-23 20:04:01 UTC
FEDORA-EPEL-2021-631b5d8a32 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-631b5d8a32

Comment 16 Fedora Update System 2021-03-24 02:45:00 UTC
FEDORA-2021-0f367a5b54 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-0f367a5b54`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-0f367a5b54

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2021-03-24 02:56:46 UTC
FEDORA-2021-f238ed0ea7 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-f238ed0ea7`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-f238ed0ea7

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2021-03-24 03:13:52 UTC
FEDORA-EPEL-2021-6b39871d3f has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-6b39871d3f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 19 Fedora Update System 2021-03-24 03:21:55 UTC
FEDORA-EPEL-2021-631b5d8a32 has been pushed to the Fedora EPEL 7 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-631b5d8a32

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 20 Fedora Update System 2021-03-24 03:43:57 UTC
FEDORA-2021-956bbc2742 has been pushed to the Fedora 32 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-956bbc2742`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-956bbc2742

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 21 SASIKUMAR K 2021-03-25 06:04:31 UTC
I previously tested with the file "/etc/default/atop". However, after referring your recent correspondence I understood that we need to use the contents of upstream file "/etc/default/atop" and use it as "/etc/sysconfig/atop". And yes, it works, here are the results from my lab.

$ cat /etc/sysconfig/atop.rpmsave
# sysconfig atop
#

# Current Day format
CURDAY=`date +%Y%m%d`
# Log files path
LOGPATH=/var/log/atop
# Binaries path
BINPATH=/usr/bin
# PID File
PIDFILE=/var/run/atop.pid
# interval (default 10 minutes)
LOGINTERVAL=60

$ cat /etc/sysconfig/atop
LOGOPTS=""
LOGINTERVAL=60
LOGGENERATIONS=28
LOGPATH=/var/log/atop

[ec2-user@ip-172-31-36-207 ~]$ date; ls -l /var/log/atop
Thu Mar 25 05:56:53 UTC 2021
total 484
-rw-r--r--. 1 root root 213606 Mar 18 13:00 atop_20210318
-rw-r--r--. 1 root root 157803 Mar 23 13:00 atop_20210322
-rw-r--r--. 1 root root  54767 Mar 25 05:56 atop_20210325
-rw-r--r--. 1 root root  22519 Mar 25 05:48 atop_20210325.save

I have also tested the patched version 2.6.0-5 and it resolves the problem completely. 

https://kojipkgs.fedoraproject.org//packages/atop/2.6.0/5.el7/x86_64/atop-2.6.0-5.el7.x86_64.rpm

Thanks a lot Gwyn Ciesla for fixing it. Really appreciate you.

Comment 22 Gwyn Ciesla 2021-03-25 13:30:48 UTC
You're very welcome!

Comment 23 gerlof.langeveld 2021-03-27 14:39:53 UTC
I recently discovered that the real issue is in the atop.service file, where you can find:

Environment=LOGOPTS=""

The two double-quotes ("") cause an empty command line argument for atop, that reacts by taking an interval time of 0 seconds (which means: no time-driven intervals any more). This environment variable LOGOPTS is overruled by the /etc/default/atop file, so it also overrules the wrong specification in the atop.service file. LOGOPTS was not set in the original /etc/sysconfig/atop file, and therefor the wrong specification in the atop.service file was not overruled.

Please remove the two double-quotes in the atop.service file.

Comment 24 Fedora Update System 2021-03-28 00:15:39 UTC
FEDORA-2021-0f367a5b54 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 25 Gwyn Ciesla 2021-03-29 14:57:27 UTC
What we now ship are the files provided in 2.6.0, with LOGOPTS="" in both the defaults/sysconfig file and the unit file, and that works in my testing. Will there be a release to address this in the near future?

Comment 26 Fedora Update System 2021-04-01 01:50:28 UTC
FEDORA-2021-f238ed0ea7 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 27 Fedora Update System 2021-04-01 01:54:39 UTC
FEDORA-2021-956bbc2742 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 28 Fedora Update System 2021-04-07 15:48:22 UTC
FEDORA-EPEL-2021-631b5d8a32 has been pushed to the Fedora EPEL 7 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 29 Fedora Update System 2021-04-07 15:56:50 UTC
FEDORA-EPEL-2021-6b39871d3f has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.