Bug 1939326 - ATOP is not collecting logs at mentioned interval
Summary: ATOP is not collecting logs at mentioned interval
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: atop
Version: epel7
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Gwyn Ciesla
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-16 05:04 UTC by SASIKUMAR K
Modified: 2021-04-12 14:36 UTC (History)
4 users (show)

Fixed In Version: atop-2.6.0-5.fc34 atop-2.6.0-5.fc33 atop-2.6.0-5.fc32 atop-2.6.0-5.el7 atop-2.6.0-5.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-28 00:15:39 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
SOSreport - Part A (12.00 MB, application/x-xz)
2021-03-18 03:04 UTC, SASIKUMAR K
no flags Details
SOSreport - Part B (8.57 MB, application/octet-stream)
2021-03-18 03:18 UTC, SASIKUMAR K
no flags Details

Description SASIKUMAR K 2021-03-16 05:04:47 UTC
Description of problem:
ATOP RPM provided by EPEL is not working. It is not collecting logs at the mentioned interval, it collects logs only one that it during the atop process start-up.


Version-Release number of selected component (if applicable):
# rpm -qi atop
Name        : atop
Version     : 2.6.0
Release     : 3.el7
Architecture: x86_64
Install Date: Tue 16 Mar 2021 04:53:02 AM UTC
Group       : Unspecified
Size        : 365765
License     : GPLv2+
Signature   : RSA/SHA256, Mon 22 Feb 2021 02:44:12 PM UTC, Key ID 6a2faea2352c64e5
Source RPM  : atop-2.6.0-3.el7.src.rpm
Build Date  : Mon 22 Feb 2021 02:38:04 PM UTC
Build Host  : buildhw-x86-16.iad2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.atoptool.nl
Bug URL     : https://bugz.fedoraproject.org/atop
Summary     : An advanced interactive monitor to view the load on system and process level
Description :
An advanced interactive monitor for Linux-systems to view the load on
system-level and process-level.
The command atop has some major advantages compared to other
performance-monitors:
   - Resource consumption by all processes
   - Utilization of all relevant resources
   - Permanent logging of resource utilization
   - Highlight critical resources
   - Watch activity only
   - Watch deviations only
   - Accumulated process activity per user
   - Accumulated process activity per program
For more informations: http://www.atcomputing.nl/Tools/atop
The package does not make use of the patches available at
http://www.atcomputing.nl/Tools/atop/kernpatch.html

How reproducible:
Always.

Steps to Reproduce:
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum -y install atop --enablerepo=epel
sudo sed -i 's/^LOGINTERVAL=600.*/LOGINTERVAL=60/' /etc/sysconfig/atop
sudo systemctl enable atop.service
sudo systemctl restart atop.service

Actual results:
The logs are not getting generated every minute.

Expected results:
The logs should be generated every minute.


Additional info:
Tried the replication steps few times in different RHEL7.7 instances, and I was able to reproduce the same.
I use the RPM from https://www.atoptool.nl/downloadatop.php and it works fine. The problem is very specific to RPM available in EPEL repository.

Hence, I request Red Hat team to validate the problem statement and investigate the same.

Comment 1 Gwyn Ciesla 2021-03-16 13:49:36 UTC
Does sudo systemctl daemon-reload, followed by sudo systemctl restart atop, help?

Comment 2 SASIKUMAR K 2021-03-18 02:37:05 UTC
It neither helps. Attached a sosreport just in case.

$ sudo systemctl status atop.service
● atop.service - Atop advanced performance monitor
   Loaded: loaded (/usr/lib/systemd/system/atop.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-03-18 02:31:13 UTC; 45s ago
     Docs: man:atop(1)
  Process: 1433 ExecStartPost=/usr/bin/find ${LOGPATH} -name atop_* -mtime +${LOGGENERATIONS} -exec rm -v {} ; (code=exited, status=0/SUCCESS)
  Process: 1430 ExecStartPre=/bin/sh -c test -n "$LOGGENERATIONS" -a "$LOGGENERATIONS" -eq "$LOGGENERATIONS" (code=exit
ed, status=0/SUCCESS)
  Process: 1429 ExecStartPre=/bin/sh -c test -n "$LOGINTERVAL" -a "$LOGINTERVAL" -eq "$LOGINTERVAL" (code=exited, status=0/SUCCESS)
 Main PID: 1432 (atop)
   CGroup: /system.slice/atop.service
           └─1432 /usr/bin/atop -w /var/log/atop/atop_20210318 60

Mar 18 02:31:13 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Starting Atop advanced performance monitor...
Mar 18 02:31:13 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Started Atop advanced performance monitor.
Hint: Some lines were ellipsized, use -l to show in full.

$ sudo systemctl daemon-reload
$ sudo systemctl restart atop
$ sudo systemctl status atop.service
● atop.service - Atop advanced performance monitor
   Loaded: loaded (/usr/lib/systemd/system/atop.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-03-18 02:32:18 UTC; 3s ago
     Docs: man:atop(1)
  Process: 1482 ExecStartPost=/usr/bin/find ${LOGPATH} -name atop_* -mtime +${LOGGENERATIONS} -exec rm -v {} ; (code=exited, status=0/SUCCESS)
  Process: 1479 ExecStartPre=/bin/sh -c test -n "$LOGGENERATIONS" -a "$LOGGENERATIONS" -eq "$LOGGENERATIONS" (code=exited, status=0/SUCCESS)
  Process: 1478 ExecStartPre=/bin/sh -c test -n "$LOGINTERVAL" -a "$LOGINTERVAL" -eq "$LOGINTERVAL" (code=exited, status=0/SUCCESS)
 Main PID: 1481 (atop)
   CGroup: /system.slice/atop.service
           └─1481 /usr/bin/atop -w /var/log/atop/atop_20210318 60

Mar 18 02:32:18 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Stopped Atop advanced performance monitor.
Mar 18 02:32:18 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Starting Atop advanced performance monitor...
Mar 18 02:32:18 ip-172-31-36-207.ap-south-1.compute.internal systemd[1]: Started Atop advanced performance monitor.
Hint: Some lines were ellipsized, use -l to show in full.

$ watch -n1 "date; ls -l /var/log/atop"
$ date; ls -l /var/log/atop
Thu Mar 18 02:34:11 UTC 2021
total 24
-rw-r--r--. 1 root root 20654 Mar 18 02:32 atop_20210318

$ sudo ps -ef | grep atop
root      1481     1  0 02:32 ?        00:00:00 /usr/bin/atop  -w /var/log/atop/atop_20210318 60
ec2-user  1762  1303  0 02:34 pts/0    00:00:00 grep --color=auto atop

$ date; ls -l /var/log/atop
Thu Mar 18 02:36:04 UTC 2021
total 24
-rw-r--r--. 1 root root 20654 Mar 18 02:32 atop_20210318

Comment 3 SASIKUMAR K 2021-03-18 03:04:38 UTC
Created attachment 1764261 [details]
SOSreport - Part A

$ md5sum sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTaa
bc489a5e84b0711cd48a07d0ddc06cc2  sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTaa

Comment 4 SASIKUMAR K 2021-03-18 03:18:39 UTC
Created attachment 1764262 [details]
SOSreport - Part B

$ md5sum sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTab
506e8b34bac50cacd226554c9fe5317f  sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTab

Merge steps:
$ cat sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz-PARTa* > sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz

$ md5sum sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz

Comment 5 SASIKUMAR K 2021-03-18 03:21:21 UTC
$ md5sum sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz
0d85c3b52b8523a08d9879fba4b14de3  sosreport-ip-172-31-36-207-2021-03-18-bxzwuiy.tar.xz

Comment 6 Gwyn Ciesla 2021-03-18 13:48:28 UTC
There seem to be some differences between the upstream atop rpm's /etc/default/atop and our /etc/sysconfig/atop. If you use there file in our location, do you see a change?

Comment 7 SASIKUMAR K 2021-03-22 18:32:15 UTC
Good guess. But it does not help.

$ cat /etc/default/atop
LOGOPTS=""
LOGINTERVAL=60
LOGGENERATIONS=28
LOGPATH=/var/log/atop

$ ls -lZ /etc/default/atop
-rw-r--r--. root root system_u:object_r:etc_t:s0       /etc/default/atop

$ sudo systemctl restart atop

--- Some moments later ---

$ date; ls -l /var/log/atop
Mon Mar 22 18:26:18 UTC 2021
total 256
-rw-r--r--. 1 root root 213606 Mar 18 13:00 atop_20210318
-rw-r--r--. 1 root root  43804 Mar 22 18:24 atop_20210322

As you made me to compare ATOP RPM from EPEL and the one from upstream, I noted Python3 gets installed along with the upstream version, whereas it is not getting installed in EPEL version. I installed Python3 and restarted atop. But my guess is also not correct.

Comment 8 Gwyn Ciesla 2021-03-22 21:13:27 UTC
Interesting. The python difference is because in EPEL we don't ship atopgpud because the python3-nvml module isn't available.

Comment 9 SASIKUMAR K 2021-03-23 06:18:26 UTC
Thanks for your update.

Let me know in case if you need any further information from my lab.

Comment 10 Gwyn Ciesla 2021-03-23 19:45:44 UTC
I've confirmed that 100% replacement of the file we ship as /etc/sysconfig/atop with the file upstream ships as /etc/sysconfig/atop fixes the issue. I'm pushing an update today. When you have it installed, replace your /etc/sysconfig/atop with /etc/sysconfig/atop.rpmnew, and customize.

Comment 11 Fedora Update System 2021-03-23 20:03:58 UTC
FEDORA-2021-f238ed0ea7 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-f238ed0ea7

Comment 12 Fedora Update System 2021-03-23 20:03:59 UTC
FEDORA-EPEL-2021-6b39871d3f has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-6b39871d3f

Comment 13 Fedora Update System 2021-03-23 20:04:00 UTC
FEDORA-2021-0f367a5b54 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-0f367a5b54

Comment 14 Fedora Update System 2021-03-23 20:04:00 UTC
FEDORA-2021-956bbc2742 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2021-956bbc2742

Comment 15 Fedora Update System 2021-03-23 20:04:01 UTC
FEDORA-EPEL-2021-631b5d8a32 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-631b5d8a32

Comment 16 Fedora Update System 2021-03-24 02:45:00 UTC
FEDORA-2021-0f367a5b54 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-0f367a5b54`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-0f367a5b54

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2021-03-24 02:56:46 UTC
FEDORA-2021-f238ed0ea7 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-f238ed0ea7`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-f238ed0ea7

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2021-03-24 03:13:52 UTC
FEDORA-EPEL-2021-6b39871d3f has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-6b39871d3f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 19 Fedora Update System 2021-03-24 03:21:55 UTC
FEDORA-EPEL-2021-631b5d8a32 has been pushed to the Fedora EPEL 7 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-631b5d8a32

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 20 Fedora Update System 2021-03-24 03:43:57 UTC
FEDORA-2021-956bbc2742 has been pushed to the Fedora 32 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-956bbc2742`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-956bbc2742

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 21 SASIKUMAR K 2021-03-25 06:04:31 UTC
I previously tested with the file "/etc/default/atop". However, after referring your recent correspondence I understood that we need to use the contents of upstream file "/etc/default/atop" and use it as "/etc/sysconfig/atop". And yes, it works, here are the results from my lab.

$ cat /etc/sysconfig/atop.rpmsave
# sysconfig atop
#

# Current Day format
CURDAY=`date +%Y%m%d`
# Log files path
LOGPATH=/var/log/atop
# Binaries path
BINPATH=/usr/bin
# PID File
PIDFILE=/var/run/atop.pid
# interval (default 10 minutes)
LOGINTERVAL=60

$ cat /etc/sysconfig/atop
LOGOPTS=""
LOGINTERVAL=60
LOGGENERATIONS=28
LOGPATH=/var/log/atop

[ec2-user@ip-172-31-36-207 ~]$ date; ls -l /var/log/atop
Thu Mar 25 05:56:53 UTC 2021
total 484
-rw-r--r--. 1 root root 213606 Mar 18 13:00 atop_20210318
-rw-r--r--. 1 root root 157803 Mar 23 13:00 atop_20210322
-rw-r--r--. 1 root root  54767 Mar 25 05:56 atop_20210325
-rw-r--r--. 1 root root  22519 Mar 25 05:48 atop_20210325.save

I have also tested the patched version 2.6.0-5 and it resolves the problem completely. 

https://kojipkgs.fedoraproject.org//packages/atop/2.6.0/5.el7/x86_64/atop-2.6.0-5.el7.x86_64.rpm

Thanks a lot Gwyn Ciesla for fixing it. Really appreciate you.

Comment 22 Gwyn Ciesla 2021-03-25 13:30:48 UTC
You're very welcome!

Comment 23 gerlof.langeveld 2021-03-27 14:39:53 UTC
I recently discovered that the real issue is in the atop.service file, where you can find:

Environment=LOGOPTS=""

The two double-quotes ("") cause an empty command line argument for atop, that reacts by taking an interval time of 0 seconds (which means: no time-driven intervals any more). This environment variable LOGOPTS is overruled by the /etc/default/atop file, so it also overrules the wrong specification in the atop.service file. LOGOPTS was not set in the original /etc/sysconfig/atop file, and therefor the wrong specification in the atop.service file was not overruled.

Please remove the two double-quotes in the atop.service file.

Comment 24 Fedora Update System 2021-03-28 00:15:39 UTC
FEDORA-2021-0f367a5b54 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 25 Gwyn Ciesla 2021-03-29 14:57:27 UTC
What we now ship are the files provided in 2.6.0, with LOGOPTS="" in both the defaults/sysconfig file and the unit file, and that works in my testing. Will there be a release to address this in the near future?

Comment 26 Fedora Update System 2021-04-01 01:50:28 UTC
FEDORA-2021-f238ed0ea7 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 27 Fedora Update System 2021-04-01 01:54:39 UTC
FEDORA-2021-956bbc2742 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 28 Fedora Update System 2021-04-07 15:48:22 UTC
FEDORA-EPEL-2021-631b5d8a32 has been pushed to the Fedora EPEL 7 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 29 Fedora Update System 2021-04-07 15:56:50 UTC
FEDORA-EPEL-2021-6b39871d3f has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.