Bug 1312576

Summary: gluster nagios addons package installed a sysstat crontab that conflicts with existing sysstat crontab
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Peter Portante <pportant>
Component: gluster-nagios-addonsAssignee: Sahina Bose <sabose>
Status: CLOSED CANTFIX QA Contact: Sweta Anandpara <sanandpa>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.0CC: ccalhoun, dnarayan, frederick.pop, pandrade, perfbz, sabose, sanandpa, tjeyasin
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-30 11:12:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Peter Portante 2016-02-27 14:06:41 UTC
The provided gluster-sysstat.crontab file ends up with conflicting metrics collection provided by the /etc/cron.d/sysstat crontab.

gluster-nagios-addons-0.1.16-1.el6rhs.x86_64 : Gluster node management add-ons for Nagios
Repo        : installed
Matched from:
Other       : Provides-match: /etc/cron.d/gluster-sysstat.crontab

It seems this file should be shipped as an example, and should probably not be run by default.

Too often I see emails from root like the following:

From: root.com (Cron Daemon)
To: root.com
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Date: Sun, 21 Feb 2016 00:00:01 +0000 (UTC)
Subject: Cron <root@foo> /usr/lib64/sa/sa1 1 1

flock: Resource temporarily unavailable

Comment 2 Sahina Bose 2016-02-29 06:12:09 UTC
Tim, can u take a look?

Comment 3 Ramesh N 2016-03-04 06:47:42 UTC
We have this file to enable nagios monitoring for gluster. We can't make it as a sample since our monitoring solution requires data collection in every minute. We have see similar issues in the past as well where this command fails exactly at 00:00:01 +0000 (UTC). Here also the same case, it failed at "Date: Sun, 21 Feb 2016 00:00:01 +0000 (UTC)". 

Darshan can u investigate further?

Comment 4 Darshan 2016-03-04 07:31:51 UTC
(In reply to Ramesh N from comment #3)
> We have this file to enable nagios monitoring for gluster. We can't make it
> as a sample since our monitoring solution requires data collection in every
> minute. We have see similar issues in the past as well where this command
> fails exactly at 00:00:01 +0000 (UTC). Here also the same case, it failed at
> "Date: Sun, 21 Feb 2016 00:00:01 +0000 (UTC)". 
> 
> Darshan can u investigate further?

Ramesh,

As you mentioned, we had the problem of an entry not being made into sysstat's saDD file during day's transition (at 00:00:00 hrs). We fixed it by adding another entry in same cron job for making an entry to the file at last minute of the day. with this that issue was solved.

But the issue here seems to be resource un availability. the cron job /etc/cron.d/sysstat tries to collect and update the sysstat data every 10 minutes once and our cron job does the same thing every minute once. and while updating the sysstat data file(var/log/sa/saDD) a lock on the file is held by entity updating it. If multiple jobs try to update simultaneously only one will succeed.

We should check if this issue arises if we have only one cron job running to update sysstat data.

Comment 5 Sahina Bose 2016-03-16 06:12:59 UTC
Can you check if this exists with gluster-systat crontab alone?

Comment 6 Cal Calhoun 2016-05-27 17:30:32 UTC
Is there a supported work-around for this issue?

Comment 7 Frédérick Pop 2017-12-04 09:17:52 UTC
Hello, is there any update on this issue ?

Comment 9 Sahina Bose 2018-01-30 11:12:57 UTC
Thank you for your report. However, this bug is being closed as it's logged against gluster-nagios monitoring for which no further new development is being undertaken.

Comment 10 Paulo Andrade 2020-01-24 18:43:51 UTC
What is the suggested workaround for parallel executions of /usr/lib64/sa1 from cron?

# egrep -r '\<sa\>' /etc/cron.*
/etc/cron.d/gluster-sysstat.crontab:59 23 * * * root /usr/lib64/sa/sa1 60 2
/etc/cron.d/gluster-sysstat.crontab:*/1 * * * * root /usr/lib64/sa/sa1 1 1
/etc/cron.d/sysstat:*/10 * * * * root /usr/lib64/sa/sa1 1 1
/etc/cron.d/sysstat:# 0 * * * * root /usr/lib64/sa/sa1 600 6 &
/etc/cron.d/sysstat:53 23 * * * root /usr/lib64/sa/sa2 -A