Bug 1312576 - gluster nagios addons package installed a sysstat crontab that conflicts with existing sysstat crontab [NEEDINFO]
gluster nagios addons package installed a sysstat crontab that conflicts with...
Status: NEW
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: gluster-nagios-addons (Show other bugs)
3.0
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Ramesh N
RHS-C QE
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-27 09:06 EST by Peter Portante
Modified: 2017-03-25 12:26 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
sabose: needinfo? (sanandpa)
ccalhoun: needinfo?


Attachments (Terms of Use)

  None (edit)
Description Peter Portante 2016-02-27 09:06:41 EST
The provided gluster-sysstat.crontab file ends up with conflicting metrics collection provided by the /etc/cron.d/sysstat crontab.

gluster-nagios-addons-0.1.16-1.el6rhs.x86_64 : Gluster node management add-ons for Nagios
Repo        : installed
Matched from:
Other       : Provides-match: /etc/cron.d/gluster-sysstat.crontab

It seems this file should be shipped as an example, and should probably not be run by default.

Too often I see emails from root like the following:

From: root@foo.example.com (Cron Daemon)
To: root@foo.example.com
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Date: Sun, 21 Feb 2016 00:00:01 +0000 (UTC)
Subject: Cron <root@foo> /usr/lib64/sa/sa1 1 1

flock: Resource temporarily unavailable
Comment 2 Sahina Bose 2016-02-29 01:12:09 EST
Tim, can u take a look?
Comment 3 Ramesh N 2016-03-04 01:47:42 EST
We have this file to enable nagios monitoring for gluster. We can't make it as a sample since our monitoring solution requires data collection in every minute. We have see similar issues in the past as well where this command fails exactly at 00:00:01 +0000 (UTC). Here also the same case, it failed at "Date: Sun, 21 Feb 2016 00:00:01 +0000 (UTC)". 

Darshan can u investigate further?
Comment 4 Darshan 2016-03-04 02:31:51 EST
(In reply to Ramesh N from comment #3)
> We have this file to enable nagios monitoring for gluster. We can't make it
> as a sample since our monitoring solution requires data collection in every
> minute. We have see similar issues in the past as well where this command
> fails exactly at 00:00:01 +0000 (UTC). Here also the same case, it failed at
> "Date: Sun, 21 Feb 2016 00:00:01 +0000 (UTC)". 
> 
> Darshan can u investigate further?

Ramesh,

As you mentioned, we had the problem of an entry not being made into sysstat's saDD file during day's transition (at 00:00:00 hrs). We fixed it by adding another entry in same cron job for making an entry to the file at last minute of the day. with this that issue was solved.

But the issue here seems to be resource un availability. the cron job /etc/cron.d/sysstat tries to collect and update the sysstat data every 10 minutes once and our cron job does the same thing every minute once. and while updating the sysstat data file(var/log/sa/saDD) a lock on the file is held by entity updating it. If multiple jobs try to update simultaneously only one will succeed.

We should check if this issue arises if we have only one cron job running to update sysstat data.
Comment 5 Sahina Bose 2016-03-16 02:12:59 EDT
Can you check if this exists with gluster-systat crontab alone?
Comment 6 Cal Calhoun 2016-05-27 13:30:32 EDT
Is there a supported work-around for this issue?

Note You need to log in before you can comment on or make changes to this bug.