Bug 1351749

Summary: [abrt] error occurs regularly on the Gluster console machine with check_cluster_vol_usage.py
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Cal Calhoun <ccalhoun>
Component: nagios-server-addonsAssignee: Ramesh N <rnachimu>
Status: CLOSED ERRATA QA Contact: Sweta Anandpara <sanandpa>
Severity: medium Docs Contact:
Priority: high    
Version: rhgs-3.1CC: amukherj, rcyriac, rhinduja, rnachimu, sabose
Target Milestone: ---   
Target Release: RHGS 3.2.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: nagios-server-addons-0.2.6-1 Doc Type: Bug Fix
Doc Text:
When Nagios retrieved live status information, this information was limited to a total of 8192 bytes. In clusters with large numbers of volumes, this meant that the data retrieved was not complete and therefore could not be parsed by Nagios. This resulted in a crash of the Nagios plugin. This update increases the allowed length for live status data in order to ensure that a complete result is retrieved and avoid the situation that led to the crash.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-23 06:29:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1351515    

Description Cal Calhoun 2016-06-30 17:55:10 UTC 
Description of problem:

A couple of times a day the customer gets an ABRT message regarding the nagios plugin check_cluster_vol_usage.py:

abrt_version:   2.0.8
cmdline:        /usr/bin/python /usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py -w 80 -c 90 -hg Default
event_log:     
executable:     /usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py
hostname:       host1.example.com
kernel:         2.6.32-573.22.1.el6.x86_64
last_occurrence: 1467059212
machineid:      sosreport_uploader-dmidecode=bd5f7ccc07cb4ade159083aac7a9611bc8d9588e5e7672b6908d8771ef49ddb3
pkg_arch:       noarch
pkg_epoch:      0
pkg_name:       nagios-server-addons
pkg_release:    1.el6rhs
pkg_version:    0.2.3
time:           Mon 20 Jun 2016 10:42:52 AM ACST
uid:            5076
username:       nagios

sosreport.tar.xz: Binary file, 1569292 bytes

backtrace:
:decoder.py:177:JSONObject:ValueError: end is out of bounds
:
:Traceback (most recent call last):
:  File "/usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py", line 93, in <module>
:    numVolumes, used, avail = checkVolumePerfData(args.hostgroup)
:  File "/usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py", line 49, in checkVolumePerfData
:    perf_data = json.loads(perf_data_output)
:  File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
:    return _default_decoder.decode(s)
:  File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
:    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
:  File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
:    obj, end = self._scanner.iterscan(s, **kw).next()
:  File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
:    rval, next_pos = action(m, context)
:  File "/usr/lib64/python2.6/json/decoder.py", line 217, in JSONArray
:    value, end = iterscan(s, idx=end, context=context).next()
:  File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
:    rval, next_pos = action(m, context)
:  File "/usr/lib64/python2.6/json/decoder.py", line 217, in JSONArray
:    value, end = iterscan(s, idx=end, context=context).next()
:  File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
:    rval, next_pos = action(m, context)
:  File "/usr/lib64/python2.6/json/decoder.py", line 177, in JSONObject
:    key, end = scanstring(s, end, encoding, strict)
:ValueError: end is out of bounds
:
:Local variables in innermost frame:
:iterscan: <bound method Scanner.iterscan of <json.scanner.Scanner object at 0x92e1d0>>
:pairs: {u'VOL_NAME': u'vol42', u'_GENERATED_BY_AUTOCONFIG': u'1'}
:end: 8192
:context: <json.decoder.JSONDecoder object at 0x92e7d0>
:encoding: None
:_w: <built-in method match of _sre.SRE_Pattern object at 0x9300b8>
:value: u'1'
:strict: True
:s: '[["Volume Utilization - vol1","Default","utilization=0.62%;70;90 total=32210167808.00 used=200765516.00 free=32009402292.00",{"VOL_NAME":"vol1","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol2","Default","utilization=0.00%;70;90 total=104857600.00 used=0.00 free=104857600.00",{"VOL_NAME":"vol2","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol3","Default","utilization=2.70%;70;90 total=104857600.00 used=2832924.00 free=102024676.00",{"VOL_NAME":"vol3","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol4","Default","utilization=0.00%;70;90 total=104857600.00 used=240.00 free=104857360.00",{"VOL_NAME":"vol4","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol5","Default","utilization=1.32%;70;90 total=104857600.00 used=1386236.00 free=103471364.00",{"VOL_NAME":"vol5","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol6","Default","utilization=0.08%;70;90 total=52395008.00 used=42068.00 free=52352940.00",{"VOL_NAME":"vol6","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol7","Default","utilization=0.06%;70;90 total=52395008.00 used=33604.00 free=52361404.00",{"VOL_NAME":"vol7","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol8","Default","utilization=0.00%;70;90 total=52428800.00 used=0.00 free=52428800.00",{"VOL_NAME":"vol8","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol9","Default","utilization=0.00%;70;90 total=1048576.00 used=0.00 free=1048576.00",{"VOL_NAME":"vol9","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol10","Default","",{"VOL_NAME":"vol10","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol11","Default","utilization=0.00%;70;90 total=1073741824.00 used=0.00 free=1073741824.00",{"VOL_NAME":"vol11","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol12","Default","",{"VOL_NAME":"vol12","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol13","Default","utilization=56.71%;70;90 total=10735331328.00 used=6087551236.00 free=4647780092.00",{"VOL_NAME":"vol13","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol14","Default","utilization=0.00%;70;90 total=524288000.00 used=16.00 free=524287984.00",{"VOL_NAME":"vol14","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol15","Default","",{"VOL_NAME":"vol15","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol16","Default","",{"VOL_NAME":"vol16","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol17","Default","utilization=1.10%;70;90 total=52428800.00 used=574384.00 free=51854416.00",{"VOL_NAME":"vol17","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol18","Default","utilization=0.00%;70;90 total=52428800.00 used=0.00 free=52428800.00",{"VOL_NAME":"vol18","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol19","Default","utilization=0.62%;70;90 total=32210167808.00 used=199485444.00 free=32010682364.00",{"VOL_NAME":"vol19","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol20","Default","",{"VOL_NAME":"vol20","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol21","Default","utilization=3.51%;70;90 total=15728640.00 used=551960.00 free=15176680.00",{"VOL_NAME":"vol21","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol22","Default","utilization=62.60%;70;90 total=209715200.00 used=131289864.00 free=78425336.00",{"VOL_NAME":"vol22","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol23","Default","utilization=0.32%;70;90 total=10467328.00 used=33612.00 free=10433716.00",{"VOL_NAME":"vol23","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol24","Default","utilization=0.32%;70;90 total=10467328.00 used=33988.00 free=10433340.00",{"VOL_NAME":"vol24","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol25","Default","",{"VOL_NAME":"vol25","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol26","Default","utilization=0.00%;70;90 total=10485760.00 used=0.00 free=10485760.00",{"VOL_NAME":"vol26","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol27","Default","utilization=27.89%;70;90 total=104857600.00 used=29247488.00 free=75610112.00",{"VOL_NAME":"vol27","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol28","Default","utilization=39.96%;70;90 total=6979321856.00 used=2788936528.00 free=4190385328.00",{"VOL_NAME":"vol28","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol29","Default","utilization=0.00%;70;90 total=104857600.00 used=0.00 free=104857600.00",{"VOL_NAME":"vol29","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol30","Default","utilization=0.00%;70;90 total=10485760.00 used=0.00 free=10485760.00",{"VOL_NAME":"vol30","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol31","Default","",{"VOL_NAME":"vol31","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol32","Default","",{"VOL_NAME":"vol32","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol33","Default","utilization=0.64%;70;90 total=32210167808.00 used=204863360.00 free=32005304448.00",{"VOL_NAME":"vol33","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol34","Default","utilization=0.64%;70;90 total=32210167808.00 used=204753548.00 free=32005414260.00",{"VOL_NAME":"vol34","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol35","Default","utilization=1.61%;70;90 total=2082816.00 used=33504.00 free=2049312.00",{"VOL_NAME":"vol35","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol36","Default","utilization=0.36%;70;90 total=20953088.00 used=75048.00 free=20878040.00",{"VOL_NAME":"vol36","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol37","Default","utilization=1.15%;70;90 total=10485760.00 used=120284.00 free=10365476.00",{"VOL_NAME":"vol37","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol38","Default","utilization=0.00%;70;90 total=52428800.00 used=0.00 free=52428800.00",{"VOL_NAME":"vol38","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol39","Default","utilization=40.59%;70;90 total=52428800.00 used=21280780.00 free=31148020.00",{"VOL_NAME":"vol39","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol40","Default","",{"VOL_NAME":"vol40","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol41","Default","utilization=5.37%;70;90 total=255232.00 used=13696.00 free=241536.00",{"VOL_NAME":"vol41","_GENERATED_BY_AUTOCONFIG":"1","GLUSTER_ENTITY":"Service"}],\n["Volume Utilization - vol42","Default","",{"VOL_NAME":"call_manager","_GENERATED_BY_AUTOCONFIG":"1","'
:key: u'_GENERATED_BY_AUTOCONFIG'
:nextchar: '"'
:match: <_sre.SRE_Match object at 0xa30a40>

Version-Release number of selected component (if applicable):

gluster-nagios-common-0.2.3-1.el6rhs

How reproducible:

Happens periodically

Steps to Reproduce:

See description
Comment 8 Sweta Anandpara 2016-11-21 11:54:48 UTC 
Steps to reproduce
===================

1. Have a cluster with Nagios set up and create 50 volumes of very long names. ( I created 50 volumes of the name 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaavol$i' where $i=(1..50))
2. Run the command '/usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py' and check the output. When all is well, it prints 'OK' else shows a traceback


Reproduced the issue in nagios-server-addons-0.2.5-1.el7rhgs.x86_64. Hit the traceback, and was able to see the ABRT message in /var/log/messages. 

Updated the package to nagios-server-addons-0.2.6-1.el7rhgs.x86_64 in the same setup. Did not see any traceback. /var/log/messages showed a clean log.
Moving this BZ to fixed in 3.2. Detailed logs are pasted below.


nagios-server-addons-0.2.5-1
==============================

/var/log/messages/ snippet

13793 Nov 21 17:13:30 dhcp46-239 python: detected unhandled Python exception in '/usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py'
13794 Nov 21 17:13:35 dhcp46-239 python: communication with ABRT daemon failed: timed out
13795 Nov 21 17:14:01 dhcp46-239 systemd: Started Session 13397 of user root.
13796 Nov 21 17:14:01 dhcp46-239 systemd: Starting Session 13397 of user root.

[root@dhcp46-239 ~]# /usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py -hg rhel7_nagios
Traceback (most recent call last):
  File "/usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py", line 93, in <module>
    numVolumes, used, avail = checkVolumePerfData(args.hostgroup)
  File "/usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py", line 49, in checkVolumePerfData
    perf_data = json.loads(perf_data_output)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 36 column 2 (char 8145)
[root@dhcp46-239 ~]# 

nagios-server-addons-0.2.6-1
==============================

[root@dhcp46-239 ~]# /usr/lib64/nagios/plugins/gluster/check_cluster_vol_usage.py -hg rhel7_nagios
OK - used 2% of available 1385.17285156 GB|used=2;70;95;0;100;
[root@dhcp46-239 ~]# 
[root@dhcp46-239 ~]#
Comment 10 errata-xmlrpc 2017-03-23 06:29:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0492.html