Bug 1341130 - there are lot of errors in salt master log
Summary: there are lot of errors in salt master log
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Storage Console
Classification: Red Hat
Component: core
Version: 2
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 2
Assignee: Darshan
QA Contact: sds-qe-bugs
URL:
Whiteboard:
Depends On:
Blocks: Console-2-DevFreeze
TreeView+ depends on / blocked
 
Reported: 2016-05-31 10:35 UTC by Martin Bukatovic
Modified: 2016-08-23 19:52 UTC (History)
3 users (show)

Fixed In Version: rhscon-core-0.0.28-1.el7scon.x86_64.rpm
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-23 19:52:54 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Gerrithub.io 281412 None None None 2016-06-23 05:40:09 UTC
Red Hat Bugzilla 1319850 None None None Never
Red Hat Product Errata RHEA-2016:1754 normal SHIPPED_LIVE New packages: Red Hat Storage Console 2.0 2017-04-18 19:09:06 UTC

Internal Links: 1319850

Description Martin Bukatovic 2016-05-31 10:35:16 UTC
Description of problem
======================

There are lot of error messages related to tracebacks in salt master log on
RHSC 2.0 machine.

Version-Release
===============

On RHSC 2.0 machine:

rhscon-core-0.0.19-1.el7scon.x86_64
rhscon-ceph-0.0.18-1.el7scon.x86_64
rhscon-ui-0.0.34-1.el7scon.noarch
ceph-installer-1.0.11-1.el7scon.noarch
ceph-ansible-1.0.5-15.el7scon.noarch
salt-2015.5.5-1.el7.noarch
salt-master-2015.5.5-1.el7.noarch

On Storage machine:

rhscon-agent-0.0.8-1.el7scon.noarch
ceph-base-10.2.1-6.el7cp.x86_64
salt-minion-2015.5.5-1.el7.noarch
salt-2015.5.5-1.el7.noarch

How reproducible
================

100 %

Steps to Reproduce
==================

1. Install RHSC 2.0 following the documentation, make sure you have few nodes
   ready to be accepted later.
2. Accept all nodes
3. Check /var/log/salt/master log file on RHSC 2.0 machine

Actual results
==============

You can find this kind of error in the log:

~~~
2016-05-29 15:19:05,543 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls":
Traceback (most recent call last):                                              
  File "/usr/lib/python2.7/site-packages/salt/utils/reactor.py", line 52, in render_reaction
    data=data)                                                                  
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 257, in render_template
    template, self.rend, self.opts['renderer'], **kwargs)                       
  File "/usr/lib/python2.7/site-packages/salt/template.py", line 79, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)                     
  File "/usr/lib/python2.7/site-packages/salt/renderers/py.py", line 112, in render
    'Unknown render error in py renderer'))                                     
SaltRenderError: Traceback (most recent call last):                             
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 511, in py
    data = mod.run()                                                            
  File "/srv/salt/push_event.sls", line 201, in run                             
    if len(t) == 2:                                                             
TypeError: object of type 'NoneType' has no len()
~~~

Sometimes there is a similar error just after this one:

~~~
2016-05-31 12:26:05,651 [salt.utils.reactor][ERROR   ][4017] Failed to render "/srv/salt/push_event.sls": 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/reactor.py", line 52, in render_reaction
    data=data)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 257, in render_template
    template, self.rend, self.opts['renderer'], **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/template.py", line 79, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python2.7/site-packages/salt/renderers/py.py", line 112, in render
    'Unknown render error in py renderer'))
SaltRenderError: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 511, in py
    data = mod.run()
  File "/srv/salt/push_event.sls", line 201, in run
    if len(t) == 2:
TypeError: object of type 'NoneType' has no len()

2016-05-31 12:26:05,716 [salt.utils.reactor][ERROR   ][4017] Failed to render "/srv/salt/push_event.sls": 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/reactor.py", line 52, in render_reaction
    data=data)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 257, in render_template
    template, self.rend, self.opts['renderer'], **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/template.py", line 79, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python2.7/site-packages/salt/renderers/py.py", line 112, in render
    'Unknown render error in py renderer'))
SaltRenderError: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 511, in py
    data = mod.run()
  File "/srv/salt/push_event.sls", line 205, in run
    if fnmatch.fnmatch(t, 'skyring/*'):
  File "/usr/lib64/python2.7/fnmatch.py", line 43, in fnmatch
    return fnmatchcase(name, pat)
  File "/usr/lib64/python2.7/fnmatch.py", line 79, in fnmatchcase
    return _cache[pat].match(name) is not None
TypeError: expected string or buffer

~~~

And there are lot of them:

~~~
# cd /var/log/salt
# zcat master-20160529.gz |  grep "TypeError: object of type 'NoneType' has no len()" | wc -l 
25033
#  grep "TypeError: object of type 'NoneType' has no len()" master | wc -l
256
~~~

Moreover it seems to be connected to some action which is performed every 10
minutes:

~~~
# grep 'salt.utils.reactor\]\[ERROR' master | head -20
2016-05-29 03:37:23,311 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:37:23,917 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:37:24,295 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:37:28,334 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:47:20,045 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:47:23,439 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:47:28,578 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:47:28,793 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:57:21,639 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:57:22,068 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 03:57:25,864 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 04:07:24,380 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 04:07:24,759 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 04:07:24,883 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 04:17:29,018 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 04:17:29,354 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 04:17:29,463 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 04:27:30,932 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 04:27:34,300 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls": 
2016-05-29 04:27:34,414 [salt.utils.reactor][ERROR   ][19522] Failed to render "/srv/salt/push_event.sls":
~~~

Expected results
================

There is no such error in salt master logs.

Comment 1 Martin Kudlej 2016-07-22 13:35:48 UTC
There are no exception anymore in salt master log.
ceph-ansible-1.0.5-27.el7scon.noarch
ceph-installer-1.0.14-1.el7scon.noarch
rhscon-ceph-0.0.33-1.el7scon.x86_64
rhscon-core-0.0.34-1.el7scon.x86_64
rhscon-core-selinux-0.0.34-1.el7scon.noarch
rhscon-ui-0.0.48-1.el7scon.noarch

Comment 3 errata-xmlrpc 2016-08-23 19:52:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2016:1754


Note You need to log in before you can comment on or make changes to this bug.