Bug 1341130
| Summary: | there are lot of errors in salt master log | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Storage Console | Reporter: | Martin Bukatovic <mbukatov> |
| Component: | core | Assignee: | Darshan <dnarayan> |
| core sub component: | provisioning | QA Contact: | sds-qe-bugs |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | dnarayan, mkudlej, nthomas |
| Version: | 2 | ||
| Target Milestone: | --- | ||
| Target Release: | 2 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | rhscon-core-0.0.28-1.el7scon.x86_64.rpm | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-08-23 19:52:54 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1344195 | ||
There are no exception anymore in salt master log. ceph-ansible-1.0.5-27.el7scon.noarch ceph-installer-1.0.14-1.el7scon.noarch rhscon-ceph-0.0.33-1.el7scon.x86_64 rhscon-core-0.0.34-1.el7scon.x86_64 rhscon-core-selinux-0.0.34-1.el7scon.noarch rhscon-ui-0.0.48-1.el7scon.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2016:1754 |
Description of problem ====================== There are lot of error messages related to tracebacks in salt master log on RHSC 2.0 machine. Version-Release =============== On RHSC 2.0 machine: rhscon-core-0.0.19-1.el7scon.x86_64 rhscon-ceph-0.0.18-1.el7scon.x86_64 rhscon-ui-0.0.34-1.el7scon.noarch ceph-installer-1.0.11-1.el7scon.noarch ceph-ansible-1.0.5-15.el7scon.noarch salt-2015.5.5-1.el7.noarch salt-master-2015.5.5-1.el7.noarch On Storage machine: rhscon-agent-0.0.8-1.el7scon.noarch ceph-base-10.2.1-6.el7cp.x86_64 salt-minion-2015.5.5-1.el7.noarch salt-2015.5.5-1.el7.noarch How reproducible ================ 100 % Steps to Reproduce ================== 1. Install RHSC 2.0 following the documentation, make sure you have few nodes ready to be accepted later. 2. Accept all nodes 3. Check /var/log/salt/master log file on RHSC 2.0 machine Actual results ============== You can find this kind of error in the log: ~~~ 2016-05-29 15:19:05,543 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/salt/utils/reactor.py", line 52, in render_reaction data=data) File "/usr/lib/python2.7/site-packages/salt/state.py", line 257, in render_template template, self.rend, self.opts['renderer'], **kwargs) File "/usr/lib/python2.7/site-packages/salt/template.py", line 79, in compile_template ret = render(input_data, saltenv, sls, **render_kwargs) File "/usr/lib/python2.7/site-packages/salt/renderers/py.py", line 112, in render 'Unknown render error in py renderer')) SaltRenderError: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 511, in py data = mod.run() File "/srv/salt/push_event.sls", line 201, in run if len(t) == 2: TypeError: object of type 'NoneType' has no len() ~~~ Sometimes there is a similar error just after this one: ~~~ 2016-05-31 12:26:05,651 [salt.utils.reactor][ERROR ][4017] Failed to render "/srv/salt/push_event.sls": Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/salt/utils/reactor.py", line 52, in render_reaction data=data) File "/usr/lib/python2.7/site-packages/salt/state.py", line 257, in render_template template, self.rend, self.opts['renderer'], **kwargs) File "/usr/lib/python2.7/site-packages/salt/template.py", line 79, in compile_template ret = render(input_data, saltenv, sls, **render_kwargs) File "/usr/lib/python2.7/site-packages/salt/renderers/py.py", line 112, in render 'Unknown render error in py renderer')) SaltRenderError: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 511, in py data = mod.run() File "/srv/salt/push_event.sls", line 201, in run if len(t) == 2: TypeError: object of type 'NoneType' has no len() 2016-05-31 12:26:05,716 [salt.utils.reactor][ERROR ][4017] Failed to render "/srv/salt/push_event.sls": Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/salt/utils/reactor.py", line 52, in render_reaction data=data) File "/usr/lib/python2.7/site-packages/salt/state.py", line 257, in render_template template, self.rend, self.opts['renderer'], **kwargs) File "/usr/lib/python2.7/site-packages/salt/template.py", line 79, in compile_template ret = render(input_data, saltenv, sls, **render_kwargs) File "/usr/lib/python2.7/site-packages/salt/renderers/py.py", line 112, in render 'Unknown render error in py renderer')) SaltRenderError: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 511, in py data = mod.run() File "/srv/salt/push_event.sls", line 205, in run if fnmatch.fnmatch(t, 'skyring/*'): File "/usr/lib64/python2.7/fnmatch.py", line 43, in fnmatch return fnmatchcase(name, pat) File "/usr/lib64/python2.7/fnmatch.py", line 79, in fnmatchcase return _cache[pat].match(name) is not None TypeError: expected string or buffer ~~~ And there are lot of them: ~~~ # cd /var/log/salt # zcat master-20160529.gz | grep "TypeError: object of type 'NoneType' has no len()" | wc -l 25033 # grep "TypeError: object of type 'NoneType' has no len()" master | wc -l 256 ~~~ Moreover it seems to be connected to some action which is performed every 10 minutes: ~~~ # grep 'salt.utils.reactor\]\[ERROR' master | head -20 2016-05-29 03:37:23,311 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:37:23,917 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:37:24,295 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:37:28,334 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:47:20,045 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:47:23,439 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:47:28,578 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:47:28,793 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:57:21,639 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:57:22,068 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 03:57:25,864 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 04:07:24,380 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 04:07:24,759 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 04:07:24,883 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 04:17:29,018 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 04:17:29,354 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 04:17:29,463 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 04:27:30,932 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 04:27:34,300 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": 2016-05-29 04:27:34,414 [salt.utils.reactor][ERROR ][19522] Failed to render "/srv/salt/push_event.sls": ~~~ Expected results ================ There is no such error in salt master logs.