Bug 1756255 - health.yml shows python backtrace because of missing key in resources map
Summary: health.yml shows python backtrace because of missing key in resources map
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 3.11.z
Assignee: Russell Teague
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-27 08:31 UTC by German Parente
Modified: 2020-01-16 10:02 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-12-16 11:57:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-ansible pull 12025 0 'None' closed Bug 1756255: Fix openshift_health_checker sdn.py key error 2020-12-05 02:58:58 UTC
Red Hat Product Errata RHBA-2019:4050 0 None None None 2019-12-16 11:57:26 UTC

Description German Parente 2019-09-27 08:31:03 UTC
Description of problem:

openshift-checks/health.yml

could show a python stacktrace with a missing key in resources map when removing some checks in this way:

ansible-playbook -vvv -i ~/hosts /usr/share/ansible/openshift-ansible/playbooks/openshift-checks/health.yml -e openshift_disable_check=curator,logging_index_time,elasticsearch,kibana,diagnostics,etcd_volume,fluentd

we can see that when checking sdn, the resources map could not have the key [module_results] defined. In sdn.py:

/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py
===================================
    def get_resource(self, kind):
        """Return a list of all resources of the specified kind."""
        for resource in self.task_vars['resources']['module_results']:
            if resource['item'] == kind:
                return resource['module_results']['results'][0]['items']
===================================

Output:

Details:  check "sdn":
               'module_results'
               Traceback (most recent call last):
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 225, in run_check
                   result = check.run()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 76, in run
                   self.check_master()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 167, in check_master
                   nodes = self.get_resource('nodes')
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 221, in get_resource
                   for resource in self.task_vars['resources']['module_results']:
               KeyError: 'module_results'



Version-Release number of the following components:
rpm -q openshift-ansible
openshift-ansible-3.11.141-1.git.0.a7e91cd.el7.noarch

rpm -q ansible
ansible-2.6.18-1.el7ae.noarch

ansible --version
ansible 2.6.18
  config file = /home/quicklab/ansible.cfg
  configured module search path = [u'/home/quicklab/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]



How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

================================================================

1. Hosts:    h1, h2, h3 
     Play:     OpenShift Health Checks
     Task:     Run health checks (@health)
     Message:  One or more checks failed
     Details:  check "sdn":
               'module_results'
               Traceback (most recent call last):
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 225, in run_check
                   result = check.run()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 76, in run
                   self.check_master()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 167, in check_master
                   nodes = self.get_resource('nodes')
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 221, in get_resource
                   for resource in self.task_vars['resources']['module_results']:
               KeyError: 'module_results'


  2. Hosts:    h4, h5, h6, h7 
     Play:     OpenShift Health Checks
     Task:     Run health checks (@health)
     Message:  One or more checks failed
     Details:  check "sdn":
               'module_results'
               Traceback (most recent call last):
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 225, in run_check
                   result = check.run()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 78, in run
                   self.check_node()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 279, in check_node
                   for hostsubnet in self.get_resource('hostsubnets')}
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 221, in get_resource
                   for resource in self.task_vars['resources']['module_results']:
               KeyError: 'module_results'
===================================================================


Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 2 Johnny Liu 2019-12-02 03:07:04 UTC
Reproduced this bug with openshift-ansible-3.11.154-2.git.0.1640c49.el7.noarch.
Failure summary:


  1. Hosts:    ci-vm-10-0-151-71.hosted.upshift.rdu2.redhat.com
     Play:     OpenShift Health Checks
     Task:     Run health checks (@health)
     Message:  One or more checks failed
     Details:  check "sdn":
               'module_results'
               Traceback (most recent call last):
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 225, in run_check
                   result = check.run()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 76, in run
                   self.check_master()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 167, in check_master
                   nodes = self.get_resource('nodes')
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 221, in get_resource
                   for resource in self.task_vars['resources']['module_results']:
               KeyError: 'module_results'
               

  2. Hosts:    ci-vm-10-0-150-39.hosted.upshift.rdu2.redhat.com, ci-vm-10-0-151-46.hosted.upshift.rdu2.redhat.com
     Play:     OpenShift Health Checks
     Task:     Run health checks (@health)
     Message:  One or more checks failed
     Details:  check "sdn":
               'module_results'
               Traceback (most recent call last):
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 225, in run_check
                   result = check.run()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 78, in run
                   self.check_node()
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 279, in check_node
                   for hostsubnet in self.get_resource('hostsubnets')}
                 File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/sdn.py", line 221, in get_resource
                   for resource in self.task_vars['resources']['module_results']:
               KeyError: 'module_results'


Verified this bug with, and PASS.

Comment 3 Johnny Liu 2019-12-02 03:08:22 UTC
Verified this bug with openshift-ansible-3.11.156-1.git.0.5f2cfa4.el7.noarch, and PASS.

Comment 5 errata-xmlrpc 2019-12-16 11:57:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4050


Note You need to log in before you can comment on or make changes to this bug.