Bug 1565852

Summary: XMLRPC Fault in watchdog.log if OpenStack instance has already gone away
Product: [Retired] Beaker Reporter: Roman Joost <rjoost>
Component: openstackAssignee: beaker-dev-list
Status: CLOSED EOL QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 25CC: mastyk
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-07 15:39:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Roman Joost 2018-04-10 23:12:34 UTC
Description of problem:

The watchdog.log on the lab controllers have quite a lot of XMLRPC Faults in them like:

Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: bkr.labcontroller.proxy ERROR Failed to run monitor for host-192-168-11-14.openstacklocal
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:  Traceback (most recent call last):
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 551, in run
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      updated |= monitor.run()
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 615, in run
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self.console_watch.update()
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 299, in update
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      output = self.proxy.get_console_log(self.watchdog['recipe_id'])
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 191, in get_console_log
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self.hub.recipes.console_output(recipe_id, length)
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self.__send(self.__name, args)
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      verbose=self.__verbose
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 571, in request
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      result = transport_class.request(self, *args, **kwargs)
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 427, in _request
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self.parse_response(h.getfile())
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1361, in parse_response
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self._parse_response(file, None)
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return u.close()
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      raise Fault(**self._stack[0])
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:  Fault: <Fault 1: "<class 'novaclient.exceptions.NotFound'>:Instance d13d9429-b31d-4ee1-b272-3842c61c4e99 could not be found. (HTTP 404) (Request-ID: req-35e84250-bf17-4f83-ae38-6fc800c43d8e)">

I don't think this is a regression or particularly bad, but perhaps just a timing problem.

So this bug is more about tracking this and perhaps see if we can avoid raising the error in the first place, since it is misleading.

Version-Release number of selected component (if applicable):
25.1

How reproducible:
not sure

Steps to Reproduce:
1. Provision recipes in openstack
2. tail the watchdog.log
3. see the error appear

Actual results:
XMLRPC Faults

Expected results:
No fault

Additional info: