Bug 1565852 - XMLRPC Fault in watchdog.log if OpenStack instance has already gone away
Summary: XMLRPC Fault in watchdog.log if OpenStack instance has already gone away
Keywords:
Status: CLOSED EOL
Alias: None
Product: Beaker
Classification: Community
Component: openstack
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: ---
Assignee: beaker-dev-list
QA Contact: tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-10 23:12 UTC by Roman Joost
Modified: 2020-04-07 15:56 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-07 15:39:36 UTC


Attachments (Terms of Use)

Description Roman Joost 2018-04-10 23:12:34 UTC
Description of problem:

The watchdog.log on the lab controllers have quite a lot of XMLRPC Faults in them like:

Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: bkr.labcontroller.proxy ERROR Failed to run monitor for host-192-168-11-14.openstacklocal
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:  Traceback (most recent call last):
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 551, in run
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      updated |= monitor.run()
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 615, in run
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self.console_watch.update()
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 299, in update
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      output = self.proxy.get_console_log(self.watchdog['recipe_id'])
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 191, in get_console_log
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self.hub.recipes.console_output(recipe_id, length)
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self.__send(self.__name, args)
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      verbose=self.__verbose
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 571, in request
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      result = transport_class.request(self, *args, **kwargs)
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 427, in _request
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self.parse_response(h.getfile())
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1361, in parse_response
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return self._parse_response(file, None)
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      return u.close()
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:      raise Fault(**self._stack[0])
Apr  9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]:  Fault: <Fault 1: "<class 'novaclient.exceptions.NotFound'>:Instance d13d9429-b31d-4ee1-b272-3842c61c4e99 could not be found. (HTTP 404) (Request-ID: req-35e84250-bf17-4f83-ae38-6fc800c43d8e)">

I don't think this is a regression or particularly bad, but perhaps just a timing problem.

So this bug is more about tracking this and perhaps see if we can avoid raising the error in the first place, since it is misleading.

Version-Release number of selected component (if applicable):
25.1

How reproducible:
not sure

Steps to Reproduce:
1. Provision recipes in openstack
2. tail the watchdog.log
3. see the error appear

Actual results:
XMLRPC Faults

Expected results:
No fault

Additional info:


Note You need to log in before you can comment on or make changes to this bug.