Description of problem: The watchdog.log on the lab controllers have quite a lot of XMLRPC Faults in them like: Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: bkr.labcontroller.proxy ERROR Failed to run monitor for host-192-168-11-14.openstacklocal Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: Traceback (most recent call last): Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 551, in run Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: updated |= monitor.run() Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 615, in run Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: return self.console_watch.update() Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 299, in update Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: output = self.proxy.get_console_log(self.watchdog['recipe_id']) Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 191, in get_console_log Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: return self.hub.recipes.console_output(recipe_id, length) Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: return self.__send(self.__name, args) Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: verbose=self.__verbose Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 571, in request Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: result = transport_class.request(self, *args, **kwargs) Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 427, in _request Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: return self.parse_response(h.getfile()) Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib64/python2.6/xmlrpclib.py", line 1361, in parse_response Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: return self._parse_response(file, None) Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: return u.close() Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: raise Fault(**self._stack[0]) Apr 9 07:20:13 lab-devel-02.rhts.eng.bos.redhat.com beaker-watchdog[2528]: Fault: <Fault 1: "<class 'novaclient.exceptions.NotFound'>:Instance d13d9429-b31d-4ee1-b272-3842c61c4e99 could not be found. (HTTP 404) (Request-ID: req-35e84250-bf17-4f83-ae38-6fc800c43d8e)"> I don't think this is a regression or particularly bad, but perhaps just a timing problem. So this bug is more about tracking this and perhaps see if we can avoid raising the error in the first place, since it is misleading. Version-Release number of selected component (if applicable): 25.1 How reproducible: not sure Steps to Reproduce: 1. Provision recipes in openstack 2. tail the watchdog.log 3. see the error appear Actual results: XMLRPC Faults Expected results: No fault Additional info: