Description of problem:
The polling loop in beaker-watchdog can sometimes fail with an exception.

Version-Release number of selected component (if applicable):
26.0

How reproducible:
unsure, happens intermittently on a busy LC

Steps to Reproduce:
1. Look at the logs as beaker-watchdog polls

Actual results:
bkr.labcontroller.watchdog ERROR Failed to poll for watchdogs
 Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/watchdog.py", line 174, in main_loop
     watchdog.poll()
   File "/usr/lib/python2.6/site-packages/bkr/labcontroller/watchdog.py", line 158, in poll
     for recipe_id, greenlet in self.monitor_greenlets.iteritems():
 RuntimeError: dictionary changed size during iteration

Expected results:
shouldn't fail

Additional info:
Probably a regression in 26.0 as that particular code is new in 26.0 for bug 991269.

The log also shows this, which might be related:

Oct 25 14:51:56 lab-02 beaker-watchdog[31762]: bkr.labcontroller.watchdog ERROR Monitor greenlet <Greenlet at 0x2381f50: run_monitor(<bkr.labcontroller.proxy.Monitor object at 0x27d5f)> had unhandled exception: <Fault 1: "<type 'exceptions.ValueError'>:Cannot record result for finished task T:82797280">