It seems the underlying issue here is that the systems have ended up in Automated while the lab controller was removed. This means instead of keeping the systems in 'Automated' it would be better to change them to Broken, to avoid confusion for users.
Patch available: https://gerrit.beaker-project.org/#/c/5064
Dumping my thoughts on the progress to fix this bug. So it turns out that setting all machines connected to the lab controller broken is currently done by a bulk update operation. The problem with that is that it does not update the SystemStatusDuration leading to an invalid state: https://beaker-project.org/jenkins-results/beaker-review-checks-dogfood-RedHatEnterpriseLinux6/511/beaker/J%3a10275/344665-test_log--distribution-beaker-dogfood-tests.log (e.g. last test failure) The problem is reproducable by running two test cases like: ./run-tests.sh bkr.inttest.server.selenium.test_labcontrollers:LabControllerViewTest bkr.inttest.server.test_beakerd:TestBeakerdMetrics.test_dirty_job_metrics -xs Next things todo: * Dan mentioned that the code which currently performs a bulk update (see labcontroller.py:74) operation was put in place because doing this in Python code will probably perform very badly if e.g. we set 2000 machines to broken. I'll have to see if I can find a bug describing a problen in regards to a BOS labcontroller decommissioned and the fallout of that. * Check since the last time, we could now perform the update in Python without a big performance impact.
Beaker 23.3 has been released.