Description of problem: Version-Release number of selected component (if applicable): 25.1 How reproducible: upgrade from 24.5 to 25.1 while recipes are running (this is probably the key as to why we did not discover this during testing, we did encounter other edge cases around this already) Steps to Reproduce: 1. Upgrade 2. Start beaker-provision Actual results: beaker-provision fails to poll for queued commands: Apr 17 01:25:13 lab-02 beaker-provision[19545]: bkr.labcontroller.provision ERROR Failed to poll for queued commands Apr 17 01:25:13 lab-02 beaker-provision[19545]: Traceback (most recent call last): Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 289, in main_loop Apr 17 01:25:13 lab-02 beaker-provision[19545]: poller.poll() Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 95, in poll Apr 17 01:25:13 lab-02 beaker-provision[19545]: for command in self.get_queued_commands(): Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 41, in get_queued_commands Apr 17 01:25:13 lab-02 beaker-provision[19545]: commands = self.hub.labcontrollers.get_queued_command_details() Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ Apr 17 01:25:13 lab-02 beaker-provision[19545]: return self.__send(self.__name, args) Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request Apr 17 01:25:13 lab-02 beaker-provision[19545]: verbose=self.__verbose Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 571, in request Apr 17 01:25:13 lab-02 beaker-provision[19545]: result = transport_class.request(self, *args, **kwargs) Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 427, in _request Apr 17 01:25:13 lab-02 beaker-provision[19545]: return self.parse_response(h.getfile()) Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib64/python2.6/xmlrpclib.py", line 1361, in parse_response Apr 17 01:25:13 lab-02 beaker-provision[19545]: return self._parse_response(file, None) Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response Apr 17 01:25:13 lab-02 beaker-provision[19545]: return u.close() Apr 17 01:25:13 lab-02 beaker-provision[19545]: File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close Apr 17 01:25:13 lab-02 beaker-provision[19545]: raise Fault(**self._stack[0]) Apr 17 01:25:13 lab-02 beaker-provision[19545]: Fault: <Fault 1: "<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'arch'"> Corresponding server-side exception is: Apr 17 01:25:53 beaker-server beaker-server[72622]: bkr.server.xmlrpccontroller ERROR Error handling XML-RPC method Apr 17 01:25:53 beaker-server beaker-server[72622]: Traceback (most recent call last): Apr 17 01:25:53 beaker-server beaker-server[72622]: File "/usr/lib/python2.6/site-packages/bkr/server/xmlrpccontroller.py", line 59, in RPC2 Apr 17 01:25:53 beaker-server beaker-server[72622]: response = self.process_rpc(method,params) Apr 17 01:25:53 beaker-server beaker-server[72622]: File "/usr/lib/python2.6/site-packages/bkr/server/xmlrpccontroller.py", line 44, in process_rpc Apr 17 01:25:53 beaker-server beaker-server[72622]: response = obj(*params) Apr 17 01:25:53 beaker-server beaker-server[72622]: File "<string>", line 2, in get_queued_command_details Apr 17 01:25:53 beaker-server beaker-server[72622]: File "/usr/lib/python2.6/site-packages/bkr/server/identity.py", line 288, in require Apr 17 01:25:53 beaker-server beaker-server[72622]: return func(*args, **kwargs) Apr 17 01:25:53 beaker-server beaker-server[72622]: File "/usr/lib/python2.6/site-packages/bkr/server/labcontroller.py", line 469, in get_queued_command_details Apr 17 01:25:53 beaker-server beaker-server[72622]: 'arch': installation.arch.arch, Apr 17 01:25:53 beaker-server beaker-server[72622]: AttributeError: 'NoneType' object has no attribute 'arch' Expected results: beaker-provision should run normally. Additional info: Most likely a regression in bug 911515 as we hit similar exceptions in other code paths, all relating to recipes that had been provisioned before the upgrade.
Indeed, there were two recipes in the database which had been provisioned right before the upgrade started, with power commands still queued: mysql> SELECT * FROM command_queue INNER JOIN installation ON command_queue.installation_id = installation.id WHERE status = 'Queued' AND installation.arch_id IS NULL; [...] 5 rows in set (0.01 sec) The queued command ids were: 59742975, 59742976, 59742977, 59742978, 59742979. Corresponding recipes were 5047226 and 5047398. As a workaround, we ran the following to forcibly mark the commands Aborted so that they would no longer appear in the queue: UPDATE command_queue SET status = 'Aborted' WHERE status = 'Queued' AND id IN (59742975, 59742976, 59742977, 59742978, 59742979); and cancelled the corresponding recipes so that they are not left in a zombie state.
https://gerrit.beaker-project.org/6075
Released with 25.2: https://beaker-project.org/docs/whats-new/release-25.html#beaker-25-2 Due to their importance and impact on systems in use, we've decided to skip an expensive self test, rely on our automated tests and get them deployed ASAP.