Bug 1568217 - beaker-provision fails to poll for queued commands: AttributeError: 'NoneType' object has no attribute 'arch'
Summary: beaker-provision fails to poll for queued commands: AttributeError: 'NoneType...
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Community
Component: general
Version: 25
Hardware: Unspecified
OS: Unspecified
urgent
unspecified vote
Target Milestone: 25.2
Assignee: Dan Callaghan
QA Contact: tools-bugs
URL:
Whiteboard:
Keywords: Patch, Regression
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-17 01:27 UTC by Dan Callaghan
Modified: 2018-04-19 03:50 UTC (History)
4 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2018-04-19 03:50:59 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Beaker Project Gerrit 6075 None None None 2018-04-18 01:16 UTC

Description Dan Callaghan 2018-04-17 01:27:02 UTC
Description of problem:


Version-Release number of selected component (if applicable):
25.1

How reproducible:
upgrade from 24.5 to 25.1 while recipes are running
(this is probably the key as to why we did not discover this during testing, we did encounter other edge cases around this already)

Steps to Reproduce:
1. Upgrade
2. Start beaker-provision

Actual results:
beaker-provision fails to poll for queued commands:

Apr 17 01:25:13 lab-02 beaker-provision[19545]: bkr.labcontroller.provision ERROR Failed to poll for queued commands
Apr 17 01:25:13 lab-02 beaker-provision[19545]:  Traceback (most recent call last):
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 289, in main_loop
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      poller.poll()
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 95, in poll
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      for command in self.get_queued_commands():
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 41, in get_queued_commands
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      commands = self.hub.labcontrollers.get_queued_command_details()
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      return self.__send(self.__name, args)
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      verbose=self.__verbose
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 571, in request
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      result = transport_class.request(self, *args, **kwargs)
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 427, in _request
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      return self.parse_response(h.getfile())
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1361, in parse_response
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      return self._parse_response(file, None)
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      return u.close()
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      raise Fault(**self._stack[0])
Apr 17 01:25:13 lab-02 beaker-provision[19545]:  Fault: <Fault 1: "<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'arch'">

Corresponding server-side exception is:

Apr 17 01:25:53 beaker-server beaker-server[72622]: bkr.server.xmlrpccontroller ERROR Error handling XML-RPC method
Apr 17 01:25:53 beaker-server beaker-server[72622]:  Traceback (most recent call last):
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "/usr/lib/python2.6/site-packages/bkr/server/xmlrpccontroller.py", line 59, in RPC2
Apr 17 01:25:53 beaker-server beaker-server[72622]:      response = self.process_rpc(method,params)
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "/usr/lib/python2.6/site-packages/bkr/server/xmlrpccontroller.py", line 44, in process_rpc
Apr 17 01:25:53 beaker-server beaker-server[72622]:      response = obj(*params)
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "<string>", line 2, in get_queued_command_details
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "/usr/lib/python2.6/site-packages/bkr/server/identity.py", line 288, in require
Apr 17 01:25:53 beaker-server beaker-server[72622]:      return func(*args, **kwargs)
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "/usr/lib/python2.6/site-packages/bkr/server/labcontroller.py", line 469, in get_queued_command_details
Apr 17 01:25:53 beaker-server beaker-server[72622]:      'arch': installation.arch.arch,
Apr 17 01:25:53 beaker-server beaker-server[72622]:  AttributeError: 'NoneType' object has no attribute 'arch'

Expected results:
beaker-provision should run normally.

Additional info:
Most likely a regression in bug 911515 as we hit similar exceptions in other code paths, all relating to recipes that had been provisioned before the upgrade.

Comment 1 Dan Callaghan 2018-04-17 01:45:41 UTC
Indeed, there were two recipes in the database which had been provisioned right before the upgrade started, with power commands still queued:

mysql> SELECT * FROM command_queue INNER JOIN installation ON command_queue.installation_id = installation.id WHERE status = 'Queued' AND installation.arch_id IS NULL;
[...]
5 rows in set (0.01 sec)

The queued command ids were: 59742975, 59742976, 59742977, 59742978, 59742979.

Corresponding recipes were 5047226 and 5047398.

As a workaround, we ran the following to forcibly mark the commands Aborted so that they would no longer appear in the queue:

UPDATE command_queue SET status = 'Aborted' WHERE status = 'Queued' AND id IN (59742975, 59742976, 59742977, 59742978, 59742979);

and cancelled the corresponding recipes so that they are not left in a zombie state.

Comment 3 Dan Callaghan 2018-04-18 01:16:22 UTC
https://gerrit.beaker-project.org/6075

Comment 4 Roman Joost 2018-04-19 03:50:59 UTC
Released with 25.2: https://beaker-project.org/docs/whats-new/release-25.html#beaker-25-2

Due to their importance and impact on systems in use, we've decided to skip an expensive self test, rely on our automated tests and get them deployed ASAP.


Note You need to log in before you can comment on or make changes to this bug.