Bug 1568217

Summary: beaker-provision fails to poll for queued commands: AttributeError: 'NoneType' object has no attribute 'arch'
Product: [Community] Beaker Reporter: Dan Callaghan <dcallagh>
Component: generalAssignee: Dan Callaghan <dcallagh>
Status: CLOSED CURRENTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: 25CC: dcallagh, ebaak, mtyson, rjoost
Target Milestone: 25.2Keywords: Patch, Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-19 03:50:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Dan Callaghan 2018-04-17 01:27:02 UTC
Description of problem:


Version-Release number of selected component (if applicable):
25.1

How reproducible:
upgrade from 24.5 to 25.1 while recipes are running
(this is probably the key as to why we did not discover this during testing, we did encounter other edge cases around this already)

Steps to Reproduce:
1. Upgrade
2. Start beaker-provision

Actual results:
beaker-provision fails to poll for queued commands:

Apr 17 01:25:13 lab-02 beaker-provision[19545]: bkr.labcontroller.provision ERROR Failed to poll for queued commands
Apr 17 01:25:13 lab-02 beaker-provision[19545]:  Traceback (most recent call last):
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 289, in main_loop
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      poller.poll()
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 95, in poll
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      for command in self.get_queued_commands():
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/labcontroller/provision.py", line 41, in get_queued_commands
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      commands = self.hub.labcontrollers.get_queued_command_details()
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      return self.__send(self.__name, args)
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      verbose=self.__verbose
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 571, in request
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      result = transport_class.request(self, *args, **kwargs)
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib/python2.6/site-packages/bkr/common/xmlrpc.py", line 427, in _request
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      return self.parse_response(h.getfile())
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1361, in parse_response
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      return self._parse_response(file, None)
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 1392, in _parse_response
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      return u.close()
Apr 17 01:25:13 lab-02 beaker-provision[19545]:    File "/usr/lib64/python2.6/xmlrpclib.py", line 838, in close
Apr 17 01:25:13 lab-02 beaker-provision[19545]:      raise Fault(**self._stack[0])
Apr 17 01:25:13 lab-02 beaker-provision[19545]:  Fault: <Fault 1: "<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'arch'">

Corresponding server-side exception is:

Apr 17 01:25:53 beaker-server beaker-server[72622]: bkr.server.xmlrpccontroller ERROR Error handling XML-RPC method
Apr 17 01:25:53 beaker-server beaker-server[72622]:  Traceback (most recent call last):
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "/usr/lib/python2.6/site-packages/bkr/server/xmlrpccontroller.py", line 59, in RPC2
Apr 17 01:25:53 beaker-server beaker-server[72622]:      response = self.process_rpc(method,params)
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "/usr/lib/python2.6/site-packages/bkr/server/xmlrpccontroller.py", line 44, in process_rpc
Apr 17 01:25:53 beaker-server beaker-server[72622]:      response = obj(*params)
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "<string>", line 2, in get_queued_command_details
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "/usr/lib/python2.6/site-packages/bkr/server/identity.py", line 288, in require
Apr 17 01:25:53 beaker-server beaker-server[72622]:      return func(*args, **kwargs)
Apr 17 01:25:53 beaker-server beaker-server[72622]:    File "/usr/lib/python2.6/site-packages/bkr/server/labcontroller.py", line 469, in get_queued_command_details
Apr 17 01:25:53 beaker-server beaker-server[72622]:      'arch': installation.arch.arch,
Apr 17 01:25:53 beaker-server beaker-server[72622]:  AttributeError: 'NoneType' object has no attribute 'arch'

Expected results:
beaker-provision should run normally.

Additional info:
Most likely a regression in bug 911515 as we hit similar exceptions in other code paths, all relating to recipes that had been provisioned before the upgrade.

Comment 1 Dan Callaghan 2018-04-17 01:45:41 UTC
Indeed, there were two recipes in the database which had been provisioned right before the upgrade started, with power commands still queued:

mysql> SELECT * FROM command_queue INNER JOIN installation ON command_queue.installation_id = installation.id WHERE status = 'Queued' AND installation.arch_id IS NULL;
[...]
5 rows in set (0.01 sec)

The queued command ids were: 59742975, 59742976, 59742977, 59742978, 59742979.

Corresponding recipes were 5047226 and 5047398.

As a workaround, we ran the following to forcibly mark the commands Aborted so that they would no longer appear in the queue:

UPDATE command_queue SET status = 'Aborted' WHERE status = 'Queued' AND id IN (59742975, 59742976, 59742977, 59742978, 59742979);

and cancelled the corresponding recipes so that they are not left in a zombie state.

Comment 3 Dan Callaghan 2018-04-18 01:16:22 UTC
https://gerrit.beaker-project.org/6075

Comment 4 Roman Joost 2018-04-19 03:50:59 UTC
Released with 25.2: https://beaker-project.org/docs/whats-new/release-25.html#beaker-25-2

Due to their importance and impact on systems in use, we've decided to skip an expensive self test, rely on our automated tests and get them deployed ASAP.