Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1636458

Summary: overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_ (OSP-13)
Product: Red Hat OpenStack Reporter: Bob Fournier <bfournie>
Component: openstack-ironicAssignee: Dmitry Tantsur <dtantsur>
Status: CLOSED ERRATA QA Contact: mlammon
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: bfournie, dtantsur, jschluet, lmarsh, mburns
Target Milestone: z3Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ironic-10.1.6-2.el7ost Doc Type: Bug Fix
Doc Text:
In prior releases, a race condition existed in the ironic-conductor hash ring code. A hash ring can be None under load, but this causes an internal server error: 'NoneType' object has no attribute '__getitem_'. This release fixes the race condition, and ironic API operations no longer fail with 'NoneType' object has no attribute '__getitem_'.
Story Points: ---
Clone Of:
: 1638003 (view as bug list) Environment:
Last Closed: 2018-11-13 22:14:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1638003    

Description Bob Fournier 2018-10-05 13:01:15 UTC
Queens periodic jobs FS01 failed at overcloud prepare image step giving the following error:
https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/59f5f7c/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2018-09-16_06_30_22
2018-09-16 06:30:14.147 23784 ERROR wsme.api [req-0cf2f1f1-bdc0-4072-9467-f1d5a02d5704 f0f20bf416fc4313983b66ffbc830981 68c9abc94e5748b3b4cd999cdb0bd00f - default default] Server-side error: "'NoneType' object has no attribute '__getitem__'". Detail:
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 85, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/ironic/api/controllers/v1/node.py", line 609, in provision
    topic = pecan.request.rpcapi.get_topic_for(rpc_node)

  File "/usr/lib/python2.7/site-packages/ironic/conductor/rpcapi.py", line 136, in get_topic_for
    ring = self.ring_manager[node.driver]

  File "/usr/lib/python2.7/site-packages/ironic/common/hash_ring.py", line 67, in __getitem__
    return self.ring[driver_name]

TypeError: 'NoneType' object has no attribute '__getitem__'

Comment 13 mlammon 2018-11-08 22:49:36 UTC
Install UC + OC with latest osp 13 and deployment was  successful 

Environment:

openstack-ironic-staging-drivers-0.9.1-1.el7ost.noarch
openstack-ironic-api-10.1.6-2.el7ost.noarch
openstack-ironic-inspector-7.2.1-4.el7ost.noarch
openstack-ironic-conductor-10.1.6-2.el7ost.noarch
openstack-ironic-common-10.1.6-2.el7ost.noarch



tail -20 overcloud_install.log
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_IN_PROGRESS  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_IN_PROGRESS  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.CephStoragePostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_IN_PROGRESS  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.BlockStoragePostConfig]: CREATE_IN_PROGRESS  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.BlockStoragePostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  Stack CREATE completed successfully
2018-11-08 21:22:32Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  state changed
2018-11-08 21:22:32Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

 Stack overcloud CREATE_COMPLETE

Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: 682bbc3f-114b-4f49-b19f-f6b60463afd3
Overcloud Endpoint: http://10.0.0.102:5000/
Overcloud Horizon Dashboard URL: http://10.0.0.102:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed

openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| 5f11df6c-bfbc-4f47-b0dd-fb128b556a82 | compute-0    | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | compute    |
| 5220c028-bf40-43bd-842f-cb1cf1e2df4e | controller-0 | ACTIVE | ctlplane=192.168.24.9  | overcloud-full | controller |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+

See some new logs put in from oslo verified as well

root@undercloud-0 ironic]# grep -r 'Resetting cached hash rings' ironic-conductor.log


2018-11-08 17:44:40.089 16712 DEBUG ironic.common.hash_ring [-] Rebuilding cached hash rings ring /usr/lib/python2.7/site-packages/ironic/common/hash_ring.py:51

018-11-08 17:44:40.107 16712 DEBUG ironic.common.hash_ring [-] Finished rebuilding hash rings, available drivers are idrac, ipmi, pxe_drac, pxe_ilo, ilo, redfish, pxe_ipmitool ring /usr/lib/python2.7/site-packages/ironic/common/hash_ring.py:56

Comment 15 errata-xmlrpc 2018-11-13 22:14:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3605