Bug 1636458 - overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_ (OSP-13)
Summary: overcloud prepare image failed by giving IronicAction.node.set_provision_stat...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z3
: 13.0 (Queens)
Assignee: Dmitry Tantsur
QA Contact: mlammon
URL:
Whiteboard:
Depends On:
Blocks: 1638003
TreeView+ depends on / blocked
 
Reported: 2018-10-05 13:01 UTC by Bob Fournier
Modified: 2018-11-13 22:15 UTC (History)
5 users (show)

Fixed In Version: openstack-ironic-10.1.6-2.el7ost
Doc Type: Bug Fix
Doc Text:
In prior releases, a race condition existed in the ironic-conductor hash ring code. A hash ring can be None under load, but this causes an internal server error: 'NoneType' object has no attribute '__getitem_'. This release fixes the race condition, and ironic API operations no longer fail with 'NoneType' object has no attribute '__getitem_'.
Clone Of:
: 1638003 (view as bug list)
Environment:
Last Closed: 2018-11-13 22:14:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1792872 0 None None None 2018-10-05 13:01:15 UTC
OpenStack gerrit 608678 0 None MERGED Fixes a race condition in the hash ring code 2020-06-15 20:10:06 UTC
Red Hat Product Errata RHBA-2018:3605 0 None None None 2018-11-13 22:15:31 UTC

Description Bob Fournier 2018-10-05 13:01:15 UTC
Queens periodic jobs FS01 failed at overcloud prepare image step giving the following error:
https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/59f5f7c/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2018-09-16_06_30_22
2018-09-16 06:30:14.147 23784 ERROR wsme.api [req-0cf2f1f1-bdc0-4072-9467-f1d5a02d5704 f0f20bf416fc4313983b66ffbc830981 68c9abc94e5748b3b4cd999cdb0bd00f - default default] Server-side error: "'NoneType' object has no attribute '__getitem__'". Detail:
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 85, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/ironic/api/controllers/v1/node.py", line 609, in provision
    topic = pecan.request.rpcapi.get_topic_for(rpc_node)

  File "/usr/lib/python2.7/site-packages/ironic/conductor/rpcapi.py", line 136, in get_topic_for
    ring = self.ring_manager[node.driver]

  File "/usr/lib/python2.7/site-packages/ironic/common/hash_ring.py", line 67, in __getitem__
    return self.ring[driver_name]

TypeError: 'NoneType' object has no attribute '__getitem__'

Comment 13 mlammon 2018-11-08 22:49:36 UTC
Install UC + OC with latest osp 13 and deployment was  successful 

Environment:

openstack-ironic-staging-drivers-0.9.1-1.el7ost.noarch
openstack-ironic-api-10.1.6-2.el7ost.noarch
openstack-ironic-inspector-7.2.1-4.el7ost.noarch
openstack-ironic-conductor-10.1.6-2.el7ost.noarch
openstack-ironic-common-10.1.6-2.el7ost.noarch



tail -20 overcloud_install.log
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_IN_PROGRESS  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_IN_PROGRESS  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.CephStoragePostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_IN_PROGRESS  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.BlockStoragePostConfig]: CREATE_IN_PROGRESS  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps.BlockStoragePostConfig]: CREATE_COMPLETE  state changed
2018-11-08 21:22:31Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  Stack CREATE completed successfully
2018-11-08 21:22:32Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  state changed
2018-11-08 21:22:32Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

 Stack overcloud CREATE_COMPLETE

Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: 682bbc3f-114b-4f49-b19f-f6b60463afd3
Overcloud Endpoint: http://10.0.0.102:5000/
Overcloud Horizon Dashboard URL: http://10.0.0.102:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed

openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| 5f11df6c-bfbc-4f47-b0dd-fb128b556a82 | compute-0    | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | compute    |
| 5220c028-bf40-43bd-842f-cb1cf1e2df4e | controller-0 | ACTIVE | ctlplane=192.168.24.9  | overcloud-full | controller |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+

See some new logs put in from oslo verified as well

root@undercloud-0 ironic]# grep -r 'Resetting cached hash rings' ironic-conductor.log


2018-11-08 17:44:40.089 16712 DEBUG ironic.common.hash_ring [-] Rebuilding cached hash rings ring /usr/lib/python2.7/site-packages/ironic/common/hash_ring.py:51

018-11-08 17:44:40.107 16712 DEBUG ironic.common.hash_ring [-] Finished rebuilding hash rings, available drivers are idrac, ipmi, pxe_drac, pxe_ilo, ilo, redfish, pxe_ipmitool ring /usr/lib/python2.7/site-packages/ironic/common/hash_ring.py:56

Comment 15 errata-xmlrpc 2018-11-13 22:14:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3605


Note You need to log in before you can comment on or make changes to this bug.