Bug 1274859

Summary:

Director Update 7.0 to 7.1 : Openstack overcloud update stuck 'IN_PROGRESS' (heat trows AttributeError: 'NoneType')

Product:

Red Hat OpenStack

Reporter:

Omri Hochman <ohochman>

Component:

openstack-heat

Assignee:

Steve Baker <sbaker>

Status:

CLOSED ERRATA

QA Contact:

Amit Ugol <augol>

Severity:

urgent

Docs Contact:

Priority:

urgent

Version:

7.0 (Kilo)

CC:

calfonso, ddomingo, jslagle, mburns, mcornea, ohochman, rhel-osp-director-maint, sbaker, shardy, yeylon, zbitter

Target Milestone:

async

Keywords:

Triaged, ZStream

Target Release:

7.0 (Kilo)

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

openstack-heat-2015.1.1-7.el7ost

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2015-11-18 16:40:34 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1272254

Attachments:

Description	Flags
update.yaml	none

Description Omri Hochman 2015-10-23 17:22:54 UTC

Update 7.0 to 7.1  : Openstack overcloud update stuck 'IN_PROGRESS'  (heat trows AttributeError: 'NoneType') 


Environment :
--------------
[stack@instack ~]$ rpm -qa | grep heat
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-45.el7ost.noarch
openstack-heat-api-2015.1.0-4.el7ost.noarch
openstack-heat-api-cfn-2015.1.1-6.el7ost.noarch
heat-cfntools-1.2.8-2.el7.noarch
openstack-heat-common-2015.1.0-4.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.1-6.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-4.el7ost.noarch
python-heatclient-0.6.0-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-4.el7ost.noarch
openstack-heat-common-2015.1.1-6.el7ost.noarch
openstack-heat-api-2015.1.1-6.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-71.el7ost.noarch
openstack-heat-engine-2015.1.1-6.el7ost.noarch
openstack-heat-engine-2015.1.0-4.el7ost.noarch



Steps:
-------
(1) Install Undercloud and Overcloud 7.0 (with 7.0 Images)
(2) Update the undercloud to 7.1 ( using rhos-release )
(3) make sure you have 7.1 repos on the overcloud nodes
(4) attempt to run the overcloud update command :  

(More details: http://etherpad.corp.redhat.com/update-ospd-7-0-to-7-1  )

openstack overcloud update stack overcloud -i --templates  -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /home/stack/update.yaml 


Results : 
---------
(1) It looks 'IN_PROGRESS' but nothing happens ": 

[stack@instack ~]$ openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /home/stack/update.yaml
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS

(2) Nothing happens but after restarting Heat service stack status change to:  'UPDATE_FAILED' 

journalctl -ab -u openstack-heat-engine:
------------------------------------------
-- Logs begin at Thu 2015-10-22 16:27:37 EDT, end at Fri 2015-10-23 12:11:18 EDT. --
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: Traceback (most recent call last):
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: timer()
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: cb(*args, **kw)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: result = function(*args, **kwargs)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 111, in _start_with_trace
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: profiler.init(**trace)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: return f(*args, **kwargs)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 867, in update
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: @scheduler.wrappertask
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 174, in __call__
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: self.start(timeout=timeout)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 200, in start
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: self.step()
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 223, in step
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: next(self._runner)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 289, in wrapper
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: subtask = next(parent)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 918, in update_task
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: updater.start(timeout=self.timeout_secs())
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 200, in start
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: self.step()
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 223, in step
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: next(self._runner)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 289, in wrapper
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: subtask = next(parent)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/update.py", line 60, in __call__
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: self.dependencies(),
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/update.py", line 185, in dependencies
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: if res_name in self.previous_stack:
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 240, in dependencies
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 312, in _get_dependencies
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: return deps
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/port.py", line 263, in
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: if res.has_interface('OS::Neutron::Subnet'):
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 319, in has_interface
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: def type(self):
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: AttributeError: 'NoneType' object has no attribute 'name'
~

Comment 3 Steve Baker 2015-10-25 09:09:00 UTC

Could you please attach /home/stack/update.yaml?

Also can you confirm that you had a successful run of "openstack overcloud deploy" before you ran "openstack overcloud update"?

Comment 4 Steve Baker 2015-10-25 22:26:50 UTC

I have a simple reproducer for upstream heat, and I think there will be a workaround by adding more type placeholders in your /home/stack/update.yaml

Comment 5 Omri Hochman 2015-10-27 14:03:07 UTC

Created attachment 1086852 [details]
update.yaml

Comment 6 Omri Hochman 2015-10-27 14:05:04 UTC

Yes, openstack 'overcloud deploy' was successful prior to the update attempt. 

Attached /home/stack/update.yaml.

Comment 8 Steve Baker 2015-10-28 22:07:11 UTC

The upstream fix has landed

Comment 12 errata-xmlrpc 2015-11-18 16:40:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2076