Bug 1274859 - Director Update 7.0 to 7.1 : Openstack overcloud update stuck 'IN_PROGRESS' (heat trows AttributeError: 'NoneType')
Director Update 7.0 to 7.1 : Openstack overcloud update stuck 'IN_PROGRESS' ...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat (Show other bugs)
7.0 (Kilo)
x86_64 Linux
urgent Severity urgent
: async
: 7.0 (Kilo)
Assigned To: Steve Baker
Amit Ugol
: Triaged, ZStream
Depends On:
Blocks: 1272254
  Show dependency treegraph
 
Reported: 2015-10-23 13:22 EDT by Omri Hochman
Modified: 2016-04-27 00:25 EDT (History)
11 users (show)

See Also:
Fixed In Version: openstack-heat-2015.1.1-7.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-18 11:40:34 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
update.yaml (2.22 KB, text/plain)
2015-10-27 10:03 EDT, Omri Hochman
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1509880 None None None Never
OpenStack gerrit 239368 None None None Never

  None (edit)
Description Omri Hochman 2015-10-23 13:22:54 EDT
Update 7.0 to 7.1  : Openstack overcloud update stuck 'IN_PROGRESS'  (heat trows AttributeError: 'NoneType') 


Environment :
--------------
[stack@instack ~]$ rpm -qa | grep heat
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-45.el7ost.noarch
openstack-heat-api-2015.1.0-4.el7ost.noarch
openstack-heat-api-cfn-2015.1.1-6.el7ost.noarch
heat-cfntools-1.2.8-2.el7.noarch
openstack-heat-common-2015.1.0-4.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.1-6.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-4.el7ost.noarch
python-heatclient-0.6.0-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-4.el7ost.noarch
openstack-heat-common-2015.1.1-6.el7ost.noarch
openstack-heat-api-2015.1.1-6.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-71.el7ost.noarch
openstack-heat-engine-2015.1.1-6.el7ost.noarch
openstack-heat-engine-2015.1.0-4.el7ost.noarch



Steps:
-------
(1) Install Undercloud and Overcloud 7.0 (with 7.0 Images)
(2) Update the undercloud to 7.1 ( using rhos-release )
(3) make sure you have 7.1 repos on the overcloud nodes
(4) attempt to run the overcloud update command :  

(More details: http://etherpad.corp.redhat.com/update-ospd-7-0-to-7-1  )

openstack overcloud update stack overcloud -i --templates  -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /home/stack/update.yaml 


Results : 
---------
(1) It looks 'IN_PROGRESS' but nothing happens ": 

[stack@instack ~]$ openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /home/stack/update.yaml
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS

(2) Nothing happens but after restarting Heat service stack status change to:  'UPDATE_FAILED' 

journalctl -ab -u openstack-heat-engine:
------------------------------------------
-- Logs begin at Thu 2015-10-22 16:27:37 EDT, end at Fri 2015-10-23 12:11:18 EDT. --
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: Traceback (most recent call last):
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: timer()
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: cb(*args, **kw)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: result = function(*args, **kwargs)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 111, in _start_with_trace
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: profiler.init(**trace)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: return f(*args, **kwargs)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 867, in update
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: @scheduler.wrappertask
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 174, in __call__
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: self.start(timeout=timeout)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 200, in start
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: self.step()
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 223, in step
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: next(self._runner)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 289, in wrapper
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: subtask = next(parent)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 918, in update_task
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: updater.start(timeout=self.timeout_secs())
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 200, in start
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: self.step()
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 223, in step
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: next(self._runner)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 289, in wrapper
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: subtask = next(parent)
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/update.py", line 60, in __call__
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: self.dependencies(),
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/update.py", line 185, in dependencies
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: if res_name in self.previous_stack:
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 240, in dependencies
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 312, in _get_dependencies
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: return deps
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/port.py", line 263, in
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: if res.has_interface('OS::Neutron::Subnet'):
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 319, in has_interface
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: def type(self):
Oct 23 10:54:17 instack.localdomain heat-engine[30198]: AttributeError: 'NoneType' object has no attribute 'name'
~
Comment 3 Steve Baker 2015-10-25 05:09:00 EDT
Could you please attach /home/stack/update.yaml?

Also can you confirm that you had a successful run of "openstack overcloud deploy" before you ran "openstack overcloud update"?
Comment 4 Steve Baker 2015-10-25 18:26:50 EDT
I have a simple reproducer for upstream heat, and I think there will be a workaround by adding more type placeholders in your /home/stack/update.yaml
Comment 5 Omri Hochman 2015-10-27 10:03 EDT
Created attachment 1086852 [details]
update.yaml
Comment 6 Omri Hochman 2015-10-27 10:05:04 EDT
Yes, openstack 'overcloud deploy' was successful prior to the update attempt. 

Attached /home/stack/update.yaml.
Comment 8 Steve Baker 2015-10-28 18:07:11 EDT
The upstream fix has landed
Comment 12 errata-xmlrpc 2015-11-18 11:40:34 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2076

Note You need to log in before you can comment on or make changes to this bug.