Bug 1385703 - OSP-9/10 upgrades fails on ceilometer migration.
Summary: OSP-9/10 upgrades fails on ceilometer migration.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 10.0 (Newton)
Assignee: Carlos Camacho
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks: 1337794
TreeView+ depends on / blocked
 
Reported: 2016-10-17 14:15 UTC by Sofer Athlan-Guyot
Modified: 2016-12-29 16:56 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:21:28 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 382737 None None None 2016-10-17 16:08:56 UTC

Description Sofer Athlan-Guyot 2016-10-17 14:15:34 UTC
Description of problem:  The upgrade from osp9 to osp10 fails during ceilometer migration (environments/major-upgrade-ceilometer-wsgi-mitaka-newton.yaml) with:

    Exception updating plan: Error parsing template http://192.0.2.1:8080/v1/AUTH_6704eac41451416d9712c3f08f6d2f54/overcloud/overcloud.yaml Template format version not found.

Version-Release number of selected component (if applicable):  

openstack-tripleo-heat-templates-5.0.0-0.20161003064637.d636e3a.1.2.el7ost.noarch from puddle of 2016/10/17.

How reproducible: Only once.

Comment 1 Sofer Athlan-Guyot 2016-10-17 14:24:49 UTC
More logs:

    mistral/executor.log:2016-10-17 09:40:04.849 23952 ERROR tripleo_common.actions.templates [-] Error occurred while processing plan files.
    mistral/executor.log:2016-10-17 09:40:04.849 23952 ERROR tripleo_common.actions.templates Traceback (most recent call last):
    mistral/executor.log:2016-10-17 09:40:04.849 23952 ERROR tripleo_common.actions.templates   File "/usr/lib/python2.7/site-packages/tripleo_common/actions/templates.py", line 237, in run
    mistral/executor.log:2016-10-17 09:40:04.849 23952 ERROR tripleo_common.actions.templates     object_request=_object_request)
    mistral/executor.log:2016-10-17 09:40:04.849 23952 ERROR tripleo_common.actions.templates   File "/usr/lib/python2.7/site-packages/heatclient/common/template_utils.py", line 93, in get_template_contents
    mistral/executor.log:2016-10-17 09:40:04.849 23952 ERROR tripleo_common.actions.templates     {'url': template_url, 'error': e})
    mistral/executor.log:2016-10-17 09:40:04.849 23952 ERROR tripleo_common.actions.templates CommandError: Error parsing template http://192.0.2.1:8080/v1/AUTH_6704eac41451416d9712c3f08f6d2f54/overcloud/overcloud.yaml Template format version not found.
    mistral/executor.log:2016-10-17 09:40:04.849 23952 ERROR tripleo_common.actions.templates 
    mistral/executor.log:2016-10-17 10:09:57.103 23952 ERROR swiftclient [-] Object GET failed: http://192.0.2.1:8080/v1/AUTH_6704eac41451416d9712c3f08f6d2f54/overcloud/j2_excludes.yaml 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<
    mistral/executor.log:2016-10-17 10:09:57.103 23952 ERROR swiftclient Traceback (most recent call last):
    mistral/executor.log:2016-10-17 10:09:57.103 23952 ERROR swiftclient   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1647, in _retry
    mistral/executor.log:2016-10-17 10:09:57.103 23952 ERROR swiftclient     service_token=self.service_token, **kwargs)
    mistral/executor.log:2016-10-17 10:09:57.103 23952 ERROR swiftclient   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1139, in get_object
    mistral/executor.log:2016-10-17 10:09:57.103 23952 ERROR swiftclient     raise ClientException.from_response(resp, 'Object GET failed', body)
    mistral/executor.log:2016-10-17 10:09:57.103 23952 ERROR swiftclient ClientException: Object GET failed: http://192.0.2.1:8080/v1/AUTH_6704eac41451416d9712c3f08f6d2f54/overcloud/j2_excludes.yaml 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<

Comment 2 Omri Hochman 2016-10-17 14:40:44 UTC
We might be able to continue with upgrade all the way without running this step - in that case it's not a test-blocker, but on the other hand we need a PM to decide if it's a GA blocker or not

Comment 3 Jaromir Coufal 2016-10-17 15:01:33 UTC
Omri, this means that we will be able to upgrade openstack but ceilometer, correct?

Comment 5 Sofer Athlan-Guyot 2016-10-17 16:08:57 UTC
Using this upstream patch this error disappears.

Comment 6 James Slagle 2016-10-17 18:30:56 UTC
(In reply to Sofer Athlan-Guyot from comment #5)
> Using this upstream patch this error disappears.

as that patch is merged upstream in newton, moving this bz to POST

Comment 7 James Slagle 2016-10-17 18:31:34 UTC
reassigning to carlos as well since he authored the usptream patch

Comment 8 Carlos Camacho 2016-10-17 18:41:31 UTC
I saw that error before, and it was because the templates were not rendering correctly (empty) when not having j2_excludes.yaml file, fixed here https://review.openstack.org/#/c/385471/1/tripleo_common/actions/templates.py

Also, this was backported to stable/newton


Are you using an updated version of tripleo-common?

We shouldn't have this error again.

Comment 9 Marios Andreou 2016-10-18 08:56:43 UTC
marking as blocking the Upgrades RFE ...

fantastic thanks very much Sofer and Carlos... fwiw and as discussed on scrum yesterday I was also hitting this on my env 

        [stack@instack ~]$ openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org'  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-ceilometer-wsgi-mitaka-newton.yaml  -e /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-overcloud-compute-hostnames.yaml
        Removing the current plan files
        Uploading new plan files
        Started Mistral Workflow. Execution ID: 1ff7969f-bf1a-4232-ab57-b6fb3cf17f87
        Exception updating plan: Error parsing template http://192.0.2.1:8080/v1/AUTH_bd6a8c1f981d41b49edf002f2d9973bc/overcloud/overcloud.yaml Template format version not found.

I see Sofer has included the workaround in the upgrades readme https://gitlab.cee.redhat.com/sathlang/ospd-9-to-10-upgrade/blob/master/README.md#patches-workaround-1 so I'll be including that in my run today as well

thanks, marios

Comment 10 Sofer Athlan-Guyot 2016-10-18 09:18:20 UTC
Hi Carlos,

As I put it in the bz, we use the latest downstream puddle and it misses the bit you've commited in the attached gerrit review.  It's just lag between upstream and downstream.  I'm going to check that the patch you've pointed out is included as well.  Thanks.

Comment 11 Sofer Athlan-Guyot 2016-10-18 16:01:01 UTC
The patch relates to this bug : https://bugzilla.redhat.com/show_bug.cgi?id=1385470 where a package has been done.

Comment 14 Omri Hochman 2016-11-07 21:47:11 UTC
unable to reproduce: openstack-tripleo-heat-templates-5.0.0-1.2.el7ost.noarch

Comment 16 errata-xmlrpc 2016-12-14 16:21:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.