Bug 1480011

Summary: Upgrading from OSP10 to OSP11 with OVS-DPDK fails
Product: Red Hat OpenStack Reporter: shivapriya.o.hiremath
Component: openstack-tripleo-heat-templatesAssignee: mathieu bultel <mbultel>
Status: CLOSED WONTFIX QA Contact: Marius Cornea <mcornea>
Severity: high Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: aschultz, dbayly, jdonohue, mbultel, mburns, mcornea, rhel-osp-director-maint, robert.h.armstrong, robert.w.love, sarita.maini, sathlang
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-25 16:11:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1335596, 1409892    
Attachments:
Description Flags
Log from the Overcloud deployment attached
none
contains all the templates used to deploy overcloud none

Description shivapriya.o.hiremath 2017-08-09 22:51:27 UTC
Created attachment 1311424 [details]
Log from the Overcloud deployment attached

Description of problem: Error in parsing the template while upgrading from OSP10 to OSP11


Version-Release number of selected component (if applicable):
OSP10 to OSP11 upgrade process




Steps to Reproduce:
1.Minor Version Upgrade of OSP 10 
2.Major version upgrade of OSP10 to OSP11 as given here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html-single/upgrading_red_hat_openstack_platform/#chap-Director-Major-Upgrade
3.Undercloud upgrade
4.Updating Red Hat OSP 10 OVE 2.5 to OVS 2.6 
5.Overcloud Upgrade

Actual results:
Error parsing template: while scanning for the next token\nfound character \'%\' that cannot start any token\n  in "<unicode string>", line 4, column 2:\n    {% set enabled_roles = roles|reje ... \n     ^',
 u'status': u'FAILED'}

Expected results: Succesful Overcloud deployment


Additional info:Log from the Overcloud deployment attached. It looks like the error arises from parsing the major_upgrade_steps.j2.yaml

Comment 1 shivapriya.o.hiremath 2017-08-16 23:41:56 UTC
Note that I had to change the file name of 'major_upgrade_steps.yaml' to 'major_upgrade_steps.j2.yaml' in 'major-upgrade-composable-steps.yaml' since the former file is not present in the folder specified. 

If the file name is not changed, the error is the following: 

ERROR: Failed to validate: : resources.AllNodesDeploySteps: : Failed to validate: resources.AllNodesPostUpgradeSteps: Error parsing template: mapping values are not allowed here\n  in "<unicode string>", line 3, column 14:\n      description: >\n                 ^',
 u'status': u'FAILED'}

Comment 3 mathieu bultel 2017-08-24 14:52:47 UTC
Hi,

Looking at your upgrade command, I guess your OSP 11 tripleo-heat-templates files are under templates_osp11/
So you need to specify it to your upgrade command:
openstack overcloud deploy --templates templates_osp11 -r ....
If not, the cli will use the /usr/share/openstack-tripleo-heat-templates by default and won't use correctly the others environment files.

Not that the templates_osp11 should be match exactly to what is under /usr/share/openstack-tripleo-heat-templates/

Could you retry and see what happen ?

Btw the e ~/templates_osp11/major-upgrade-composable-steps.yaml should be the one under environments/major-upgrade-composable-steps.yaml

Comment 4 shivapriya.o.hiremath 2017-08-24 17:13:19 UTC
When I add 'openstack overcloud deploy --templates templates_osp11' I get the following error.

[stack@vm-director ~]$ openstack overcloud deploy --templates templates_osp11/ -r ~/templates_osp11/roles_data_dpdk.yaml -e ~/templates_osp11/computeovsdpdk-env.yaml -e ~/templates_osp11/custom-network-environment.yaml  -e ~/templates_osp11/major-upgrade-composable-steps.yaml --control-scale 1 --compute-scale 0 --control-flavor control --ntp-server pool.ntp.org
Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: aaf3e161-a0c4-4904-8af9-7cac9a8ea8ba
Exception updating plan: File missing from container: Object GET failed: https://192.168.24.62:13808/v1/AUTH_45949e349ce04f2daa5e702883d3c166/overcloud/capabilities-map.yaml 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<

[stack@vm-director ~]$ curl https://192.168.24.62:13808/v1/AUTH_45949e349ce04f2daa5e702883d3c166/overcloud/capabilities-map.yaml
<html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>

I have used and modified the templates from the /usr/share/openstack-tripleo-heat-templates/. 
~/templates_osp11/major-upgrade-composable-steps.yaml is the one under environments/major-upgrade-composable-steps.yaml. I have also copied the puppet folder from /usr/share/openstack-tripleo-heat-templates into /home/stack/ folder since major-upgrade-composable-steps.yaml refers to it.

Comment 5 mathieu bultel 2017-08-24 17:18:42 UTC
I think the issue is still due to the path of your environment files.
Why do you use a modified major-upgrade-composable-steps.yaml ?
And could you paste the content of the file, or even better, attach all your custom environment files ?

Thank you by advance.

Comment 6 shivapriya.o.hiremath 2017-08-24 17:27:33 UTC
Created attachment 1317844 [details]
contains all the templates used to deploy overcloud

Comment 7 shivapriya.o.hiremath 2017-08-24 17:29:24 UTC
I have not modified major-upgrade-composable-steps.yaml. I have made a copy of it  in templates_osp11 folder. 
Please find attached all the files from templates_osp11 used to deploy overcloud.

Comment 8 mathieu bultel 2017-08-25 15:01:09 UTC
Hi,

Thank you for the additional information.
So only "user files" could be outside of the tripleo-heat-templates tree, if not you will hit some issue between full-path relative-path.
I think this is what happening here.
I suggest you to do something like:
$ openstack overcloud deploy --templates -r ~/templates_osp11/roles_data_dpdk.yaml -e ~/templates_osp11/computeovsdpdk-env.yaml -e ~/templates_osp11/custom-network-environment.yaml  -e /usr/share/openstack-tripleo-heat-templates/major-upgrade-composable-steps.yaml --control-scale 1 --compute-scale 0 --control-flavor control --ntp-server pool.ntp.org

I think the upgrade will be able to start correctly then.

Comment 9 shivapriya.o.hiremath 2017-08-25 16:11:05 UTC
Thank you for looking into the issue. I did try using the files from /usr/share/ folder directly but still got issues. 
We have decided to deploy OSP 11 on the latest RHEL 7.4 as recommended by Red Hat 
 since this issue has been open for a long time. I will close this ticket since we do not have the stack up and running to debug this issue.