Bug 1467672

Summary: Cannot upgrade OSP11 to OSP12: Overcloud upgrade failed "ERROR: Property error: : resources.AllNodesDeploySteps.properties: : Unknown Property ctlplane_service_ips"
Product: Red Hat OpenStack Reporter: Artem Hrechanychenko <ahrechan>
Component: openstack-containersAssignee: mathieu bultel <mbultel>
Status: CLOSED WORKSFORME QA Contact: Omri Hochman <ohochman>
Severity: high Docs Contact: Andrew Burden <aburden>
Priority: high    
Version: 12.0 (Pike)CC: ahrechan, m.andre, mbultel, mcornea, morazi, rhallise, tvignaud
Target Milestone: rcKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-25 14:54:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Artem Hrechanychenko 2017-07-04 14:03:01 UTC
Description of problem:
Cannot upgrade OSP11->OSP12 overcloud - ERROR: Property error: : resources.AllNodesDeploySteps.properties: : Unknown Property ctlplane_service_ips


Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: 223ad668-ae27-44d3-b9cf-30435af6dc7d
Waiting for messages on queue 'c43065b4-63b1-4208-8747-adca2776fad7' with no timeout.






Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: 751aea40-6b8b-4d29-a81f-8fbf5aa53576
Plan updated.
Plan updated
Deploying templates in the directory /tmp/tripleoclient-jZoTq7/tripleo-heat-templates
Started Mistral Workflow tripleo.deployment.v1.deploy_plan. Execution ID: 4b743f4b-3da4-4d95-a87c-b19b5ddbeb7d
{u'execution': {u'created_at': u'2017-07-04 13:43:24',
                u'id': u'4b743f4b-3da4-4d95-a87c-b19b5ddbeb7d',
                u'input': {u'container': u'overcloud',
                           u'queue_name': u'2577618e-4c53-407d-a50e-3487d90e25e8',
                           u'run_validations': False,
                           u'skip_deploy_identifier': False,
                           u'timeout': 240},
                u'name': u'tripleo.deployment.v1.deploy_plan',
                u'params': {},
                u'spec': {u'description': u'Deploy the overcloud for a plan.\n',
                          u'input': [u'container',
                                     {u'run_validations': False},
                                     {u'timeout': 240},
                                     {u'skip_deploy_identifier': False},
                                     {u'queue_name': u'tripleo'}],
                          u'name': u'deploy_plan',
                          u'tasks': {u'add_validation_ssh_key': {u'input': {u'container': u'<% $.container %>',
                                                                            u'queue_name': u'<% $.queue_name %>'},
                                                                 u'name': u'add_validation_ssh_key',
                                                                 u'on-complete': [{u'run_validations': u'<% $.run_validations %>'},
                                                                                  {u'create_swift_rings_backup_plan': u'<% not $.run_validations %>'}],
                                                                 u'type': u'direct',
                                                                 u'version': u'2.0',
                                                                 u'workflow': u'tripleo.validations.v1.add_validation_ssh_key_parameter'},
                                     u'create_swift_rings_backup_plan': {u'input': {u'container': u'<% $.container %>',
                                                                                    u'queue_name': u'<% $.queue_name %>',
                                                                                    u'use_default_templates': True},
                                                                         u'name': u'create_swift_rings_backup_plan',
                                                                         u'on-error': u'create_swift_rings_backup_plan_set_status_failed',
                                                                         u'on-success': u'get_heat_stack',
                                                                         u'type': u'direct',
                                                                         u'version': u'2.0',
                                                                         u'workflow': u'tripleo.swift_rings_backup.v1.create_swift_rings_backup_container_plan'},
                                     u'create_swift_rings_backup_plan_set_status_failed': {u'name': u'create_swift_rings_backup_plan_set_status_failed',
                                                                                           u'on-success': u'send_message',
                                                                                           u'publish': {u'message': u'<% task(create_swift_rings_backup_plan).result %>',
                                                                                                        u'status': u'FAILED'},
                                                                                           u'type': u'direct',
                                                                                           u'version': u'2.0'},
                                     u'deploy': {u'action': u'tripleo.deployment.deploy',
                                                 u'input': {u'container': u'<% $.container %>',
                                                            u'skip_deploy_identifier': u'<% $.skip_deploy_identifier %>',
                                                            u'timeout': u'<% $.timeout %>'},
                                                 u'name': u'deploy',
                                                 u'on-error': u'set_deployment_failed',
                                                 u'on-success': u'send_message',
                                                 u'type': u'direct',
                                                 u'version': u'2.0'},
                                     u'get_heat_stack': {u'action': u'heat.stacks_get stack_id=<% $.container %>',
                                                         u'name': u'get_heat_stack',
                                                         u'on-error': u'deploy',
                                                         u'on-success': [{u'set_stack_in_progress': u'<% "_IN_PROGRESS" in task(get_heat_stack).result.stack_status %>'},
                                                                         {u'deploy': u'<% not "_IN_PROGRESS" in task(get_heat_stack).result.stack_status %>'}],
                                                         u'type': u'direct',
                                                         u'version': u'2.0'},
                                     u'run_validations': {u'input': {u'group_names': [u'pre-deployment'],
                                                                     u'plan': u'<% $.container %>',
                                                                     u'queue_name': u'<% $.queue_name %>'},
                                                          u'name': u'run_validations',
                                                          u'on-error': u'set_validations_failed',
                                                          u'on-success': u'create_swift_rings_backup_plan',
                                                          u'type': u'direct',
                                                          u'version': u'2.0',
                                                          u'workflow': u'tripleo.validations.v1.run_groups'},
                                     u'send_message': {u'action': u'zaqar.queue_post',
                                                       u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
                                                                                                       u'message': u"<% $.get('message', '') %>",
                                                                                                       u'status': u"<% $.get('status', 'SUCCESS') %>"},
                                                                                          u'type': u'tripleo.deployment.v1.deploy_plan'}},
                                                                  u'queue_name': u'<% $.queue_name %>'},
                                                       u'name': u'send_message',
                                                       u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}],
                                                       u'retry': u'count=5 delay=1',
                                                       u'type': u'direct',
                                                       u'version': u'2.0'},
                                     u'set_deployment_failed': {u'name': u'set_deployment_failed',
                                                                u'on-success': u'send_message',
                                                                u'publish': {u'message': u'<% task(deploy).result %>',
                                                                             u'status': u'FAILED'},
                                                                u'type': u'direct',
                                                                u'version': u'2.0'},
                                     u'set_stack_in_progress': {u'name': u'set_stack_in_progress',
                                                                u'on-success': u'send_message',
                                                                u'publish': {u'message': u'The Heat stack is busy.',
                                                                             u'status': u'FAILED'},
                                                                u'type': u'direct',
                                                                u'version': u'2.0'},
                                     u'set_validations_failed': {u'name': u'set_validations_failed',
                                                                 u'on-success': u'send_message',
                                                                 u'publish': {u'message': u'<% task(run_validations).result %>',
                                                                              u'status': u'FAILED'},
                                                                 u'type': u'direct',
                                                                 u'version': u'2.0'}},
                          u'version': u'2.0'}},
 u'message': u"Failed to run action [action_ex_id=98f19235-6394-424e-87f1-326fd208b3e6, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'skip_deploy_identifier': False, u'container': u'overcloud', u'timeout': 240}']\n ERROR: Property error: : resources.AllNodesDeploySteps.properties: : Unknown Property ctlplane_service_ips",
 u'status': u'FAILED'}


Version-Release number of selected component (if applicable):
OSP11->OSP12

openstack-neutron-common-11.0.0-0.20170628060509.9a72bfe.el7ost.noarch
openstack-neutron-openvswitch-11.0.0-0.20170628060509.9a72bfe.el7ost.noarch
openstack-nova-compute-16.0.0-0.20170627215041.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170628002128.el7ost.noarch
openstack-mistral-common-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
openstack-swift-object-2.14.1-0.20170622024006.2d18ecd.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170627215041.el7ost.noarch
openstack-heat-api-9.0.0-0.20170628061551.30c87ba.el7ost.noarch
openstack-keystone-12.0.0-0.20170628062203.e4c0c8d.el7ost.noarch
openstack-tripleo-image-elements-7.0.0-0.20170607161959.401d861.el7ost.noarch
openstack-puppet-modules-10.0.0-1.el7ost.noarch
openstack-heat-common-9.0.0-0.20170628061551.30c87ba.el7ost.noarch
openstack-nova-common-16.0.0-0.20170627215041.el7ost.noarch
openstack-swift-container-2.14.1-0.20170622024006.2d18ecd.el7ost.noarch
openstack-tempest-16.0.1-0.20170628065146.f6288d7.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170627215041.el7ost.noarch
openstack-glance-15.0.0-0.20170623215940.8188eca.el7ost.noarch
openstack-mistral-api-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
openstack-zaqar-5.0.0-0.20170626084003.b1cd63c.el7ost.noarch
puppet-openstack_extras-11.2.0-0.20170626060428.de67a39.el7ost.noarch
openstack-tripleo-common-containers-7.2.1-0.20170628001837.5a6423b.el7ost.noarch
openstack-selinux-0.8.8-0.20170622195307.74ddc0e.el7ost.noarch
openstack-swift-proxy-2.14.1-0.20170622024006.2d18ecd.el7ost.noarch
openstack-neutron-ml2-11.0.0-0.20170628060509.9a72bfe.el7ost.noarch
openstack-ironic-conductor-8.0.1-0.20170628015314.3521203.el7ost.noarch
openstack-nova-api-16.0.0-0.20170627215041.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170628061551.30c87ba.el7ost.noarch
openstack-mistral-engine-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
openstack-tripleo-validations-7.1.1-0.20170627224206.591a65f.el7ost.noarch
openstack-tripleo-ui-7.1.1-0.20170627161345.68e8684.el7ost.noarch
puppet-openstacklib-11.2.0-0.20170626053453.8264861.el7ost.noarch
python-openstackclient-3.11.0-0.20170613232431.c69304e.el7ost.noarch
python-openstack-mistral-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
openstack-ironic-common-8.0.1-0.20170628015314.3521203.el7ost.noarch
openstack-swift-account-2.14.1-0.20170622024006.2d18ecd.el7ost.noarch
openstack-ironic-api-8.0.1-0.20170628015314.3521203.el7ost.noarch
openstack-heat-api-cfn-9.0.0-0.20170628061551.30c87ba.el7ost.noarch
openstack-mistral-executor-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
python-openstacksdk-0.9.17-0.20170621195806.7946243.el7ost.noarch
openstack-neutron-11.0.0-0.20170628060509.9a72bfe.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170627215041.el7ost.noarch
openstack-ironic-inspector-5.1.1-0.20170627095652.c172b2e.el7ost.noarch
openstack-tripleo-common-7.2.1-0.20170628001837.5a6423b.el7ost.noarch
openstack-tripleo-puppet-elements-7.0.0-0.20170614005502.9285877.el7ost.noarch


How reproducible:


Steps to Reproduce:
1.Install OSP11
infrared virsh -v --host-address 10.9.76.22 --host-key ~/.ssh/id_rsa --cleanup yes && iinfrared virsh -v --host-address 10.9.76.22 --host-key ~/.ssh/id_rsa  --topology-nodes undercloud:1,controller:3,compute:1  -e  override.controller.cpu=8 -e override.controller.memory=16384 -e  override.undercloud.disks.disk1.size=100G --image-url http://download-node-02.eng.bos.redhat.com/brewroot/packages/rhel-guest-image/7.4/176/images/rhel-guest-image-7.4-176.x86_64.qcow2 
 &&infrared tripleo-undercloud --version 11 --images-task=rpm --build 7.4-testing && infrared tripleo-overcloud -v --introspect yes --tagging yes --post no --deployment-files virt --version 11 --deploy yes --build 7.4-testing  


2.Upgrade undercloud and overcloud to osp11 latest+ rhel7.4
ir tripleo-undercloud -v --update-undercloud yes --mirror qeos --build 7.4-testing --osrelease 7.4 

ir tripleo-overcloud -v --updateto 7.4-testing --deployment-files virt --mirror qeos --osrelease 7.4


3. Upgrade undercloud and overcloud to osp12
http://etherpad.corp.redhat.com/osp12-upgrade
#please note, before updating overcloud apply w/a for https://bugzilla.redhat.com/show_bug.cgi?id=1456562

Actual results:

overcloud upgrade failed
Expected results:
overcloud upgrade succeed

Additional info:

Comment 1 Red Hat Bugzilla Rules Engine 2017-07-04 14:03:07 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 2 Artem Hrechanychenko 2017-07-04 14:18:43 UTC
fix for upstream didn't helped

http://pastebin.test.redhat.com/500235

Comment 3 mathieu bultel 2017-07-10 13:37:44 UTC
I was not able to reproduce it anymore, is it this failing for you ?

Comment 4 Mike Orazi 2017-07-25 14:54:03 UTC
Since the needinfo has gone unanswered for several weeks, I"m going to assume this is no longer presenting itself in tests.

If this problem still exists, please re-open.

Comment 5 Artem Hrechanychenko 2017-07-31 11:28:05 UTC
Yep, didn't reproduced in last puddles