Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1507976

Summary: OSP11 -> OSP12 upgrade: overcloud upgrade step is failed to finish.
Product: Red Hat OpenStack Reporter: Ziv Greenberg <zgreenbe>
Component: python-tripleoclientAssignee: RHOS Maint <rhos-maint>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 12.0 (Pike)CC: aasmith, atelang, augol, dbecker, hbrock, jpichon, jschluet, jslagle, mandreou, mbracho, mbultel, mburns, mcornea, morazi, ohochman, rhel-osp-director-maint, rhos-maint, sathlang, shardy, yrachman
Target Milestone: rcKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: python-tripleoclient-7.3.3-5.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 22:19:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
initial installation templates none

Description Ziv Greenberg 2017-10-31 15:35:17 UTC 
Created attachment 1345982 [details]
initial installation templates

Description of problem:

The overcloud upgrade step is failing while using the infrared tool:
infrared tripleo-upgrade --overcloud-upgrade yes --upgrade-workarounds yes -e @workarounds.yaml


Version-Release number of selected component (if applicable):
2017-10-30.1


How reproducible:
use the attached initial installation templates.

Steps to Reproduce:
1. install a clean osp11 dpdk ctlplane-bonding env
2. curl -O http://file.brq.redhat.com/~mcornea/tripleo/osp12/workarounds.yaml
3. infrared tripleo-undercloud --upgrade yes --mirror ${mirror_location} --ansible-args="tags=upgrade_repos"
4. infrared tripleo-upgrade --undercloud-upgrade yes --upgrade-workarounds yes -e @workarounds.yaml
5. infrared tripleo-overcloud --deployment-files virt --upgrade yes --mirror ${mirror_location} --ansible-args="tags=upgrade_collect_info,upgrade_repos"
6. infrared tripleo-upgrade --overcloud-upgrade yes --upgrade-workarounds yes -e @workarounds.yaml

Actual results:

/home/stack/overcloud_composable_upgrade.log

Waiting for messages on queue '85e2d4e1-a515-4132-acda-f27c95f9676a' with no timeout.
{u'execution': {u'created_at': u'2017-10-31 13:46:18',
                u'id': u'e6e1c835-7e25-466b-867b-f14b4f6c9025',
                u'input': {u'container': u'overcloud',
                           u'queue_name': u'c860148a-6929-4140-94e7-eba3e8b64a32',
                           u'run_validations': False,
                           u'skip_deploy_identifier': False,
                           u'timeout': 240},
                u'name': u'tripleo.deployment.v1.deploy_plan',
                u'params': {u'namespace': u''},
                u'spec': {u'description': u'Deploy the overcloud for a plan.\n',
                          u'input': [u'container',
                                     {u'run_validations': False},
                                     {u'timeout': 240},
                                     {u'skip_deploy_identifier': False},
                                     {u'queue_name': u'tripleo'}],
                          u'name': u'deploy_plan',
                          u'tags': [u'tripleo-common-managed'],
                          u'tasks': {u'add_validation_ssh_key': {u'input': {u'container': u'<% $.container %>',
                                                                            u'queue_name': u'<% $.queue_name %>'},
                                                                 u'name': u'add_validation_ssh_key',
                                                                 u'on-complete': [{u'run_validations': u'<% $.run_validations %>'},
                                                                                  {u'create_swift_rings_backup_plan': u'<% not $.run_validations %>'}],
                                                                 u'type': u'direct',
                                                                 u'version': u'2.0',
                                                                 u'workflow': u'tripleo.validations.v1.add_validation_ssh_key_parameter'},
                                     u'create_swift_rings_backup_plan': {u'input': {u'container': u'<% $.container %>',
                                                                                    u'queue_name': u'<% $.queue_name %>',
                                                                                    u'use_default_templates': True},
                                                                         u'name': u'create_swift_rings_backup_plan',
                                                                         u'on-error': u'create_swift_rings_backup_plan_set_status_failed',
                                                                         u'on-success': u'get_heat_stack',
                                                                         u'type': u'direct',
                                                                         u'version': u'2.0',
                                                                         u'workflow': u'tripleo.swift_rings_backup.v1.create_swift_rings_backup_container_plan'},
                                     u'create_swift_rings_backup_plan_set_status_failed': {u'name': u'create_swift_rings_backup_plan_set_status_failed',
                                                                                           u'on-success': u'send_message',
                                                                                           u'publish': {u'message': u'<% task(create_swift_rings_backup_plan).result %>',
                                                                                                        u'status': u'FAILED'},
                                                                                           u'type': u'direct',
                                                                                           u'version': u'2.0'},
                                     u'deploy': {u'action': u'tripleo.deployment.deploy',
                                                 u'input': {u'container': u'<% $.container %>',
                                                            u'skip_deploy_identifier': u'<% $.skip_deploy_identifier %>',
                                                            u'timeout': u'<% $.timeout %>'},
                                                 u'name': u'deploy',
                                                 u'on-error': u'set_deployment_failed',
                                                 u'on-success': u'send_message',
                                                 u'type': u'direct',
                                                 u'version': u'2.0'},
                                     u'get_heat_stack': {u'action': u'heat.stacks_get stack_id=<% $.container %>',
                                                         u'name': u'get_heat_stack',
                                                         u'on-error': u'deploy',
                                                         u'on-success': [{u'set_stack_in_progress': u'<% "_IN_PROGRESS" in task(get_heat_stack).result.stack_status %>'},
                                                                         {u'deploy': u'<% not "_IN_PROGRESS" in task(get_heat_stack).result.stack_status %>'}],
                                                         u'type': u'direct',
                                                         u'version': u'2.0'},
                                     u'run_validations': {u'input': {u'group_names': [u'pre-deployment'],
                                                                     u'plan': u'<% $.container %>',
                                                                     u'queue_name': u'<% $.queue_name %>'},
                                                          u'name': u'run_validations',
                                                          u'on-error': u'set_validations_failed',
                                                          u'on-success': u'create_swift_rings_backup_plan',
                                                          u'type': u'direct',
                                                          u'version': u'2.0',
                                                          u'workflow': u'tripleo.validations.v1.run_groups'},
                                     u'send_message': {u'action': u'zaqar.queue_post',
                                                       u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
                                                                                                       u'message': u"<% $.get('message', '') %>",
                                                                                                       u'status': u"<% $.get('status', 'SUCCESS') %>"},
                                                                                          u'type': u'tripleo.deployment.v1.deploy_plan'}},
                                                                  u'queue_name': u'<% $.queue_name %>'},
                                                       u'name': u'send_message',
                                                       u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}],
                                                       u'retry': u'count=5 delay=1',
                                                       u'type': u'direct',
                                                       u'version': u'2.0'},
                                     u'set_deployment_failed': {u'name': u'set_deployment_failed',
                                                                u'on-success': u'send_message',
                                                                u'publish': {u'message': u'<% task(deploy).result %>',
                                                                             u'status': u'FAILED'},
                                                                u'type': u'direct',
                                                                u'version': u'2.0'},
                                     u'set_stack_in_progress': {u'name': u'set_stack_in_progress',
                                                                u'on-success': u'send_message',
                                                                u'publish': {u'message': u'The Heat stack is busy.',
                                                                             u'status': u'FAILED'},
                                                                u'type': u'direct',
                                                                u'version': u'2.0'},
                                     u'set_validations_failed': {u'name': u'set_validations_failed',
                                                                 u'on-success': u'send_message',
                                                                 u'publish': {u'message': u'<% task(run_validations).result %>',
                                                                              u'status': u'FAILED'},
                                                                 u'type': u'direct',
                                                                 u'version': u'2.0'}},
                          u'version': u'2.0'}},
 u'message': u"Failed to run action [action_ex_id=006d5cb1-db43-4b4d-8101-9f55f8850557, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'skip_deploy_identifier': False, u'container': u'overcloud', u'timeout': 240}']\n ERROR: resources.Networks<http://192.0.20.1:8080/v1/AUTH_ecb57f4c91ff488ab8677af7358a18c0/overcloud/network/networks.yaml>: : The Resource Type (OS::TripleO::Network::Management) could not be found.",
 u'status': u'FAILED'}
Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: 2de73f2e-ea12-49ad-a4db-f0ae2aa9fa17
Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: adb485a9-65b6-4ed9-a535-8c9b86a21866
Plan updated.
Processing templates in the directory /tmp/tripleoclient-0HN2bl/tripleo-heat-templates
Started Mistral Workflow tripleo.plan_management.v1.get_deprecated_parameters. Execution ID: 3cb99b23-7990-4981-9dbc-d4b29af949e5
Deploying templates in the directory /tmp/tripleoclient-0HN2bl/tripleo-heat-templates
Started Mistral Workflow tripleo.deployment.v1.deploy_plan. Execution ID: e6e1c835-7e25-466b-867b-f14b4f6c9025


Expected results:
the upgrade should finish successfully
Comment 1 Yariv 2017-11-01 08:04:02 UTC 
This bug, is a bit misleading, it is not related to InfraRed, it is related to TripleO update/upgrade
Comment 2 Marius Cornea 2017-11-01 14:07:19 UTC 
Deploy command:

[stack@undercloud-0 ~]$ cat overcloud_deploy.sh
#!/bin/bash

openstack overcloud deploy \
--templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovs-dpdk.yaml \
-e /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/network-environment.yaml \
--log-file overcloud_install.log &> overcloud_install.log


Upgrade command:

[stack@undercloud-0 ~]$ cat composable_docker_upgrade.sh 
#!/bin/bash

openstack overcloud deploy \
--templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovs-dpdk.yaml \
-e /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/docker-osp12.yaml \


network-environment.yaml:

[stack@undercloud-0 ~]$ cat /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/network-environment.yaml
resource_registry:
  # Specify the relative/absolute path to the config files you want to use for override the default.
  OS::TripleO::Compute::Net::SoftwareConfig: nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: nic-configs/controller.yaml
  OS::TripleO::NodeUserData: first-boot.yaml
  OS::TripleO::NodeExtraConfigPost: post-install.yaml

  # Network isolation configuration
  # Service section
  OS::TripleO::Network::External: /usr/share/openstack-tripleo-heat-templates/network/external.yaml
  OS::TripleO::Network::InternalApi: /usr/share/openstack-tripleo-heat-templates/network/internal_api.yaml
  OS::TripleO::Network::Tenant: /usr/share/openstack-tripleo-heat-templates/network/tenant.yaml
  OS::TripleO::Network::Management: OS::Heat::None
  OS::TripleO::Network::StorageMgmt: OS::Heat::None
  OS::TripleO::Network::Storage: OS::Heat::None

  # Port assignments for the VIPs
  OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/vip.yaml
  OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml

  # Port assignments for the controller role
  OS::TripleO::Controller::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external.yaml
  OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml
  OS::TripleO::Controller::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant.yaml
  OS::TripleO::Controller::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Controller::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Controller::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml

  # Port assignments for the compute role
  OS::TripleO::Compute::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external.yaml
  OS::TripleO::Compute::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml
  OS::TripleO::Compute::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant.yaml
  OS::TripleO::Compute::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Compute::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Compute::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml

  # Port assignments for service virtual IPs for the controller role
  OS::TripleO::Controller::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/vip.yaml

parameter_defaults:
  # MTU global configuration
  #NeutronGlobalPhysnetMtu: 2000
  # Customize all these values to match the local environment
  InternalApiNetCidr: 10.10.125.0/24
  TenantNetCidr: 10.10.126.0/24
  ExternalNetCidr: 10.35.185.0/28
  # CIDR subnet mask length for provisioning network
  ControlPlaneSubnetCidr: '24'
  InternalApiAllocationPools: [{'start': '10.10.125.10', 'end': '10.10.125.200'}]
  TenantAllocationPools: [{'start': '10.10.126.10', 'end': '10.10.126.200'}]
  # Use an External allocation pool which will leave room for floating IPs
  ExternalAllocationPools: [{'start': '10.35.185.2', 'end': '10.35.185.13'}]
  # Set to the router gateway on the external network
  ExternalInterfaceDefaultRoute: 10.35.185.14
  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.0.20.1
  # Generally the IP of the Undercloud
  EC2MetadataIp: 192.0.20.1
  InternalApiNetworkVlanID: 525
  TenantNetworkVlanID: 526
  ExternalNetworkVlanID: 418
  # Define the DNS servers (maximum 2) for the overcloud nodes
  DnsServers: ["10.35.28.28","8.8.8.8"]
  # May set to br-ex if using floating IPs only on native VLAN on bridge br-ex
  NeutronExternalNetworkBridge: "''"
  # The tunnel type for the tenant network (vxlan or gre). Set to '' to disable tunneling.
  NeutronTunnelTypes: ''
  # The tenant network type for Neutron (vlan or vxlan).
  NeutronNetworkType: 'vlan'
  # The OVS logical->physical bridge mappings to use.
  NeutronBridgeMappings: 'tenant:br-link'
  # The Neutron ML2 and OpenVSwitch vlan mapping range to support.
  NeutronNetworkVLANRanges: 'tenant:420:420,tenant:421:421,tenant:527:528'
  # Nova flavor to use.
  OvercloudControlFlavor: controller
  OvercloudComputeFlavor: compute
  #Number of nodes to deploy.
  ControllerCount: 3
  ComputeCount: 1
  # NTP server configuration.
  NtpServer: clock.redhat.com
  # Configure the classname of the firewall driver to use for implementing security groups.
  NeutronOVSFirewallDriver: openvswitch
  # DHCP provide metadata route to VM
  NeutronEnableIsolatedMetadata: True
  # DHCP always provides metadata route to VM.
  NeutronEnableForceMetadata: True

  # Sets overcloud nodes custom names
  # http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/node_placement.html#custom-hostnames
  ControllerHostnameFormat: 'controller-%index%'
  ComputeHostnameFormat: 'compute-%index%'
  CephStorageHostnameFormat: 'ceph-%index%'
  ObjectStorageHostnameFormat: 'swift-%index%'

  ##########################
  # OVS DPDK configuration #
  ##########################
  ## NeutronDpdkCoreList and NeutronDpdkMemoryChannels are REQUIRED settings.
  ## Attempting to deploy DPDK without appropriate values will cause deployment to fail or lead to unstable deployments.
  # List of cores to be used for DPDK Poll Mode Driver
  NeutronDpdkCoreList: "4,6,20,22"
  # Number of memory channels to be used for DPDK
  NeutronDpdkMemoryChannels: "4"
  # NeutronDpdkSocketMemory
  NeutronDpdkSocketMemory: "1024"
  # NeutronDpdkDriverType
  NeutronDpdkDriverType: "vfio-pci"
  # The vhost-user socket directory for OVS
  NeutronVhostuserSocketDir: "/var/lib/vhost_sockets"

  ########################
  # Additional settings  #
  ########################
  # Reserved RAM for host processes
  NovaReservedHostMemory: 2048
  # A list or range of physical CPU cores to reserve for virtual machine processes.
  # Example: NovaVcpuPinSet: ['4-12','^8'] will reserve cores from 4-12 excluding 8
  NovaVcpuPinSet: "1,2,3,5,7,17,18,19,21,23"
  # An array of filters used by Nova to filter a node.These filters will be applied in the order they are listed,
  # so place your most restrictive filters first to make the filtering process more efficient.
  NovaSchedulerDefaultFilters: "RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,NUMATopologyFilter"
  # Kernel arguments for Compute node
  ComputeKernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on"
  # A list or range of physical CPU cores to be tuned.
  # The given args will be appended to the tuned cpu-partitioning profile.
  HostIsolatedCoreList: "1,2,3,4,5,16,17,18,19,20,21"
  # List of logical cores to be used by ovs-dpdk processess (dpdk-lcore-mask)
  HostCpusList: "6,7,22,23"
Comment 3 Marius Cornea 2017-11-01 14:08:53 UTC 
Looks like the issue is:

shardy ╡ mcornea: I think the code in tripleoclient which calculates the absolute path is doing the wrong thing when the resource_registry points to another resource instead of a template path:


[stack@undercloud-0 ~]$ cat /home/stack/shdebug/saved_plan/user-environment.yaml
resource_registry:
  OS::TripleO::Compute::Net::SoftwareConfig: user-files/home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/nic-configs/compute.yaml
  OS::TripleO::Compute::Ports::ExternalPort: network/ports/external.yaml
  OS::TripleO::Compute::Ports::InternalApiPort: network/ports/internal_api.yaml
  OS::TripleO::Compute::Ports::ManagementPort: network/ports/noop.yaml
  OS::TripleO::Compute::Ports::StorageMgmtPort: network/ports/noop.yaml
  OS::TripleO::Compute::Ports::StoragePort: network/ports/noop.yaml
  OS::TripleO::Compute::Ports::TenantPort: network/ports/tenant.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: user-files/home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/nic-configs/controller.yaml
  OS::TripleO::Controller::Ports::ExternalPort: network/ports/external.yaml
  OS::TripleO::Controller::Ports::InternalApiPort: network/ports/internal_api.yaml
  OS::TripleO::Controller::Ports::ManagementPort: network/ports/noop.yaml
  OS::TripleO::Controller::Ports::RedisVipPort: network/ports/vip.yaml
  OS::TripleO::Controller::Ports::StorageMgmtPort: network/ports/noop.yaml
  OS::TripleO::Controller::Ports::StoragePort: network/ports/noop.yaml
  OS::TripleO::Controller::Ports::TenantPort: network/ports/tenant.yaml
  OS::TripleO::Network::External: network/external.yaml
  OS::TripleO::Network::InternalApi: network/internal_api.yaml
  OS::TripleO::Network::Management: /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/OS::Heat::None
  OS::TripleO::Network::Ports::ExternalVipPort: network/ports/external.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: network/ports/internal_api.yaml
  OS::TripleO::Network::Ports::RedisVipPort: network/ports/vip.yaml
  OS::TripleO::Network::Ports::StorageMgmtVipPort: network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageVipPort: network/ports/noop.yaml
  OS::TripleO::Network::Storage: /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/OS::Heat::None
  OS::TripleO::Network::StorageMgmt: /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/OS::Heat::None
  OS::TripleO::Network::Tenant: network/tenant.yaml
  OS::TripleO::NodeExtraConfigPost: user-files/home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/post-install.yaml
  OS::TripleO::NodeUserData: user-files/home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/first-boot.yaml
  OS::TripleO::PostDeploySteps: common/major_upgrade_steps.yaml
  OS::TripleO::Services::ComputeNeutronOvsDpdk: puppet/services/neutron-ovs-dpdk-agent.yaml
  resources:
    '*':
      '*':
        UpdateDeployment:
          hooks: []
Comment 4 Marius Cornea 2017-11-01 14:23:08 UTC 
Workaround:

In /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/network-environment.yaml comment out:

#  OS::TripleO::Network::Management: OS::Heat::None
#  OS::TripleO::Network::StorageMgmt: OS::Heat::None
#  OS::TripleO::Network::Storage: OS::Heat::None
Comment 6 Steven Hardy 2017-11-01 16:04:28 UTC 
I reproduced this in an environment provided by Marius, root cause described in https://bugs.launchpad.net/tripleo/+bug/1729039, working on a fix
Comment 7 mathieu bultel 2017-11-02 09:23:47 UTC 
Thank you Steve,
I put the untriaged flag since its already on dev.
Comment 14 errata-xmlrpc 2017-12-13 22:19:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462