Hide Forgot
Created attachment 1345982 [details] initial installation templates Description of problem: The overcloud upgrade step is failing while using the infrared tool: infrared tripleo-upgrade --overcloud-upgrade yes --upgrade-workarounds yes -e @workarounds.yaml Version-Release number of selected component (if applicable): 2017-10-30.1 How reproducible: use the attached initial installation templates. Steps to Reproduce: 1. install a clean osp11 dpdk ctlplane-bonding env 2. curl -O http://file.brq.redhat.com/~mcornea/tripleo/osp12/workarounds.yaml 3. infrared tripleo-undercloud --upgrade yes --mirror ${mirror_location} --ansible-args="tags=upgrade_repos" 4. infrared tripleo-upgrade --undercloud-upgrade yes --upgrade-workarounds yes -e @workarounds.yaml 5. infrared tripleo-overcloud --deployment-files virt --upgrade yes --mirror ${mirror_location} --ansible-args="tags=upgrade_collect_info,upgrade_repos" 6. infrared tripleo-upgrade --overcloud-upgrade yes --upgrade-workarounds yes -e @workarounds.yaml Actual results: /home/stack/overcloud_composable_upgrade.log Waiting for messages on queue '85e2d4e1-a515-4132-acda-f27c95f9676a' with no timeout. {u'execution': {u'created_at': u'2017-10-31 13:46:18', u'id': u'e6e1c835-7e25-466b-867b-f14b4f6c9025', u'input': {u'container': u'overcloud', u'queue_name': u'c860148a-6929-4140-94e7-eba3e8b64a32', u'run_validations': False, u'skip_deploy_identifier': False, u'timeout': 240}, u'name': u'tripleo.deployment.v1.deploy_plan', u'params': {u'namespace': u''}, u'spec': {u'description': u'Deploy the overcloud for a plan.\n', u'input': [u'container', {u'run_validations': False}, {u'timeout': 240}, {u'skip_deploy_identifier': False}, {u'queue_name': u'tripleo'}], u'name': u'deploy_plan', u'tags': [u'tripleo-common-managed'], u'tasks': {u'add_validation_ssh_key': {u'input': {u'container': u'<% $.container %>', u'queue_name': u'<% $.queue_name %>'}, u'name': u'add_validation_ssh_key', u'on-complete': [{u'run_validations': u'<% $.run_validations %>'}, {u'create_swift_rings_backup_plan': u'<% not $.run_validations %>'}], u'type': u'direct', u'version': u'2.0', u'workflow': u'tripleo.validations.v1.add_validation_ssh_key_parameter'}, u'create_swift_rings_backup_plan': {u'input': {u'container': u'<% $.container %>', u'queue_name': u'<% $.queue_name %>', u'use_default_templates': True}, u'name': u'create_swift_rings_backup_plan', u'on-error': u'create_swift_rings_backup_plan_set_status_failed', u'on-success': u'get_heat_stack', u'type': u'direct', u'version': u'2.0', u'workflow': u'tripleo.swift_rings_backup.v1.create_swift_rings_backup_container_plan'}, u'create_swift_rings_backup_plan_set_status_failed': {u'name': u'create_swift_rings_backup_plan_set_status_failed', u'on-success': u'send_message', u'publish': {u'message': u'<% task(create_swift_rings_backup_plan).result %>', u'status': u'FAILED'}, u'type': u'direct', u'version': u'2.0'}, u'deploy': {u'action': u'tripleo.deployment.deploy', u'input': {u'container': u'<% $.container %>', u'skip_deploy_identifier': u'<% $.skip_deploy_identifier %>', u'timeout': u'<% $.timeout %>'}, u'name': u'deploy', u'on-error': u'set_deployment_failed', u'on-success': u'send_message', u'type': u'direct', u'version': u'2.0'}, u'get_heat_stack': {u'action': u'heat.stacks_get stack_id=<% $.container %>', u'name': u'get_heat_stack', u'on-error': u'deploy', u'on-success': [{u'set_stack_in_progress': u'<% "_IN_PROGRESS" in task(get_heat_stack).result.stack_status %>'}, {u'deploy': u'<% not "_IN_PROGRESS" in task(get_heat_stack).result.stack_status %>'}], u'type': u'direct', u'version': u'2.0'}, u'run_validations': {u'input': {u'group_names': [u'pre-deployment'], u'plan': u'<% $.container %>', u'queue_name': u'<% $.queue_name %>'}, u'name': u'run_validations', u'on-error': u'set_validations_failed', u'on-success': u'create_swift_rings_backup_plan', u'type': u'direct', u'version': u'2.0', u'workflow': u'tripleo.validations.v1.run_groups'}, u'send_message': {u'action': u'zaqar.queue_post', u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>', u'message': u"<% $.get('message', '') %>", u'status': u"<% $.get('status', 'SUCCESS') %>"}, u'type': u'tripleo.deployment.v1.deploy_plan'}}, u'queue_name': u'<% $.queue_name %>'}, u'name': u'send_message', u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}], u'retry': u'count=5 delay=1', u'type': u'direct', u'version': u'2.0'}, u'set_deployment_failed': {u'name': u'set_deployment_failed', u'on-success': u'send_message', u'publish': {u'message': u'<% task(deploy).result %>', u'status': u'FAILED'}, u'type': u'direct', u'version': u'2.0'}, u'set_stack_in_progress': {u'name': u'set_stack_in_progress', u'on-success': u'send_message', u'publish': {u'message': u'The Heat stack is busy.', u'status': u'FAILED'}, u'type': u'direct', u'version': u'2.0'}, u'set_validations_failed': {u'name': u'set_validations_failed', u'on-success': u'send_message', u'publish': {u'message': u'<% task(run_validations).result %>', u'status': u'FAILED'}, u'type': u'direct', u'version': u'2.0'}}, u'version': u'2.0'}}, u'message': u"Failed to run action [action_ex_id=006d5cb1-db43-4b4d-8101-9f55f8850557, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'skip_deploy_identifier': False, u'container': u'overcloud', u'timeout': 240}']\n ERROR: resources.Networks<http://192.0.20.1:8080/v1/AUTH_ecb57f4c91ff488ab8677af7358a18c0/overcloud/network/networks.yaml>: : The Resource Type (OS::TripleO::Network::Management) could not be found.", u'status': u'FAILED'} Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: 2de73f2e-ea12-49ad-a4db-f0ae2aa9fa17 Removing the current plan files Uploading new plan files Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: adb485a9-65b6-4ed9-a535-8c9b86a21866 Plan updated. Processing templates in the directory /tmp/tripleoclient-0HN2bl/tripleo-heat-templates Started Mistral Workflow tripleo.plan_management.v1.get_deprecated_parameters. Execution ID: 3cb99b23-7990-4981-9dbc-d4b29af949e5 Deploying templates in the directory /tmp/tripleoclient-0HN2bl/tripleo-heat-templates Started Mistral Workflow tripleo.deployment.v1.deploy_plan. Execution ID: e6e1c835-7e25-466b-867b-f14b4f6c9025 Expected results: the upgrade should finish successfully
This bug, is a bit misleading, it is not related to InfraRed, it is related to TripleO update/upgrade
Deploy command: [stack@undercloud-0 ~]$ cat overcloud_deploy.sh #!/bin/bash openstack overcloud deploy \ --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovs-dpdk.yaml \ -e /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/network-environment.yaml \ --log-file overcloud_install.log &> overcloud_install.log Upgrade command: [stack@undercloud-0 ~]$ cat composable_docker_upgrade.sh #!/bin/bash openstack overcloud deploy \ --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovs-dpdk.yaml \ -e /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \ -e /home/stack/docker-osp12.yaml \ network-environment.yaml: [stack@undercloud-0 ~]$ cat /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/network-environment.yaml resource_registry: # Specify the relative/absolute path to the config files you want to use for override the default. OS::TripleO::Compute::Net::SoftwareConfig: nic-configs/compute.yaml OS::TripleO::Controller::Net::SoftwareConfig: nic-configs/controller.yaml OS::TripleO::NodeUserData: first-boot.yaml OS::TripleO::NodeExtraConfigPost: post-install.yaml # Network isolation configuration # Service section OS::TripleO::Network::External: /usr/share/openstack-tripleo-heat-templates/network/external.yaml OS::TripleO::Network::InternalApi: /usr/share/openstack-tripleo-heat-templates/network/internal_api.yaml OS::TripleO::Network::Tenant: /usr/share/openstack-tripleo-heat-templates/network/tenant.yaml OS::TripleO::Network::Management: OS::Heat::None OS::TripleO::Network::StorageMgmt: OS::Heat::None OS::TripleO::Network::Storage: OS::Heat::None # Port assignments for the VIPs OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external.yaml OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/vip.yaml OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml # Port assignments for the controller role OS::TripleO::Controller::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external.yaml OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml OS::TripleO::Controller::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant.yaml OS::TripleO::Controller::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Controller::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Controller::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml # Port assignments for the compute role OS::TripleO::Compute::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external.yaml OS::TripleO::Compute::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml OS::TripleO::Compute::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant.yaml OS::TripleO::Compute::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Compute::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Compute::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml # Port assignments for service virtual IPs for the controller role OS::TripleO::Controller::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/vip.yaml parameter_defaults: # MTU global configuration #NeutronGlobalPhysnetMtu: 2000 # Customize all these values to match the local environment InternalApiNetCidr: 10.10.125.0/24 TenantNetCidr: 10.10.126.0/24 ExternalNetCidr: 10.35.185.0/28 # CIDR subnet mask length for provisioning network ControlPlaneSubnetCidr: '24' InternalApiAllocationPools: [{'start': '10.10.125.10', 'end': '10.10.125.200'}] TenantAllocationPools: [{'start': '10.10.126.10', 'end': '10.10.126.200'}] # Use an External allocation pool which will leave room for floating IPs ExternalAllocationPools: [{'start': '10.35.185.2', 'end': '10.35.185.13'}] # Set to the router gateway on the external network ExternalInterfaceDefaultRoute: 10.35.185.14 # Gateway router for the provisioning network (or Undercloud IP) ControlPlaneDefaultRoute: 192.0.20.1 # Generally the IP of the Undercloud EC2MetadataIp: 192.0.20.1 InternalApiNetworkVlanID: 525 TenantNetworkVlanID: 526 ExternalNetworkVlanID: 418 # Define the DNS servers (maximum 2) for the overcloud nodes DnsServers: ["10.35.28.28","8.8.8.8"] # May set to br-ex if using floating IPs only on native VLAN on bridge br-ex NeutronExternalNetworkBridge: "''" # The tunnel type for the tenant network (vxlan or gre). Set to '' to disable tunneling. NeutronTunnelTypes: '' # The tenant network type for Neutron (vlan or vxlan). NeutronNetworkType: 'vlan' # The OVS logical->physical bridge mappings to use. NeutronBridgeMappings: 'tenant:br-link' # The Neutron ML2 and OpenVSwitch vlan mapping range to support. NeutronNetworkVLANRanges: 'tenant:420:420,tenant:421:421,tenant:527:528' # Nova flavor to use. OvercloudControlFlavor: controller OvercloudComputeFlavor: compute #Number of nodes to deploy. ControllerCount: 3 ComputeCount: 1 # NTP server configuration. NtpServer: clock.redhat.com # Configure the classname of the firewall driver to use for implementing security groups. NeutronOVSFirewallDriver: openvswitch # DHCP provide metadata route to VM NeutronEnableIsolatedMetadata: True # DHCP always provides metadata route to VM. NeutronEnableForceMetadata: True # Sets overcloud nodes custom names # http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/node_placement.html#custom-hostnames ControllerHostnameFormat: 'controller-%index%' ComputeHostnameFormat: 'compute-%index%' CephStorageHostnameFormat: 'ceph-%index%' ObjectStorageHostnameFormat: 'swift-%index%' ########################## # OVS DPDK configuration # ########################## ## NeutronDpdkCoreList and NeutronDpdkMemoryChannels are REQUIRED settings. ## Attempting to deploy DPDK without appropriate values will cause deployment to fail or lead to unstable deployments. # List of cores to be used for DPDK Poll Mode Driver NeutronDpdkCoreList: "4,6,20,22" # Number of memory channels to be used for DPDK NeutronDpdkMemoryChannels: "4" # NeutronDpdkSocketMemory NeutronDpdkSocketMemory: "1024" # NeutronDpdkDriverType NeutronDpdkDriverType: "vfio-pci" # The vhost-user socket directory for OVS NeutronVhostuserSocketDir: "/var/lib/vhost_sockets" ######################## # Additional settings # ######################## # Reserved RAM for host processes NovaReservedHostMemory: 2048 # A list or range of physical CPU cores to reserve for virtual machine processes. # Example: NovaVcpuPinSet: ['4-12','^8'] will reserve cores from 4-12 excluding 8 NovaVcpuPinSet: "1,2,3,5,7,17,18,19,21,23" # An array of filters used by Nova to filter a node.These filters will be applied in the order they are listed, # so place your most restrictive filters first to make the filtering process more efficient. NovaSchedulerDefaultFilters: "RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,NUMATopologyFilter" # Kernel arguments for Compute node ComputeKernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on" # A list or range of physical CPU cores to be tuned. # The given args will be appended to the tuned cpu-partitioning profile. HostIsolatedCoreList: "1,2,3,4,5,16,17,18,19,20,21" # List of logical cores to be used by ovs-dpdk processess (dpdk-lcore-mask) HostCpusList: "6,7,22,23"
Looks like the issue is: shardy ╡ mcornea: I think the code in tripleoclient which calculates the absolute path is doing the wrong thing when the resource_registry points to another resource instead of a template path: [stack@undercloud-0 ~]$ cat /home/stack/shdebug/saved_plan/user-environment.yaml resource_registry: OS::TripleO::Compute::Net::SoftwareConfig: user-files/home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/nic-configs/compute.yaml OS::TripleO::Compute::Ports::ExternalPort: network/ports/external.yaml OS::TripleO::Compute::Ports::InternalApiPort: network/ports/internal_api.yaml OS::TripleO::Compute::Ports::ManagementPort: network/ports/noop.yaml OS::TripleO::Compute::Ports::StorageMgmtPort: network/ports/noop.yaml OS::TripleO::Compute::Ports::StoragePort: network/ports/noop.yaml OS::TripleO::Compute::Ports::TenantPort: network/ports/tenant.yaml OS::TripleO::Controller::Net::SoftwareConfig: user-files/home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/nic-configs/controller.yaml OS::TripleO::Controller::Ports::ExternalPort: network/ports/external.yaml OS::TripleO::Controller::Ports::InternalApiPort: network/ports/internal_api.yaml OS::TripleO::Controller::Ports::ManagementPort: network/ports/noop.yaml OS::TripleO::Controller::Ports::RedisVipPort: network/ports/vip.yaml OS::TripleO::Controller::Ports::StorageMgmtPort: network/ports/noop.yaml OS::TripleO::Controller::Ports::StoragePort: network/ports/noop.yaml OS::TripleO::Controller::Ports::TenantPort: network/ports/tenant.yaml OS::TripleO::Network::External: network/external.yaml OS::TripleO::Network::InternalApi: network/internal_api.yaml OS::TripleO::Network::Management: /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/OS::Heat::None OS::TripleO::Network::Ports::ExternalVipPort: network/ports/external.yaml OS::TripleO::Network::Ports::InternalApiVipPort: network/ports/internal_api.yaml OS::TripleO::Network::Ports::RedisVipPort: network/ports/vip.yaml OS::TripleO::Network::Ports::StorageMgmtVipPort: network/ports/noop.yaml OS::TripleO::Network::Ports::StorageVipPort: network/ports/noop.yaml OS::TripleO::Network::Storage: /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/OS::Heat::None OS::TripleO::Network::StorageMgmt: /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/OS::Heat::None OS::TripleO::Network::Tenant: network/tenant.yaml OS::TripleO::NodeExtraConfigPost: user-files/home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/post-install.yaml OS::TripleO::NodeUserData: user-files/home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/first-boot.yaml OS::TripleO::PostDeploySteps: common/major_upgrade_steps.yaml OS::TripleO::Services::ComputeNeutronOvsDpdk: puppet/services/neutron-ovs-dpdk-agent.yaml resources: '*': '*': UpdateDeployment: hooks: []
Workaround: In /home/stack/ospd-11-vlan-dpdk-single-port-ctlplane-bonding/network-environment.yaml comment out: # OS::TripleO::Network::Management: OS::Heat::None # OS::TripleO::Network::StorageMgmt: OS::Heat::None # OS::TripleO::Network::Storage: OS::Heat::None
I reproduced this in an environment provided by Marius, root cause described in https://bugs.launchpad.net/tripleo/+bug/1729039, working on a fix
Thank you Steve, I put the untriaged flag since its already on dev.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462