Bug 1477938 - overcloud update fails when scaling compute nodes
Summary: overcloud update fails when scaling compute nodes
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 11.0 (Ocata)
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: ---
Assignee: Emilien Macchi
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-03 09:36 UTC by Karim Latouche
Modified: 2023-02-22 23:02 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-06 13:39:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
templates folder used for this deployment (11.68 KB, application/x-gzip)
2017-08-07 16:53 UTC, Karim Latouche
no flags Details

Description Karim Latouche 2017-08-03 09:36:07 UTC
Description of problem:

When updating overcloud I get a failure at step 5

overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.25:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 836dcd59-adf5-4f08-9e0a-381729eb8bfd
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
  deploy_stdout: |
    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
    Notice: Compiled catalog for lmck-compute025.france-bioinformatique.fr in environment production in 6.19 seconds
    Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_compute5]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/bridge_mappings]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/l2_population]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/arp_responder]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/enable_distributed_routing]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/drop_flows_on_start]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/extensions]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/integration_bridge]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[securitygroup/firewall_driver]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/tunnel_bridge]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/local_ip]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/tunnel_types]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/vxlan_udp_port]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[ovs-cleanup-service]/enable: enable changed 'false' to 'true'
    Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[ip_conntrack_proto_sctp]/Exec[modprobe ip_conntrack_proto_sctp]/returns: executed successfully
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::config::end]: Triggered 'refresh' from 12 events
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::begin]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron::Plugins::Ovs::Bridge[datacentre:br-ex]/Vs_bridge[br-ex]/external_ids: external_ids changed '' to 'bridge-id=br-ex'
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[neutron-ovs-agent-service]/ensure: ensure changed 'stopped' to 'running'
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::end]: Triggered 'refresh' from 1 events
    Notice: Applied catalog in 14.43 seconds
  deploy_stderr: |
    exception: connect failed
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-e
xpressions. at ["/var/lib/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-ex
pressions. at ["/var/lib/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: ModuleLoader: module 'ceph' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/var/li
b/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/var/li
b/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 34]:
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at 
["/var/lib/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
[Errno 32] Broken pipe


Version-Release number of selected component (if applicable):

OSP11

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Lukas Bezdicka 2017-08-07 12:47:22 UTC
I need to know what command was called and full error output. Ideally I would like to get sosreports from the nodes. This looks like scale which is not upgrades.

Comment 3 Karim Latouche 2017-08-07 16:53:33 UTC
Created attachment 1310182 [details]
templates folder used for this deployment

Comment 4 Karim Latouche 2017-08-07 16:58:22 UTC
Here is the command I used. 

exec openstack overcloud deploy \
        --templates /usr/share/openstack-tripleo-heat-templates \
        --ntp-server ntp.client.blahblah \
        -t 420 \
        --control-flavor control --control-scale 3 \
        -e /home/stack/templates/count-and-flavor.yaml \
        -r ~/templates/custom_roles.yaml \
        -e ~/templates/first-boot-env.yaml \
        -e /home/stack/templates/network-environment.yaml \
        -e /home/stack/templates/network-isolation.yaml \
        -e /home/stack/templates/storage-environment.yaml

When rerunning it, it worked and I was not able to reproduce the bug so far.

Comment 5 Carlos Camacho 2018-09-06 13:39:23 UTC
Hi, 

As the reporter is not able to reproduce this again, I'll close this, please feel free to reopen if you can reproduce it.


Note You need to log in before you can comment on or make changes to this bug.