Bug 1477938 - overcloud update fails when scaling compute nodes
overcloud update fails when scaling compute nodes
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
11.0 (Ocata)
x86_64 Linux
urgent Severity urgent
: ---
: ---
Assigned To: Emilien Macchi
Gurenko Alex
: Triaged, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-03 05:36 EDT by Karim Latouche
Modified: 2017-08-28 09:14 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
templates folder used for this deployment (11.68 KB, application/x-gzip)
2017-08-07 12:53 EDT, Karim Latouche
no flags Details

  None (edit)
Description Karim Latouche 2017-08-03 05:36:07 EDT
Description of problem:

When updating overcloud I get a failure at step 5

overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.25:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 836dcd59-adf5-4f08-9e0a-381729eb8bfd
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
  deploy_stdout: |
    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
    Notice: Compiled catalog for lmck-compute025.france-bioinformatique.fr in environment production in 6.19 seconds
    Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_compute5]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/bridge_mappings]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/l2_population]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/arp_responder]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/enable_distributed_routing]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/drop_flows_on_start]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/extensions]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/integration_bridge]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[securitygroup/firewall_driver]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/tunnel_bridge]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/local_ip]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/tunnel_types]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/vxlan_udp_port]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[ovs-cleanup-service]/enable: enable changed 'false' to 'true'
    Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[ip_conntrack_proto_sctp]/Exec[modprobe ip_conntrack_proto_sctp]/returns: executed successfully
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::config::end]: Triggered 'refresh' from 12 events
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::begin]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron::Plugins::Ovs::Bridge[datacentre:br-ex]/Vs_bridge[br-ex]/external_ids: external_ids changed '' to 'bridge-id=br-ex'
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[neutron-ovs-agent-service]/ensure: ensure changed 'stopped' to 'running'
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::end]: Triggered 'refresh' from 1 events
    Notice: Applied catalog in 14.43 seconds
  deploy_stderr: |
    exception: connect failed
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-e
xpressions. at ["/var/lib/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-ex
pressions. at ["/var/lib/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: ModuleLoader: module 'ceph' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/var/li
b/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/var/li
b/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 34]:
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at 
["/var/lib/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
[Errno 32] Broken pipe


Version-Release number of selected component (if applicable):

OSP11

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Lukas Bezdicka 2017-08-07 08:47:22 EDT
I need to know what command was called and full error output. Ideally I would like to get sosreports from the nodes. This looks like scale which is not upgrades.
Comment 3 Karim Latouche 2017-08-07 12:53 EDT
Created attachment 1310182 [details]
templates folder used for this deployment
Comment 4 Karim Latouche 2017-08-07 12:58:22 EDT
Here is the command I used. 

exec openstack overcloud deploy \
        --templates /usr/share/openstack-tripleo-heat-templates \
        --ntp-server ntp.client.blahblah \
        -t 420 \
        --control-flavor control --control-scale 3 \
        -e /home/stack/templates/count-and-flavor.yaml \
        -r ~/templates/custom_roles.yaml \
        -e ~/templates/first-boot-env.yaml \
        -e /home/stack/templates/network-environment.yaml \
        -e /home/stack/templates/network-isolation.yaml \
        -e /home/stack/templates/storage-environment.yaml

When rerunning it, it worked and I was not able to reproduce the bug so far.

Note You need to log in before you can comment on or make changes to this bug.