Bug 1477938

Summary: overcloud update fails when scaling compute nodes
Product: Red Hat OpenStack Reporter: Karim Latouche <klatouch>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED NOTABUG QA Contact: Gurenko Alex <agurenko>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 11.0 (Ocata)CC: ccamacho, klatouch, lbezdick, mbultel, mburns, rhel-osp-director-maint
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-06 13:39:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
templates folder used for this deployment none

Description Karim Latouche 2017-08-03 09:36:07 UTC
Description of problem:

When updating overcloud I get a failure at step 5

overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.25:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 836dcd59-adf5-4f08-9e0a-381729eb8bfd
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
  deploy_stdout: |
    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
    Notice: Compiled catalog for lmck-compute025.france-bioinformatique.fr in environment production in 6.19 seconds
    Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_compute5]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/bridge_mappings]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/l2_population]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/arp_responder]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/enable_distributed_routing]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/drop_flows_on_start]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/extensions]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/integration_bridge]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[securitygroup/firewall_driver]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/tunnel_bridge]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[ovs/local_ip]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/tunnel_types]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron_agent_ovs[agent/vxlan_udp_port]/ensure: created
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[ovs-cleanup-service]/enable: enable changed 'false' to 'true'
    Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[ip_conntrack_proto_sctp]/Exec[modprobe ip_conntrack_proto_sctp]/returns: executed successfully
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::config::end]: Triggered 'refresh' from 12 events
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::begin]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron::Plugins::Ovs::Bridge[datacentre:br-ex]/Vs_bridge[br-ex]/external_ids: external_ids changed '' to 'bridge-id=br-ex'
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[neutron-ovs-agent-service]/ensure: ensure changed 'stopped' to 'running'
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::end]: Triggered 'refresh' from 1 events
    Notice: Applied catalog in 14.43 seconds
  deploy_stderr: |
    exception: connect failed
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-e
xpressions. at ["/var/lib/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-ex
pressions. at ["/var/lib/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: ModuleLoader: module 'ceph' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/var/li
b/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 32]:["/etc/puppet/modules/tripleo/manifests/profile/base/ceph/client.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/var/li
b/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 34]:
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at 
["/var/lib/heat-config/heat-config-puppet/dfa5cee1-8abc-4dbb-bed2-9b460bc8509a.pp", 35]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29]
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
[Errno 32] Broken pipe


Version-Release number of selected component (if applicable):

OSP11

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Lukas Bezdicka 2017-08-07 12:47:22 UTC
I need to know what command was called and full error output. Ideally I would like to get sosreports from the nodes. This looks like scale which is not upgrades.

Comment 3 Karim Latouche 2017-08-07 16:53:33 UTC
Created attachment 1310182 [details]
templates folder used for this deployment

Comment 4 Karim Latouche 2017-08-07 16:58:22 UTC
Here is the command I used. 

exec openstack overcloud deploy \
        --templates /usr/share/openstack-tripleo-heat-templates \
        --ntp-server ntp.client.blahblah \
        -t 420 \
        --control-flavor control --control-scale 3 \
        -e /home/stack/templates/count-and-flavor.yaml \
        -r ~/templates/custom_roles.yaml \
        -e ~/templates/first-boot-env.yaml \
        -e /home/stack/templates/network-environment.yaml \
        -e /home/stack/templates/network-isolation.yaml \
        -e /home/stack/templates/storage-environment.yaml

When rerunning it, it worked and I was not able to reproduce the bug so far.

Comment 5 Carlos Camacho 2018-09-06 13:39:23 UTC
Hi, 

As the reporter is not able to reproduce this again, I'll close this, please feel free to reopen if you can reproduce it.