Bug 1292946 - rhel-osp-director: failed to update 7.0->7.2 :ControllerNodesPostDeployment: resources.ControllerOvercloudServicesDeployment_Step7: Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
rhel-osp-director: failed to update 7.0->7.2 :ControllerNodesPostDeployment: ...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: y3
: 7.0 (Kilo)
Assigned To: Hugh Brock
Alexander Chuzhoy
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-18 14:33 EST by Alexander Chuzhoy
Modified: 2016-02-18 11:48 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-18 11:48:04 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/messages from controller (1.18 MB, application/x-gzip)
2015-12-18 14:38 EST, Alexander Chuzhoy
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0264 normal SHIPPED_LIVE Red Hat Enterprise Linux OSP 7 director Bug Fix Advisory 2016-02-18 16:41:29 EST

  None (edit)
Description Alexander Chuzhoy 2015-12-18 14:33:06 EST
rhel-osp-director: failed to update 7.0->7.2 :ControllerNodesPostDeployment: resources.ControllerOvercloudServicesDeployment_Step7: Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6

Environment:
openstack-tripleo-heat-templates-0.8.6-94.el7ost.noarch
instack-undercloud-2.1.2-36.el7ost.noarch

Steps to reproduce:

1. Deploy 7.0 with "openstack overcloud deploy --templates --control-scale 3 --compute-scale 1  --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server 10.5.26.10 --timeout 90 -e network-environment.yaml"

2. Update the undercloud node and reboot it - verify nothing reports as failed.

3. Follow the procedure to prepare the overcloud nodes for the update and update the included environment files.

4. Run the update command: "yes "" | openstack overcloud update stack overcloud -i --templates   -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml    -e /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-keystone-admin-internal-api.yaml  -e network-environment.yaml


Result:
The update fails:
heat resource-show overcloud ControllerNodesPostDeployment
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                                                                 |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {}                                                                                                                                                                                                                    |
| description            |                                                                                                                                                                                                                       |
| links                  | http://192.0.2.1:8004/v1/99ac7f5e5f124ce5bd59dbdb456beefd/stacks/overcloud/fc3d8f63-a2fc-48c1-a7ad-d72368695ad4/resources/ControllerNodesPostDeployment (self)                                                        |
|                        | http://192.0.2.1:8004/v1/99ac7f5e5f124ce5bd59dbdb456beefd/stacks/overcloud/fc3d8f63-a2fc-48c1-a7ad-d72368695ad4 (stack)                                                                                               |
|                        | http://192.0.2.1:8004/v1/99ac7f5e5f124ce5bd59dbdb456beefd/stacks/overcloud-ControllerNodesPostDeployment-yhuym3luxvfe/a805fec5-39f9-484a-8871-77b2c764d180 (nested)                                                   |
| logical_resource_id    | ControllerNodesPostDeployment                                                                                                                                                                                         |
| physical_resource_id   | a805fec5-39f9-484a-8871-77b2c764d180                                                                                                                                                                                  |
| required_by            | CephStorageNodesPostDeployment                                                                                                                                                                                        |
|                        | BlockStorageNodesPostDeployment                                                                                                                                                                                       |
| resource_name          | ControllerNodesPostDeployment                                                                                                                                                                                         |
| resource_status        | UPDATE_FAILED                                                                                                                                                                                                         |
| resource_status_reason | resources.ControllerNodesPostDeployment: resources.ControllerOvercloudServicesDeployment_Step7: Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 |
| resource_type          | OS::TripleO::ControllerPostDeployment                                                                                                                                                                                 |
| updated_time           | 2015-12-18T18:53:12Z                                                                                                                                                                                                  |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+



I see the following errors on a controller:
Dec 18 14:06:35 overcloud-controller-2 os-collect-config: instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please
 use sync_db instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated\u001b[
0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning:
 Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy:
:Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Concat::Setup]): concat::setup is deprecate
d as a public API of the concat module and should no longer be directly included in the manifest.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6002]): The default incoming_chmod set to 0644 may yiel
d in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6002]): The default outgoing_chmod set to 0644 may yield in error prone directories
and will be changed in a later release.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6001]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a late
r release.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6001]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1
;31mWarning: Scope(Swift::Storage::Server[6000]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1;31mW
Dec 18 14:06:35 overcloud-controller-2 os-collect-config: arning: Scope(Swift::Storage::Server[6000]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later rel
ease.\u001b[0m\n", "deploy_status_code": 0}
Dec 18 14:06:35 overcloud-controller-2 os-collect-config: #033[1;31mWarning: Scope(Swift::Storage::Server[6002]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in
a later release.#033[0m
Dec 18 14:06:35 overcloud-controller-2 os-collect-config: #033[1;31mWarning: Scope(Swift::Storage::Server[6002]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in
a later release.#033[0m
Dec 18 14:06:35 overcloud-controller-2 os-collect-config: #033[1;31mWarning: Scope(Swift::Storage::Server[6001]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in
a later release.#033[0m
Dec 18 14:06:35 overcloud-controller-2 os-collect-config: #033[1;31mWarning: Scope(Swift::Storage::Server[6001]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.#033[0m
Dec 18 14:06:35 overcloud-controller-2 os-collect-config: #033[1;31mWarning: Scope(Swift::Storage::Server[6000]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a later release.#033[0m
Dec 18 14:06:35 overcloud-controller-2 os-collect-config: #033[1;31mWarning: Scope(Swift::Storage::Server[6000]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.#033[0m
Dec 18 14:09:22 overcloud-controller-2 os-collect-config: \nWarning: nmcli (1.0.6) and NetworkManager (1.0.0) versions don't match. Use --nocheck to suppress the warning.\nWarning: nmcli (1.0.6) and NetworkManager (1.0.0) versions don't match. Use --nocheck to suppress the warning.\nWarning: nmcli (1.0.6) and NetworkManager (1.0.0) versions don't match. Use --nocheck to suppress the warning.\nWarning: nmcli (1.0.6) and NetworkManager (1.0.0) versions don't match. Use --nocheck to suppress the warning.\nWarning: nmcli (1.0.6) and NetworkManager (1.0.0) versions don't match. Use --nocheck to suppress the warning.\n\u001b[1;31mWarning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Concat::Setup]): concat::setup is deprecated as a public API of the concat module and should no longer be directly included in the manifest.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6002]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6002]): The def
Dec 18 14:09:22 overcloud-controller-2 os-collect-config: ault outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6001]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6001]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6000]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n\u001b[1;31mWarning: Scope(Swift::Storage::Server[6000]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.\u001b[0m\n", "deploy_status_code": 0}
Comment 2 Alexander Chuzhoy 2015-12-18 14:38 EST
Created attachment 1107398 [details]
/var/log/messages from controller
Comment 4 James Slagle 2016-01-13 15:57:13 EST
can you run heat deployment-show on the failed deployment?

is the environment still around for further debugging?
Comment 5 Mike Burns 2016-01-20 12:31:36 EST
Sasha, can you reproduce this and provide logs?
Comment 6 Mike Burns 2016-01-27 10:20:27 EST
If this is reproduced with 7.3, please reopen.
Comment 7 Omri Hochman 2016-01-29 11:32:35 EST
Re-open the Bz as ON_QA and with need_info flag to get more logs. 

Mike correct me if I'm wrong, even if it does not reproduces when updating to 7.3 
- still we need to test and to support update to 7.2?
Comment 8 Omri Hochman 2016-01-29 11:35:02 EST
we need to reproduce and add logs .
Comment 10 Alexander Chuzhoy 2016-02-10 12:54:43 EST
Verified:
Environment:
instack-undercloud-2.1.2-39.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-117.el7ost.noarch


Able to update successfully 7.0-> 7.3
Comment 12 errata-xmlrpc 2016-02-18 11:48:04 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0264.html

Note You need to log in before you can comment on or make changes to this bug.