Bug 1455616 - [UPDATES] minor updates failed on step overcloud.AllNodesDeploySteps.ControllerSwiftRingUpdate.SwiftRingUpdate.2
Summary: [UPDATES] minor updates failed on step overcloud.AllNodesDeploySteps.Controll...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
urgent
unspecified
Target Milestone: z3
: 10.0 (Newton)
Assignee: Emilien Macchi
QA Contact: Mike Abrams
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-25 15:31 UTC by Yurii Prokulevych
Modified: 2020-12-14 08:45 UTC (History)
21 users (show)

Fixed In Version: openstack-tripleo-common-5.4.2-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-28 14:50:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1695778 0 None None None 2017-06-04 22:30:47 UTC
OpenStack gerrit 468960 0 None None None 2017-05-30 05:45:25 UTC
OpenStack gerrit 469131 0 None None None 2017-06-13 21:30:15 UTC
Red Hat Product Errata RHBA-2017:1585 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 director Bug Fix Advisory 2017-06-28 18:42:51 UTC

Description Yurii Prokulevych 2017-05-25 15:31:19 UTC
Description of problem:
-----------------------
Minor update failed:

openstack stack failures list --long overcloud
overcloud.AllNodesDeploySteps.ControllerPostPuppet.ControllerPostPuppetMaintenanceModeDeployment.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 675ef980-3dde-4831-aa7f-c95f87ffd796
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
  deploy_stdout: |

  deploy_stderr: |

overcloud.AllNodesDeploySteps.ControllerSwiftRingUpdate.SwiftRingUpdate.1:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: f63cea0d-2671-4f01-83a5-0414fa69ebd6
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    /tmp/tmp.4dNxWPGmsS /var/lib/heat-config/heat-config-script
    /etc/swift/account.builder
    /etc/swift/container.builder
    /etc/swift/object.builder
    /etc/swift/account.ring.gz
    /etc/swift/container.ring.gz
    /etc/swift/object.ring.gz
    /etc/swift/backups/1495631808.account.builder
    /etc/swift/backups/1495631808.container.builder
    /etc/swift/backups/1495631808.object.builder
    /etc/swift/backups/1495631812.account.builder
    /etc/swift/backups/1495631812.account.ring.gz
    /etc/swift/backups/1495631812.container.builder
    /etc/swift/backups/1495631812.container.ring.gz
    /etc/swift/backups/1495631812.object.builder
    /etc/swift/backups/1495631812.object.ring.gz
    /var/lib/heat-config/heat-config-script
  deploy_stderr: |
    tar: Removing leading `/' from member names
overcloud.AllNodesDeploySteps.ControllerSwiftRingUpdate.SwiftRingUpdate.0:                                                                                                                                 [0/1710]
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: c025e76e-f6d5-4b3d-8cfb-c2b0fbfa5403
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    /tmp/tmp.KOT5sf8zFd /var/lib/heat-config/heat-config-script
    /etc/swift/account.builder
    /etc/swift/container.builder
    /etc/swift/object.builder
    /etc/swift/account.ring.gz
    /etc/swift/container.ring.gz
    /etc/swift/object.ring.gz
    /etc/swift/backups/1495631816.container.builder
    /etc/swift/backups/1495631816.object.builder
    /etc/swift/backups/1495631818.account.builder
    /etc/swift/backups/1495631821.account.builder
    /etc/swift/backups/1495631821.account.ring.gz
    /etc/swift/backups/1495631822.container.builder
    /etc/swift/backups/1495631822.container.ring.gz
    /etc/swift/backups/1495631822.object.builder
    /etc/swift/backups/1495631822.object.ring.gz
    /var/lib/heat-config/heat-config-script
  deploy_stderr: |
    tar: Removing leading `/' from member names

overcloud.AllNodesDeploySteps.ControllerSwiftRingUpdate.SwiftRingUpdate.2:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 004c67d4-303b-446d-96b7-2ad53db54083
  status: CREATE_FAILED
  status_reason: |
    Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    /tmp/tmp.oztmKgyCmR /var/lib/heat-config/heat-config-script
    /etc/swift/account.builder
    /etc/swift/container.builder
    /etc/swift/object.builder
    /etc/swift/account.ring.gz
    /etc/swift/container.ring.gz
    /etc/swift/object.ring.gz
    /etc/swift/backups/1495631808.container.builder
    /etc/swift/backups/1495631808.object.builder
    /etc/swift/backups/1495631809.account.builder
    /etc/swift/backups/1495631812.account.builder
    /etc/swift/backups/1495631812.account.ring.gz
    /etc/swift/backups/1495631812.container.builder
    /etc/swift/backups/1495631812.container.ring.gz
    /etc/swift/backups/1495631813.object.builder
    /etc/swift/backups/1495631813.object.ring.gz
    /var/lib/heat-config/heat-config-script
  deploy_stderr: |
    tar: Removing leading `/' from member names


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-tripleo-heat-templates-5.2.0-18.el7ost.noarch
openstack-heat-templates-0-0.11.1e6015dgit.el7ost.noarch

openstack-tripleo-puppet-elements-5.3.0-1.el7ost.noarch
openstack-tripleo-heat-templates-5.2.0-18.el7ost.noarch
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
openstack-tripleo-image-elements-5.2.0-2.el7ost.noarch
openstack-tripleo-ui-1.2.0-1.el7ost.noarch
openstack-tripleo-validations-5.1.1-1.el7ost.noarch
puppet-tripleo-5.6.0-3.el7ost.noarch
openstack-tripleo-common-5.4.2-1.el7ost.noarch
python-tripleoclient-5.4.2-1.el7ost.noarch

Steps to Reproduce:
-------------------
1. Install RHOS-10(z2)
2. Install repos
3. Update UC
4. Update plan
5. Update OC

Additional info:
----------------
Virtual setup: 3controllers + 2computes + 3ceph

Comment 8 Christian Schwede (cschwede) 2017-05-29 14:49:26 UTC
Is it possible to get the content from /var/lib/heat-config/heat-config-puppet from one of the controller nodes? Unfortunately it's not included in the sosreport.

Comment 9 Lukas Bezdicka 2017-05-29 15:30:06 UTC
Long story short cherrypick to OSP10 in middle of release means that deploy part of workflow never ran and the container overcloud-swift-rings is missing and so are the URLs. I have WIP patch in progress to plan_management.yaml. Will send it shortly.

Comment 10 Christian Schwede (cschwede) 2017-05-30 05:45:25 UTC
Proposed upstream fix from Lukas: https://review.openstack.org/#/c/468960/1

Comment 11 Lukas Bezdicka 2017-05-30 21:48:45 UTC
Restarting mistral won't reload the workbooks, proper reload is to:
# load workflows
for workbook in $(openstack workbook list -f value -c Name | grep tripleo); do
    openstack workbook delete $workbook
done
for workflow in $(openstack workflow list -f value -c Name | grep tripleo); do
    openstack workflow delete $workflow
done
for workbook in $(ls /usr/share/openstack-tripleo-common/workbooks/*); do
    openstack workbook create $workbook
done
see https://github.com/openstack/instack-undercloud/blob/master/elements/undercloud-install/os-refresh-config/post-configure.d/98-undercloud-setup#L93

Comment 15 Amit Ugol 2017-06-19 13:47:47 UTC
We need this in the official z.3 for updates to work.

Comment 47 errata-xmlrpc 2017-06-28 14:50:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1585


Note You need to log in before you can comment on or make changes to this bug.