Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1455616

Summary: [UPDATES] minor updates failed on step overcloud.AllNodesDeploySteps.ControllerSwiftRingUpdate.SwiftRingUpdate.2
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-tripleo-commonAssignee: Emilien Macchi <emacchi>
Status: CLOSED ERRATA QA Contact: Mike Abrams <mabrams>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: agurenko, aschultz, atelang, augol, cschwede, dsariel, ebarrera, jjoyce, lbezdick, mandreou, mbracho, mburns, pgrist, rhel-osp-director-maint, sathlang, sclewis, scohen, slinaber, ssbarnea, thiago, yprokule
Target Milestone: z3Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-5.4.2-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-28 14:50:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yurii Prokulevych 2017-05-25 15:31:19 UTC
Description of problem:
-----------------------
Minor update failed:

openstack stack failures list --long overcloud
overcloud.AllNodesDeploySteps.ControllerPostPuppet.ControllerPostPuppetMaintenanceModeDeployment.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 675ef980-3dde-4831-aa7f-c95f87ffd796
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
  deploy_stdout: |

  deploy_stderr: |

overcloud.AllNodesDeploySteps.ControllerSwiftRingUpdate.SwiftRingUpdate.1:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: f63cea0d-2671-4f01-83a5-0414fa69ebd6
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    /tmp/tmp.4dNxWPGmsS /var/lib/heat-config/heat-config-script
    /etc/swift/account.builder
    /etc/swift/container.builder
    /etc/swift/object.builder
    /etc/swift/account.ring.gz
    /etc/swift/container.ring.gz
    /etc/swift/object.ring.gz
    /etc/swift/backups/1495631808.account.builder
    /etc/swift/backups/1495631808.container.builder
    /etc/swift/backups/1495631808.object.builder
    /etc/swift/backups/1495631812.account.builder
    /etc/swift/backups/1495631812.account.ring.gz
    /etc/swift/backups/1495631812.container.builder
    /etc/swift/backups/1495631812.container.ring.gz
    /etc/swift/backups/1495631812.object.builder
    /etc/swift/backups/1495631812.object.ring.gz
    /var/lib/heat-config/heat-config-script
  deploy_stderr: |
    tar: Removing leading `/' from member names
overcloud.AllNodesDeploySteps.ControllerSwiftRingUpdate.SwiftRingUpdate.0:                                                                                                                                 [0/1710]
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: c025e76e-f6d5-4b3d-8cfb-c2b0fbfa5403
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    /tmp/tmp.KOT5sf8zFd /var/lib/heat-config/heat-config-script
    /etc/swift/account.builder
    /etc/swift/container.builder
    /etc/swift/object.builder
    /etc/swift/account.ring.gz
    /etc/swift/container.ring.gz
    /etc/swift/object.ring.gz
    /etc/swift/backups/1495631816.container.builder
    /etc/swift/backups/1495631816.object.builder
    /etc/swift/backups/1495631818.account.builder
    /etc/swift/backups/1495631821.account.builder
    /etc/swift/backups/1495631821.account.ring.gz
    /etc/swift/backups/1495631822.container.builder
    /etc/swift/backups/1495631822.container.ring.gz
    /etc/swift/backups/1495631822.object.builder
    /etc/swift/backups/1495631822.object.ring.gz
    /var/lib/heat-config/heat-config-script
  deploy_stderr: |
    tar: Removing leading `/' from member names

overcloud.AllNodesDeploySteps.ControllerSwiftRingUpdate.SwiftRingUpdate.2:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 004c67d4-303b-446d-96b7-2ad53db54083
  status: CREATE_FAILED
  status_reason: |
    Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    /tmp/tmp.oztmKgyCmR /var/lib/heat-config/heat-config-script
    /etc/swift/account.builder
    /etc/swift/container.builder
    /etc/swift/object.builder
    /etc/swift/account.ring.gz
    /etc/swift/container.ring.gz
    /etc/swift/object.ring.gz
    /etc/swift/backups/1495631808.container.builder
    /etc/swift/backups/1495631808.object.builder
    /etc/swift/backups/1495631809.account.builder
    /etc/swift/backups/1495631812.account.builder
    /etc/swift/backups/1495631812.account.ring.gz
    /etc/swift/backups/1495631812.container.builder
    /etc/swift/backups/1495631812.container.ring.gz
    /etc/swift/backups/1495631813.object.builder
    /etc/swift/backups/1495631813.object.ring.gz
    /var/lib/heat-config/heat-config-script
  deploy_stderr: |
    tar: Removing leading `/' from member names


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-tripleo-heat-templates-5.2.0-18.el7ost.noarch
openstack-heat-templates-0-0.11.1e6015dgit.el7ost.noarch

openstack-tripleo-puppet-elements-5.3.0-1.el7ost.noarch
openstack-tripleo-heat-templates-5.2.0-18.el7ost.noarch
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
openstack-tripleo-image-elements-5.2.0-2.el7ost.noarch
openstack-tripleo-ui-1.2.0-1.el7ost.noarch
openstack-tripleo-validations-5.1.1-1.el7ost.noarch
puppet-tripleo-5.6.0-3.el7ost.noarch
openstack-tripleo-common-5.4.2-1.el7ost.noarch
python-tripleoclient-5.4.2-1.el7ost.noarch

Steps to Reproduce:
-------------------
1. Install RHOS-10(z2)
2. Install repos
3. Update UC
4. Update plan
5. Update OC

Additional info:
----------------
Virtual setup: 3controllers + 2computes + 3ceph

Comment 8 Christian Schwede (cschwede) 2017-05-29 14:49:26 UTC
Is it possible to get the content from /var/lib/heat-config/heat-config-puppet from one of the controller nodes? Unfortunately it's not included in the sosreport.

Comment 9 Lukas Bezdicka 2017-05-29 15:30:06 UTC
Long story short cherrypick to OSP10 in middle of release means that deploy part of workflow never ran and the container overcloud-swift-rings is missing and so are the URLs. I have WIP patch in progress to plan_management.yaml. Will send it shortly.

Comment 10 Christian Schwede (cschwede) 2017-05-30 05:45:25 UTC
Proposed upstream fix from Lukas: https://review.openstack.org/#/c/468960/1

Comment 11 Lukas Bezdicka 2017-05-30 21:48:45 UTC
Restarting mistral won't reload the workbooks, proper reload is to:
# load workflows
for workbook in $(openstack workbook list -f value -c Name | grep tripleo); do
    openstack workbook delete $workbook
done
for workflow in $(openstack workflow list -f value -c Name | grep tripleo); do
    openstack workflow delete $workflow
done
for workbook in $(ls /usr/share/openstack-tripleo-common/workbooks/*); do
    openstack workbook create $workbook
done
see https://github.com/openstack/instack-undercloud/blob/master/elements/undercloud-install/os-refresh-config/post-configure.d/98-undercloud-setup#L93

Comment 15 Amit Ugol 2017-06-19 13:47:47 UTC
We need this in the official z.3 for updates to work.

Comment 47 errata-xmlrpc 2017-06-28 14:50:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1585