Bug 1562764

Summary: [UPDATES] failed to update from z2 to latest "Deployment exited with non-zero status code: 1"
Product: Red Hat OpenStack Reporter: Raviv Bar-Tal <rbartal>
Component: collectdAssignee: Matthias Runge <mrunge>
Status: CLOSED DUPLICATE QA Contact: Leonid Natapov <lnatapov>
Severity: medium Docs Contact:
Priority: medium    
Version: 11.0 (Ocata)CC: aschultz, dbecker, jbadiapa, lars, mburns, mmagr, morazi, mrunge, pkilambi, rbartal, rhel-osp-director-maint, rmccabe
Target Milestone: z3Keywords: Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-18 07:37:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ceph
none
compute logs
none
controller logs
none
update log none

Description Raviv Bar-Tal 2018-04-02 12:19:12 UTC 
Created attachment 1416245 [details]
ceph

Description of problem:
Trying to update osp11 Z2 cloud to latest fails with error:
Deployment exited with non-zero status code: 1

Version-Release number of selected component (if applicable):

openstack-tripleo-validations-5.6.2-1.el7ost.noarch
puppet-tripleo-6.5.5-2.el7ost.noarch
openstack-tripleo-ui-3.2.2-1.el7ost.noarch
openstack-tripleo-heat-templates-6.2.7-2.el7ost.noarch
openstack-tripleo-common-6.1.4-1.el7ost.noarch
python-tripleoclient-6.2.3-1.el7ost.noarch
openstack-tripleo-puppet-elements-6.2.4-1.el7ost.noarch
openstack-tripleo-image-elements-6.1.2-1.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. Install osp11 z2 cloud with 3 controllers. 2 compute and 3 ceph's
2. run update command
3.

Actual results:
update fails

Expected results:


Additional info:
logs are attached,
reproduce with this Jenkins job:
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-upgrades-updates-11_director-rhel-7.4-virthost-3cont_2comp_3ceph-ipv4-vxlan/
Comment 1 Raviv Bar-Tal 2018-04-02 12:20:20 UTC 
Created attachment 1416246 [details]
compute logs
Comment 2 Raviv Bar-Tal 2018-04-02 12:22:08 UTC 
Created attachment 1416247 [details]
controller logs
Comment 3 Raviv Bar-Tal 2018-04-02 12:22:46 UTC 
Created attachment 1416248 [details]
update log
Comment 4 Raviv Bar-Tal 2018-04-02 12:24:42 UTC 
stack failures output
[stack@undercloud-0 ~]$ . stackrc ; openstack stack failures list overcloud --long
overcloud.CephStorage.0:
  resource_type: OS::TripleO::CephStorage
  physical_resource_id: 988384a6-5d09-428b-a683-66486ca96d4c
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
overcloud.CephStorage.2:
  resource_type: OS::TripleO::CephStorage
  physical_resource_id: a6613619-68ae-439b-a059-554b17b0609e
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
overcloud.Controller.1:
  resource_type: OS::TripleO::Controller
  physical_resource_id: 27bc89ed-6c23-4d5a-b5da-6627f3a6d453
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
overcloud.Controller.0:
  resource_type: OS::TripleO::Controller
  physical_resource_id: 3ae2b1da-15f8-4298-a361-0c2267dce3a9
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
overcloud.Controller.2.UpdateDeployment:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: e5cbb412-703e-4e57-932f-b9b3c3543f7e
  status: UPDATE_FAILED
  status_reason: |
    Error: resources.UpdateDeployment: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    Started yum_update.sh on server 1e03539b-2395-43df-b29b-c02bd6edbef2 at Sun Apr  1 15:24:15 UTC 2018
    Loaded plugins: product-id, search-disabled-repos, subscription-manager
    This system is not registered with an entitlement server. You can use subscription-manager to register.
    Metadata Cache Created
    Loaded plugins: product-id, search-disabled-repos, subscription-manager
    This system is not registered with an entitlement server. You can use subscription-manager to register.
    Metadata Cache Created
    Skipping manual upgrade of openvswitch - no restart in postun detected
    Checking for ceph-osd dependency issues
    ceph-osd package is available from an enabled repo
    Pacemaker running, stopping cluster node and doing full package update
    Stopping Cluster (pacemaker)...
    Stopping Cluster (corosync)...
    Running: yum -q -y update 
  deploy_stderr: |
    Error: Package: collectd-curl_json-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-write_http-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-apache-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-ceph-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-disk-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-curl-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-utils-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-curl_xml-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-ping-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-iptables-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd = 5.7.2-1.el7cp
    Error: Package: collectd-rrdtool-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-smart-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-mysql-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-lvm-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-write_sensu-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-sensors-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-rrdcached-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-ipmi-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-virt-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
    Error: Package: collectd-netlink-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
               Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp
overcloud.Compute.1:
  resource_type: OS::TripleO::Compute
  physical_resource_id: 8bd45057-4fd3-4a21-89be-500bf0f83e86
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
overcloud.Compute.0:
  resource_type: OS::TripleO::Compute
  physical_resource_id: 84d7d005-3e19-4607-b1c4-d436d80070ef
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted
[stack@undercloud-0 ~]$
Comment 6 Matthias Runge 2018-04-18 07:37:25 UTC 
I've seen that, the issue is collectd in ceph.

https://bugzilla.redhat.com/show_bug.cgi?id=1550149

See
      Requires: collectd(x86-64) = 5.7.0-4.el7ost
               Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle)
                   collectd(x86-64) = 5.7.0-4.el7ost
               Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
                   collectd(x86-64) = 5.7.2-1.el7cp

collectd-...ost is provided from us, collectd-...cp is coming from ceph.

However, we're upgrading to collectd 5.8.0

*** This bug has been marked as a duplicate of bug 1549129 ***
Comment 8 Raviv Bar-Tal 2018-04-26 11:56:40 UTC 
Hi Mattias
You are right, I'm not installing to newest z release,
I'm testing update, and there are Customers who do not update their cloud every time we release new z release.