Created attachment 1416245 [details] ceph Description of problem: Trying to update osp11 Z2 cloud to latest fails with error: Deployment exited with non-zero status code: 1 Version-Release number of selected component (if applicable): openstack-tripleo-validations-5.6.2-1.el7ost.noarch puppet-tripleo-6.5.5-2.el7ost.noarch openstack-tripleo-ui-3.2.2-1.el7ost.noarch openstack-tripleo-heat-templates-6.2.7-2.el7ost.noarch openstack-tripleo-common-6.1.4-1.el7ost.noarch python-tripleoclient-6.2.3-1.el7ost.noarch openstack-tripleo-puppet-elements-6.2.4-1.el7ost.noarch openstack-tripleo-image-elements-6.1.2-1.el7ost.noarch How reproducible: Steps to Reproduce: 1. Install osp11 z2 cloud with 3 controllers. 2 compute and 3 ceph's 2. run update command 3. Actual results: update fails Expected results: Additional info: logs are attached, reproduce with this Jenkins job: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-upgrades-updates-11_director-rhel-7.4-virthost-3cont_2comp_3ceph-ipv4-vxlan/
Created attachment 1416246 [details] compute logs
Created attachment 1416247 [details] controller logs
Created attachment 1416248 [details] update log
stack failures output [stack@undercloud-0 ~]$ . stackrc ; openstack stack failures list overcloud --long overcloud.CephStorage.0: resource_type: OS::TripleO::CephStorage physical_resource_id: 988384a6-5d09-428b-a683-66486ca96d4c status: UPDATE_FAILED status_reason: | UPDATE aborted overcloud.CephStorage.2: resource_type: OS::TripleO::CephStorage physical_resource_id: a6613619-68ae-439b-a059-554b17b0609e status: UPDATE_FAILED status_reason: | UPDATE aborted overcloud.Controller.1: resource_type: OS::TripleO::Controller physical_resource_id: 27bc89ed-6c23-4d5a-b5da-6627f3a6d453 status: UPDATE_FAILED status_reason: | UPDATE aborted overcloud.Controller.0: resource_type: OS::TripleO::Controller physical_resource_id: 3ae2b1da-15f8-4298-a361-0c2267dce3a9 status: UPDATE_FAILED status_reason: | UPDATE aborted overcloud.Controller.2.UpdateDeployment: resource_type: OS::Heat::SoftwareDeployment physical_resource_id: e5cbb412-703e-4e57-932f-b9b3c3543f7e status: UPDATE_FAILED status_reason: | Error: resources.UpdateDeployment: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 deploy_stdout: | Started yum_update.sh on server 1e03539b-2395-43df-b29b-c02bd6edbef2 at Sun Apr 1 15:24:15 UTC 2018 Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Metadata Cache Created Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Metadata Cache Created Skipping manual upgrade of openvswitch - no restart in postun detected Checking for ceph-osd dependency issues ceph-osd package is available from an enabled repo Pacemaker running, stopping cluster node and doing full package update Stopping Cluster (pacemaker)... Stopping Cluster (corosync)... Running: yum -q -y update deploy_stderr: | Error: Package: collectd-curl_json-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-write_http-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-apache-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-ceph-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-disk-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-curl-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-utils-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-curl_xml-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-ping-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-iptables-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd = 5.7.2-1.el7cp Error: Package: collectd-rrdtool-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-smart-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-mysql-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-lvm-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-write_sensu-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-sensors-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-rrdcached-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-ipmi-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-virt-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp Error: Package: collectd-netlink-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp overcloud.Compute.1: resource_type: OS::TripleO::Compute physical_resource_id: 8bd45057-4fd3-4a21-89be-500bf0f83e86 status: UPDATE_FAILED status_reason: | UPDATE aborted overcloud.Compute.0: resource_type: OS::TripleO::Compute physical_resource_id: 84d7d005-3e19-4607-b1c4-d436d80070ef status: UPDATE_FAILED status_reason: | UPDATE aborted [stack@undercloud-0 ~]$
I've seen that, the issue is collectd in ceph. https://bugzilla.redhat.com/show_bug.cgi?id=1550149 See Requires: collectd(x86-64) = 5.7.0-4.el7ost Removing: collectd-5.7.0-4.el7ost.x86_64 (@rhos-11.0-optools-puddle) collectd(x86-64) = 5.7.0-4.el7ost Updated By: collectd-5.7.2-1.el7cp.x86_64 (rhelosp-ceph-2.0-mon) collectd(x86-64) = 5.7.2-1.el7cp collectd-...ost is provided from us, collectd-...cp is coming from ceph. However, we're upgrading to collectd 5.8.0 *** This bug has been marked as a duplicate of bug 1549129 ***
Hi Mattias You are right, I'm not installing to newest z release, I'm testing update, and there are Customers who do not update their cloud every time we release new z release.