Bug 1394587
| Summary: | rhel-osp-director: OSP10 minor update fails: Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install yum-plugin-priorities' returned 1: Error: Nothing to do | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> | ||||
| Component: | openstack-tripleo-heat-templates | Assignee: | John Fulton <johfulto> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> | ||||
| Severity: | urgent | Docs Contact: | Derek <dcadzow> | ||||
| Priority: | urgent | ||||||
| Version: | 10.0 (Newton) | CC: | dbecker, gfidente, jcoufal, johfulto, jomurphy, jschluet, jslagle, kschinck, mandreou, mburns, mcornea, morazi, pgrist, rhel-osp-director-maint, sasha, scohen | ||||
| Target Milestone: | rc | Keywords: | Triaged | ||||
| Target Release: | 10.0 (Newton) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-tripleo-heat-templates-5.1.0-3.el7ost | Doc Type: | No Doc Update | ||||
| Doc Text: |
undefined
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-12-14 16:32:14 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
1)
Deploy command:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 2 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 0 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml -e /home/stack/extceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log
Update command:
yes ""| openstack overcloud update stack -i overcloud
2)
I only see this issue with external ceph.
I was able to deploy with external ceph and to launch an instance.
The update fails. Here's the status from ceph:
cluster 1c50eddf-56fb-470f-a72d-47a960f520a0
health HEALTH_OK
monmap e3: 3 mons at {ceph-node1=10.8.29.226:6789/0,ceph-node2=10.8.30.200:6789/0,ceph-node3=10.8.29.1:6789/0}
election epoch 80, quorum 0,1,2 ceph-node3,ceph-node1,ceph-node2
osdmap e882: 6 osds: 6 up, 6 in
pgmap v605366: 1344 pgs, 8 pools, 109 MB data, 1484 objects
605 MB used, 269 GB / 269 GB avail
1344 active+clean
3)
This is how the update look on the console:
20:01:48 stdout: starting package update on stack overcloud
20:01:48 IN_PROGRESS
20:01:48 WAITING
20:01:48 on_breakpoint: [u'controller-2', u'controller-1', u'compute-0', u'controller-0', u'compute-1']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear d84a7955-43dd-4cfc-9260-b110b8d5458b), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
20:01:48 WAITING
20:01:48 completed: [u'compute-1']
20:01:48 on_breakpoint: [u'controller-2', u'controller-1', u'compute-0', u'controller-0']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear d2c056c4-d5fd-465d-b917-241a0769f425), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 WAITING
20:01:48 completed: [u'compute-1', u'controller-0']
20:01:48 on_breakpoint: [u'controller-2', u'controller-1', u'compute-0']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear 39145360-475b-45f6-b267-f40a57172a54), no=cancel update, C-c=quit interactive mode: WAITING
20:01:48 completed: [u'compute-1', u'compute-0', u'controller-0']
20:01:48 on_breakpoint: [u'controller-2', u'controller-1']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear 7917f829-1853-4d40-ba5a-513e0b01e3cb), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 WAITING
20:01:48 completed: [u'controller-1', u'compute-0', u'controller-0', u'compute-1']
20:01:48 on_breakpoint: [u'controller-2']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear 8e57bd1a-ee3e-4da5-96a4-cff927dc33c6), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 FAILED
20:01:48 update finished with status FAILED
When I try to run "yum update" on any node now (I left the setup running over night) - I get:
---> Package policycoreutils-python.x86_64 0:2.5-8.el7 will be updated
---> Package policycoreutils-python.x86_64 0:2.5-9.el7 will be an update
[stack@undercloud-0 ~]$ for i in `nova list|awk -F"|" '/Running/ {print $(NF-1)}'|awk -F"=" '{print $NF}'`; do echo "####################################"; ssh heat-admin@$i "hostname; sudo grep -i update /var/log/yum.log"; done
####################################
compute-0.localdomain
Nov 13 19:57:52 Updated: 1:librados2-10.2.3-0.el7.x86_64
Nov 13 19:57:52 Updated: 1:python-rados-10.2.3-0.el7.x86_64
Nov 13 19:57:53 Updated: 1:librbd1-10.2.3-0.el7.x86_64
Nov 13 19:57:53 Updated: 1:librgw2-10.2.3-0.el7.x86_64
Nov 13 19:57:53 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64
Nov 13 19:57:54 Updated: 1:v8-3.14.5.10-25.el7.x86_64
Nov 13 19:57:54 Updated: leveldb-1.12.0-11.el7.x86_64
Nov 13 19:57:54 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64
Nov 13 19:57:54 Updated: 1:python-rbd-10.2.3-0.el7.x86_64
Nov 13 19:57:57 Updated: 1:ceph-common-10.2.3-0.el7.x86_64
Nov 13 19:58:04 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64
Nov 13 19:58:05 Updated: 1:ceph-base-10.2.3-0.el7.x86_64
Nov 13 19:58:05 Updated: python-tables-3.2.0-1.el7.x86_64
Nov 13 19:58:05 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch
Nov 13 19:58:05 Updated: python-Bottleneck-0.7.0-1.el7.x86_64
Nov 13 19:58:06 Updated: rubygem-daemons-1.1.9-4.el7.noarch
Nov 13 19:58:06 Updated: rubygem-thin-1.6.4-3.el7.x86_64
Nov 13 19:58:06 Updated: jq-1.5-1.el7.x86_64
Nov 13 19:58:07 Updated: python-pandas-0.17.1-1.el7.x86_64
Nov 13 19:58:09 Updated: python-unittest2-1.1.0-4.el7.noarch
Nov 13 19:58:09 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64
Nov 13 19:58:11 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64
Nov 13 19:58:11 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64
Nov 13 19:58:17 Updated: mongodb-2.6.12-3.el7.x86_64
Nov 13 19:58:18 Updated: mongodb-server-2.6.12-3.el7.x86_64
Nov 13 19:58:18 Updated: python-unicodecsv-0.14.1-4.el7.noarch
Nov 13 19:58:18 Updated: 1:hiera-1.3.4-5.el7.noarch
Nov 13 19:58:18 Updated: galera-25.3.12-2.el7.x86_64
Nov 13 19:58:18 Updated: ruby-shadow-1.4.1-23.el7.x86_64
Nov 13 19:58:18 Updated: python-psutil-2.2.1-1.el7.x86_64
Nov 13 19:58:19 Updated: pysnmp-4.2.5-2.el7.noarch
Nov 13 19:58:19 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch
Nov 13 19:58:19 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64
Nov 13 19:58:19 Updated: python-contextlib2-0.5.1-2.el7.noarch
Nov 13 19:58:19 Updated: userspace-rcu-0.7.16-1.el7.x86_64
Nov 13 19:58:19 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch
####################################
compute-1.localdomain
Nov 13 19:57:39 Updated: 1:librados2-10.2.3-0.el7.x86_64
Nov 13 19:57:39 Updated: 1:python-rados-10.2.3-0.el7.x86_64
Nov 13 19:57:39 Updated: 1:librbd1-10.2.3-0.el7.x86_64
Nov 13 19:57:39 Updated: 1:librgw2-10.2.3-0.el7.x86_64
Nov 13 19:57:40 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64
Nov 13 19:57:40 Updated: 1:v8-3.14.5.10-25.el7.x86_64
Nov 13 19:57:40 Updated: leveldb-1.12.0-11.el7.x86_64
Nov 13 19:57:40 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64
Nov 13 19:57:40 Updated: 1:python-rbd-10.2.3-0.el7.x86_64
Nov 13 19:57:43 Updated: 1:ceph-common-10.2.3-0.el7.x86_64
Nov 13 19:57:50 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64
Nov 13 19:57:51 Updated: 1:ceph-base-10.2.3-0.el7.x86_64
Nov 13 19:57:52 Updated: python-tables-3.2.0-1.el7.x86_64
Nov 13 19:57:52 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch
Nov 13 19:57:52 Updated: python-Bottleneck-0.7.0-1.el7.x86_64
Nov 13 19:57:52 Updated: rubygem-daemons-1.1.9-4.el7.noarch
Nov 13 19:57:52 Updated: rubygem-thin-1.6.4-3.el7.x86_64
Nov 13 19:57:52 Updated: jq-1.5-1.el7.x86_64
Nov 13 19:57:54 Updated: python-pandas-0.17.1-1.el7.x86_64
Nov 13 19:57:55 Updated: python-unittest2-1.1.0-4.el7.noarch
Nov 13 19:57:55 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64
Nov 13 19:57:57 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64
Nov 13 19:57:57 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64
Nov 13 19:58:03 Updated: mongodb-2.6.12-3.el7.x86_64
Nov 13 19:58:05 Updated: mongodb-server-2.6.12-3.el7.x86_64
Nov 13 19:58:05 Updated: python-unicodecsv-0.14.1-4.el7.noarch
Nov 13 19:58:05 Updated: 1:hiera-1.3.4-5.el7.noarch
Nov 13 19:58:05 Updated: galera-25.3.12-2.el7.x86_64
Nov 13 19:58:05 Updated: ruby-shadow-1.4.1-23.el7.x86_64
Nov 13 19:58:05 Updated: python-psutil-2.2.1-1.el7.x86_64
Nov 13 19:58:05 Updated: pysnmp-4.2.5-2.el7.noarch
Nov 13 19:58:05 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch
Nov 13 19:58:05 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64
Nov 13 19:58:05 Updated: python-contextlib2-0.5.1-2.el7.noarch
Nov 13 19:58:05 Updated: userspace-rcu-0.7.16-1.el7.x86_64
Nov 13 19:58:05 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch
####################################
controller-0.localdomain
Nov 13 19:45:43 Updated: 1:librados2-10.2.3-0.el7.x86_64
Nov 13 19:45:43 Updated: 1:python-rados-10.2.3-0.el7.x86_64
Nov 13 19:45:44 Updated: 1:librbd1-10.2.3-0.el7.x86_64
Nov 13 19:45:44 Updated: 1:librgw2-10.2.3-0.el7.x86_64
Nov 13 19:45:44 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64
Nov 13 19:45:45 Updated: 1:v8-3.14.5.10-25.el7.x86_64
Nov 13 19:45:45 Updated: leveldb-1.12.0-11.el7.x86_64
Nov 13 19:45:45 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64
Nov 13 19:45:45 Updated: 1:python-rbd-10.2.3-0.el7.x86_64
Nov 13 19:45:48 Updated: 1:ceph-common-10.2.3-0.el7.x86_64
Nov 13 19:45:54 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64
Nov 13 19:45:55 Updated: 1:ceph-base-10.2.3-0.el7.x86_64
Nov 13 19:45:56 Updated: python-tables-3.2.0-1.el7.x86_64
Nov 13 19:45:56 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch
Nov 13 19:45:56 Updated: python-Bottleneck-0.7.0-1.el7.x86_64
Nov 13 19:45:56 Updated: rubygem-daemons-1.1.9-4.el7.noarch
Nov 13 19:45:56 Updated: rubygem-thin-1.6.4-3.el7.x86_64
Nov 13 19:45:56 Updated: jq-1.5-1.el7.x86_64
Nov 13 19:45:58 Updated: python-pandas-0.17.1-1.el7.x86_64
Nov 13 19:45:59 Updated: python-unittest2-1.1.0-4.el7.noarch
Nov 13 19:46:00 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64
Nov 13 19:46:01 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64
Nov 13 19:46:01 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64
Nov 13 19:46:07 Updated: mongodb-2.6.12-3.el7.x86_64
Nov 13 19:46:08 Updated: mongodb-server-2.6.12-3.el7.x86_64
Nov 13 19:46:08 Updated: python-unicodecsv-0.14.1-4.el7.noarch
Nov 13 19:46:08 Updated: 1:hiera-1.3.4-5.el7.noarch
Nov 13 19:46:09 Updated: galera-25.3.12-2.el7.x86_64
Nov 13 19:46:09 Updated: ruby-shadow-1.4.1-23.el7.x86_64
Nov 13 19:46:09 Updated: python-psutil-2.2.1-1.el7.x86_64
Nov 13 19:46:09 Updated: pysnmp-4.2.5-2.el7.noarch
Nov 13 19:46:09 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch
Nov 13 19:46:09 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64
Nov 13 19:46:09 Updated: python-contextlib2-0.5.1-2.el7.noarch
Nov 13 19:46:09 Updated: userspace-rcu-0.7.16-1.el7.x86_64
Nov 13 19:46:09 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch
####################################
controller-1.localdomain
Nov 13 19:50:31 Updated: 1:librados2-10.2.3-0.el7.x86_64
Nov 13 19:50:31 Updated: 1:python-rados-10.2.3-0.el7.x86_64
Nov 13 19:50:32 Updated: 1:librbd1-10.2.3-0.el7.x86_64
Nov 13 19:50:32 Updated: 1:librgw2-10.2.3-0.el7.x86_64
Nov 13 19:50:33 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64
Nov 13 19:50:33 Updated: 1:v8-3.14.5.10-25.el7.x86_64
Nov 13 19:50:33 Updated: leveldb-1.12.0-11.el7.x86_64
Nov 13 19:50:33 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64
Nov 13 19:50:33 Updated: 1:python-rbd-10.2.3-0.el7.x86_64
Nov 13 19:50:36 Updated: 1:ceph-common-10.2.3-0.el7.x86_64
Nov 13 19:50:43 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64
Nov 13 19:50:44 Updated: 1:ceph-base-10.2.3-0.el7.x86_64
Nov 13 19:50:45 Updated: python-tables-3.2.0-1.el7.x86_64
Nov 13 19:50:45 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch
Nov 13 19:50:45 Updated: python-Bottleneck-0.7.0-1.el7.x86_64
Nov 13 19:50:45 Updated: rubygem-daemons-1.1.9-4.el7.noarch
Nov 13 19:50:45 Updated: rubygem-thin-1.6.4-3.el7.x86_64
Nov 13 19:50:45 Updated: jq-1.5-1.el7.x86_64
Nov 13 19:50:47 Updated: python-pandas-0.17.1-1.el7.x86_64
Nov 13 19:50:48 Updated: python-unittest2-1.1.0-4.el7.noarch
Nov 13 19:50:48 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64
Nov 13 19:50:50 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64
Nov 13 19:50:50 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64
Nov 13 19:50:56 Updated: mongodb-2.6.12-3.el7.x86_64
Nov 13 19:50:57 Updated: mongodb-server-2.6.12-3.el7.x86_64
Nov 13 19:50:57 Updated: python-unicodecsv-0.14.1-4.el7.noarch
Nov 13 19:50:57 Updated: 1:hiera-1.3.4-5.el7.noarch
Nov 13 19:50:57 Updated: galera-25.3.12-2.el7.x86_64
Nov 13 19:50:57 Updated: ruby-shadow-1.4.1-23.el7.x86_64
Nov 13 19:50:57 Updated: python-psutil-2.2.1-1.el7.x86_64
Nov 13 19:50:57 Updated: pysnmp-4.2.5-2.el7.noarch
Nov 13 19:50:57 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch
Nov 13 19:50:57 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64
Nov 13 19:50:57 Updated: python-contextlib2-0.5.1-2.el7.noarch
Nov 13 19:50:58 Updated: userspace-rcu-0.7.16-1.el7.x86_64
Nov 13 19:50:58 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch
####################################
controller-2.localdomain
Nov 13 19:54:23 Updated: 1:librados2-10.2.3-0.el7.x86_64
Nov 13 19:54:23 Updated: 1:python-rados-10.2.3-0.el7.x86_64
Nov 13 19:54:24 Updated: 1:librbd1-10.2.3-0.el7.x86_64
Nov 13 19:54:24 Updated: 1:librgw2-10.2.3-0.el7.x86_64
Nov 13 19:54:25 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64
Nov 13 19:54:25 Updated: 1:v8-3.14.5.10-25.el7.x86_64
Nov 13 19:54:25 Updated: leveldb-1.12.0-11.el7.x86_64
Nov 13 19:54:25 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64
Nov 13 19:54:25 Updated: 1:python-rbd-10.2.3-0.el7.x86_64
Nov 13 19:54:28 Updated: 1:ceph-common-10.2.3-0.el7.x86_64
Nov 13 19:54:35 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64
Nov 13 19:54:36 Updated: 1:ceph-base-10.2.3-0.el7.x86_64
Nov 13 19:54:36 Updated: python-tables-3.2.0-1.el7.x86_64
Nov 13 19:54:36 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch
Nov 13 19:54:36 Updated: python-Bottleneck-0.7.0-1.el7.x86_64
Nov 13 19:54:37 Updated: rubygem-daemons-1.1.9-4.el7.noarch
Nov 13 19:54:37 Updated: rubygem-thin-1.6.4-3.el7.x86_64
Nov 13 19:54:37 Updated: jq-1.5-1.el7.x86_64
Nov 13 19:54:38 Updated: python-pandas-0.17.1-1.el7.x86_64
Nov 13 19:54:40 Updated: python-unittest2-1.1.0-4.el7.noarch
Nov 13 19:54:40 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64
Nov 13 19:54:41 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64
Nov 13 19:54:41 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64
Nov 13 19:54:48 Updated: mongodb-2.6.12-3.el7.x86_64
Nov 13 19:54:49 Updated: mongodb-server-2.6.12-3.el7.x86_64
Nov 13 19:54:49 Updated: python-unicodecsv-0.14.1-4.el7.noarch
Nov 13 19:54:49 Updated: 1:hiera-1.3.4-5.el7.noarch
Nov 13 19:54:49 Updated: galera-25.3.12-2.el7.x86_64
Nov 13 19:54:49 Updated: ruby-shadow-1.4.1-23.el7.x86_64
Nov 13 19:54:49 Updated: python-psutil-2.2.1-1.el7.x86_64
Nov 13 19:54:49 Updated: pysnmp-4.2.5-2.el7.noarch
Nov 13 19:54:49 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch
Nov 13 19:54:49 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64
Nov 13 19:54:49 Updated: python-contextlib2-0.5.1-2.el7.noarch
Nov 13 19:54:50 Updated: userspace-rcu-0.7.16-1.el7.x86_64
Nov 13 19:54:50 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch
4) Attaching the file with enabled repos on nodes.
The issue reproduced twice only for ext ceph deployment.
Created attachment 1220464 [details]
enabled repos on nodes.
the issue here is with using puppet-ceph/manifests/repo.pp where it has:
package {'yum-plugin-priorities':
ensure => present
}
we need to either patch that manifest to not do that in cases where we are using RHEL or we should not be using this class from repo.pp.
We really don't want repo management done by puppet-ceph, as it is configured externally via subscription-manager or rhos-release for internal testing.
this should be DFG:Ceph Can you attach your below two files? /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml /home/stack/extceph.yaml [stack@undercloud-0 ~]$ cat /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml # A Heat environment file which can be used to enable the # use of an externally managed Ceph cluster. resource_registry: OS::TripleO::Services::CephExternal: ../puppet/services/ceph-external.yaml OS::TripleO::Services::CephMon: OS::Heat::None OS::TripleO::Services::CephClient: OS::Heat::None OS::TripleO::Services::CephOSD: OS::Heat::None parameter_defaults: # NOTE: These example parameters are required when using CephExternal #CephClusterFSID: '4b5c8c0a-ff60-454b-a1b4-9747aa737d19' CephClusterFSID: '1c50eddf-56fb-470f-a72d-47a960f520a0' #CephClientKey: 'AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ==' CephClientKey: 'AQBeskBWuKG8HhAA0iscCNXPa1yZkIb3y62Q7w==' #CephExternalMonHost: '172.16.1.7, 172.16.1.8' CephExternalMonHost: '10.8.29.226, 10.8.30.200, 10.8.29.1' # the following parameters enable Ceph backends for Cinder, Glance, Gnocchi and Nova NovaEnableRbdBackend: true CinderEnableRbdBackend: true CinderBackupBackend: ceph GlanceBackend: rbd GnocchiBackend: rbd # If the Ceph pools which host VMs, Volumes and Images do not match these # names OR the client keyring to use is not named 'openstack', edit the # following as needed. NovaRbdPoolName: vms CinderRbdPoolName: volumes GlanceRbdPoolName: images GnocchiRbdPoolName: metrics CephClientUserName: openstack # finally we disable the Cinder LVM backend CinderEnableIscsiBackend: false # Backward compatibility setting, will be removed in the future CephAdminKey: '' [stack@undercloud-0 ~]$ cat extceph.yaml
parameter_defaults:
ExtraConfig:
ceph::conf::args:
client/rbd_default_features:
value: "1"
Sasha,
Thanks for building the env for me to troubleshoot. Would you
please try to reproduce this error but make the following
changes:
a. Use the following extceph.yaml:
[stack@undercloud-0 ~]$ cat extceph.yaml
parameter_defaults:
ExtraConfig:
ceph::conf::args:
client/rbd_default_features:
value: "1"
ceph::profile::params::manage_repo: false
[stack@undercloud-0 ~]$
Note that I'm passing the manage_repo flag set to false to keep
repo.pp affecting your repo.
b. Use the following deploy command:
openstack overcloud deploy --debug --templates --libvirt-type kvm \
--ntp-server clock.redhat.com \
--neutron-network-type vxlan \
--neutron-tunnel-types vxlan \
--control-scale 3 --control-flavor controller-d75f3dec-c7-5f88-9d4c-3fea1bf9c484 \
--compute-scale 2 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppeceph-external.yaml \
-e /home/stack/extceph.yaml \
-e virt/hostnames.yml \
-e virt/network/network-environment.yaml \
--log-file overcloud_deployment_48.log
The change in the above is to not include storage-environment.yaml.
Since you're not installing ceph on your overcloud it shouldn't be
included. I also removed the ceph-storage scale and flavor simply
because they're not applicable in an external ceph situation.
Otherwise I just added some new lines for readability.
So I tried the following:
deploy_command:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 2 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml -e /home/stack/extceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log
[stack@undercloud-0 ~]$ cat extceph.yaml
parameter_defaults:
ExtraConfig:
ceph::conf::args:
client/rbd_default_features:
value: "1"
ceph::profile::params::manage_repo: false
The update failed with:
Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install ceph' returned 1: Error: Nothing to do
Error: /Stage[main]/Ceph/Package[ceph]/ensure: change from absent to latest failed: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install ceph' returned 1: Error: Nothing to do
Alexander, yep that is because we also nees to customize the list of packages to be installed, as per comment #19. That has been added to the upstream patch https://review.openstack.org/397819 Alexander, the patch has had another change (sorry but there was an indentation problem that may explain the most recent problem you had yesterday). Would you mind running the test again with the following (make sure it's patchset 5)? https://review.openstack.org/#/c/397819/5/puppet/services/ceph-external.yaml Alexander, as per our conversation, I understand that dropping in the new ceph-external.yaml and re-running your test resolved the issue. The fix merged into Octa and the backport needs to merge into Newton next. Thanks. --John https://review.openstack.org/#/c/398475/ The newton backport merged. Updating BZ to POST. Verified: Environment: openstack-tripleo-heat-templates-5.1.0-6.el7ost.noarch The reported issue doesn't reproduce. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html |
rhel-osp-director: OSP10 minor update fails: Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install yum-plugin-priorities' returned 1: Error: Nothing to do Environment: openstack-puppet-modules-9.3.0-1.el7ost.noarch openstack-tripleo-heat-templates-5.0.0-1.6.el7ost.noarch instack-undercloud-5.0.0-4.el7ost.noarch Steps to reproduce: 1. Deploy a setup with external ceph. 2. Update the undercloud and attempt to update the overcloud. Result: [stack@undercloud-0 ~]$ heat stack-list +--------------------------------------+------------+---------------+----------------------+----------------------+ | id | stack_name | stack_status | creation_time | updated_time | +--------------------------------------+------------+---------------+----------------------+----------------------+ | b065e928-307a-4a81-9cb2-91a13436b45a | overcloud | UPDATE_FAILED | 2016-11-13T18:47:20Z | 2016-11-13T19:35:44Z | +--------------------------------------+------------+---------------+----------------------+----------------------+ [stack@undercloud-0 ~]$ heat resource-list -n5 overcloud |grep -v COMPLE WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead heat deployment-show +-------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------+ | resource_name|physical_resource_id| resource_type| resource_status | updated_time | stack_name |+-------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------+ | AllNodesDeploySteps | dd0fad74-b976-455e-b7ac-35043575d785 | OS::TripleO::PostDeploySteps | UPDATE_FAILED | 2016-11-13T19:56:44Z | overcloud | | 1 | b0d329f1-c62a-4506-9966-d60ceb23afd3 | OS::Heat::StructuredDeployment | UPDATE_FAILED | 2016-11-13T19:58:46Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ControllerDeployment_Step2-w57753pd7fcj | | ControllerDeployment_Step2 | 72a34e44-e4a4-4fec-b1db-0ccbaec1b79e | OS::Heat::StructuredDeploymentGroup | UPDATE_FAILED | 2016-11-13T19:58:46Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox | | 1 | e4ed1f61-88cb-4413-95bd-e151923856f1 | OS::Heat::StructuredDeployment | UPDATE_FAILED | 2016-11-13T19:58:47Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ComputeDeployment_Step2-inigzo63jrs4 | | ComputeDeployment_Step2 | dd4e57e8-bae4-4b8c-8d95-9d5c3a6e820f | OS::Heat::StructuredDeploymentGroup | UPDATE_FAILED | 2016-11-13T19:58:47Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox | | 0 | ef0d2f1c-d9c8-4dc2-ac31-0eeb53f4dff0 | OS::Heat::StructuredDeployment | UPDATE_FAILED | 2016-11-13T19:58:49Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ComputeDeployment_Step2-inigzo63jrs4 | | 0 | fddef5ea-0d20-4066-9b9c-7321f631d46b | OS::Heat::StructuredDeployment | UPDATE_FAILED | 2016-11-13T19:58:49Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ControllerDeployment_Step2-w57753pd7fcj | | 2 | 9b4add07-2d5e-43e4-a243-9723a4472250 | OS::Heat::StructuredDeployment | UPDATE_FAILED | 2016-11-13T19:58:49Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ControllerDeployment_Step2-w57753pd7fcj | +-------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------+ Trancated output is below: Notice: /Firewall[999 drop all]: Dependency Package[yum-plugin-priorities] has failures: true Notice: Finished catalog run in 3.69 seconds ", "deploy_stderr": "exception: connect failed Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install yum-plugin-priorities' returned 1: Error: Nothing to do Error: /Stage[main]/Ceph::Repo/Package[yum-plugin-priorities]/ensure: change from absent to latest failed: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install yum-plugin-priorities' returned 1: Error: Nothing to do Note: Checked all the overcloud nodes - yum update took place on all and no updates are available for any node.