Bug 1388977

Summary:	rhel-osp-director: After a successful minor update the Ceph nodes aren't updated.
Product:	Red Hat OpenStack	Reporter:	Alexander Chuzhoy <sasha>
Component:	openstack-tripleo-heat-templates	Assignee:	Lukas Bezdicka <lbezdick>
Status:	CLOSED ERRATA	QA Contact:	Alexander Chuzhoy <sasha>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	10.0 (Newton)	CC:	dbecker, jcoufal, jschluet, jslagle, mandreou, mburns, morazi, rhel-osp-director-maint
Target Milestone:	rc	Keywords:	Triaged
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-5.0.0-1.4.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-14 16:25:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2016-10-26 15:29:07 UTC

rhel-osp-director:   After a successful minor update the Ceph nodes aren't updated.

Environment:
openstack-puppet-modules-9.3.0-0.20161003154825.8c758d6.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20161008015357.0d3e3e3.1.el7ost.noarch
instack-undercloud-5.0.0-2.el7ost.noarch

Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 1 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 2 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log


2. Minor update the setup
03:26:23 IN_PROGRESS
03:26:23 IN_PROGRESS
03:26:23 IN_PROGRESS
03:26:23 IN_PROGRESS
03:26:23 COMPLETE
03:26:23 update finished with status COMPLETE

[stack@undercloud-0 ~]$ heat stack-list                             
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+----------------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        | updated_time         |
+--------------------------------------+------------+-----------------+----------------------+----------------------+
| 3eda1753-c3b7-42c4-b32f-788462b6dbb8 | overcloud  | UPDATE_COMPLETE | 2016-10-26T04:10:17Z | 2016-10-26T06:07:15Z |
+--------------------------------------+------------+-----------------+----------------------+----------------------+



3. Check that yum update took place on OC nodes:

[stack@undercloud-0 ~]$ nova list
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks            |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+
| 4db520ca-edbf-439d-bb56-04673d95ce15 | ceph-0       | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| f9fe6011-1491-4334-a1ce-798e64bbdaeb | ceph-1       | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| 66470382-ae12-4c92-9723-c50964b0fef8 | compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 9a593210-2891-46bb-a892-cd386344e6f8 | controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
| 3279fc65-b436-410d-ba21-5463e20c0aa2 | controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| b246347e-c549-46a0-ac45-71c1bd70eea5 | controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.14 |
+--------------------------------------+--------------+--------+------------+-------------+---------------------+

[stack@undercloud-0 ~]$ ssh heat-admin.2.9 "sudo yum update"                                                   
Loaded plugins: product-id, search-disabled-repos, subscription-manager                                              
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.      
Resolving Dependencies                                                                                               
--> Running transaction check                                                                                        
---> Package kernel.x86_64 0:3.10.0-514.el7 will be installed                                                        
---> Package kernel-devel.x86_64 0:3.10.0-514.el7 will be installed                                                  
---> Package kernel-headers.x86_64 0:3.10.0-513.el7 will be updated                                                  
---> Package kernel-headers.x86_64 0:3.10.0-514.el7 will be an update                                                
---> Package kernel-tools.x86_64 0:3.10.0-513.el7 will be updated                                                    
---> Package kernel-tools.x86_64 0:3.10.0-514.el7 will be an update                                                  
---> Package kernel-tools-libs.x86_64 0:3.10.0-513.el7 will be updated                                               
---> Package kernel-tools-libs.x86_64 0:3.10.0-514.el7 will be an update                                             
---> Package liberasurecode.x86_64 0:1.2.0-1.el7ost will be updated                                                  
---> Package liberasurecode.x86_64 0:1.2.0-2.el7ost will be an update                                                
---> Package openstack-cinder.noarch 1:9.0.0-2.el7ost will be updated                                                
---> Package openstack-cinder.noarch 1:9.0.0-3.el7ost will be an update                                              
---> Package openstack-dashboard-theme.noarch 1:10.0.0-2.el7ost will be updated                                      
---> Package openstack-dashboard-theme.noarch 1:10.0.0-5.el7ost will be an update                                    
--> Processing Dependency: openstack-dashboard = 1:10.0.0-5.el7ost for package: 1:openstack-dashboard-theme-10.0.0-5.el7ost.noarch
---> Package openstack-gnocchi-api.noarch 0:3.0.1-0.20161017182629.a44d88f.el7ost will be updated                                 
---> Package openstack-gnocchi-api.noarch 0:3.0.1-1.1.el7ost will be an update                                                    
---> Package openstack-gnocchi-carbonara.noarch 0:3.0.1-0.20161017182629.a44d88f.el7ost will be updated                           
---> Package openstack-gnocchi-carbonara.noarch 0:3.0.1-1.1.el7ost will be an update                                              
---> Package openstack-gnocchi-common.noarch 0:3.0.1-0.20161017182629.a44d88f.el7ost will be updated                              
---> Package openstack-gnocchi-common.noarch 0:3.0.1-1.1.el7ost will be an update                                                 
---> Package openstack-gnocchi-indexer-sqlalchemy.noarch 0:3.0.1-0.20161017182629.a44d88f.el7ost will be updated                  
---> Package openstack-gnocchi-indexer-sqlalchemy.noarch 0:3.0.1-1.1.el7ost will be an update                                     
---> Package openstack-gnocchi-metricd.noarch 0:3.0.1-0.20161017182629.a44d88f.el7ost will be updated                             
---> Package openstack-gnocchi-metricd.noarch 0:3.0.1-1.1.el7ost will be an update                                                
---> Package openstack-gnocchi-statsd.noarch 0:3.0.1-0.20161017182629.a44d88f.el7ost will be updated                              
---> Package openstack-gnocchi-statsd.noarch 0:3.0.1-1.1.el7ost will be an update                                                 
---> Package openstack-heat-api.noarch 1:7.0.0-3.el7ost will be updated                                                           
---> Package openstack-heat-api.noarch 1:7.0.0-4.el7ost will be an update                                                         
---> Package openstack-heat-api-cfn.noarch 1:7.0.0-3.el7ost will be updated                                                       
---> Package openstack-heat-api-cfn.noarch 1:7.0.0-4.el7ost will be an update                                                     
---> Package openstack-heat-api-cloudwatch.noarch 1:7.0.0-3.el7ost will be updated                                                
---> Package openstack-heat-api-cloudwatch.noarch 1:7.0.0-4.el7ost will be an update                                              
---> Package openstack-heat-common.noarch 1:7.0.0-3.el7ost will be updated                                                        
---> Package openstack-heat-common.noarch 1:7.0.0-4.el7ost will be an update                                                      
---> Package openstack-heat-engine.noarch 1:7.0.0-3.el7ost will be updated                                                        
---> Package openstack-heat-engine.noarch 1:7.0.0-4.el7ost will be an update                                                      
---> Package openstack-manila-ui.noarch 0:2.5.1-2.el7ost will be updated                                                          
---> Package openstack-manila-ui.noarch 0:2.5.1-5.el7ost will be an update                                                        
---> Package openstack-mistral-api.noarch 0:3.0.1-0.20161013000829.6356bce.el7ost will be updated                                 
---> Package openstack-mistral-api.noarch 0:3.0.2-2.el7ost will be an update                                                      
---> Package openstack-mistral-common.noarch 0:3.0.1-0.20161013000829.6356bce.el7ost will be updated                              
---> Package openstack-mistral-common.noarch 0:3.0.2-2.el7ost will be an update                                                   
---> Package openstack-mistral-engine.noarch 0:3.0.1-0.20161013000829.6356bce.el7ost will be updated                              
---> Package openstack-mistral-engine.noarch 0:3.0.2-2.el7ost will be an update                                                   
---> Package openstack-mistral-executor.noarch 0:3.0.1-0.20161013000829.6356bce.el7ost will be updated                            
---> Package openstack-mistral-executor.noarch 0:3.0.2-2.el7ost will be an update                                                 
---> Package openstack-neutron.noarch 1:9.0.0-1.4.el7ost will be updated                                                          
---> Package openstack-neutron.noarch 1:9.0.0-1.5.el7ost will be an update                                                        
---> Package openstack-neutron-common.noarch 1:9.0.0-1.4.el7ost will be updated                                                   
---> Package openstack-neutron-common.noarch 1:9.0.0-1.5.el7ost will be an update                                                 
---> Package openstack-neutron-metering-agent.noarch 1:9.0.0-1.4.el7ost will be updated                                           
---> Package openstack-neutron-metering-agent.noarch 1:9.0.0-1.5.el7ost will be an update                                         
---> Package openstack-neutron-ml2.noarch 1:9.0.0-1.4.el7ost will be updated                                                      
---> Package openstack-neutron-ml2.noarch 1:9.0.0-1.5.el7ost will be an update                                                    
---> Package openstack-neutron-openvswitch.noarch 1:9.0.0-1.4.el7ost will be updated                                              
---> Package openstack-neutron-openvswitch.noarch 1:9.0.0-1.5.el7ost will be an update                                            
---> Package openstack-neutron-sriov-nic-agent.noarch 1:9.0.0-1.4.el7ost will be updated                                          
---> Package openstack-neutron-sriov-nic-agent.noarch 1:9.0.0-1.5.el7ost will be an update                                        
---> Package openstack-nova-api.noarch 1:14.0.1-3.el7ost will be updated                                                          
---> Package openstack-nova-api.noarch 1:14.0.1-4.el7ost will be an update                                                        
---> Package openstack-nova-cert.noarch 1:14.0.1-3.el7ost will be updated                                                         
---> Package openstack-nova-cert.noarch 1:14.0.1-4.el7ost will be an update                                                       
---> Package openstack-nova-common.noarch 1:14.0.1-3.el7ost will be updated                                                       
---> Package openstack-nova-common.noarch 1:14.0.1-4.el7ost will be an update                                                     
---> Package openstack-nova-compute.noarch 1:14.0.1-3.el7ost will be updated                                                      
---> Package openstack-nova-compute.noarch 1:14.0.1-4.el7ost will be an update                                                    
---> Package openstack-nova-conductor.noarch 1:14.0.1-3.el7ost will be updated                                                    
---> Package openstack-nova-conductor.noarch 1:14.0.1-4.el7ost will be an update                                                  
---> Package openstack-nova-console.noarch 1:14.0.1-3.el7ost will be updated                                                      
---> Package openstack-nova-console.noarch 1:14.0.1-4.el7ost will be an update                                                    
---> Package openstack-nova-novncproxy.noarch 1:14.0.1-3.el7ost will be updated                                                   
---> Package openstack-nova-novncproxy.noarch 1:14.0.1-4.el7ost will be an update                                                 
---> Package openstack-nova-scheduler.noarch 1:14.0.1-3.el7ost will be updated                                                    
---> Package openstack-nova-scheduler.noarch 1:14.0.1-4.el7ost will be an update                                                  
---> Package puppet-apache.noarch 0:1.10.0-0.20161015235625.cf2ff7e.el7ost will be updated                                        
---> Package puppet-apache.noarch 0:1.10.0-0.20161019180724.b31bd45.el7ost will be an update                                      
---> Package puppet-barbican.noarch 0:9.4.0-1.el7ost will be updated                                                              
---> Package puppet-barbican.noarch 0:9.4.0-2.el7ost will be an update                                                            
---> Package puppet-cassandra.noarch 0:2.0.2-0.20161015225641.782ccbc.el7ost will be updated
---> Package puppet-cassandra.noarch 0:2.0.2-0.20161020062951.f2d90db.el7ost will be an update
---> Package puppet-ceph.noarch 0:2.2.1-2.el7ost will be updated
---> Package puppet-ceph.noarch 0:2.2.1-3.el7ost will be an update
---> Package puppet-collectd.noarch 0:5.1.0-0.20161018180615.b26caaa.el7ost will be updated
---> Package puppet-collectd.noarch 0:5.1.0-0.20161019184217.c31ec50.el7ost will be an update
---> Package puppet-corosync.noarch 0:5.0.0-0.20161013095720.950324c.el7ost will be updated
---> Package puppet-corosync.noarch 0:5.0.0-0.20161019182612.48e1e2b.el7ost will be an update
---> Package puppet-gnocchi.noarch 0:9.4.0-1.el7ost will be updated
---> Package puppet-gnocchi.noarch 0:9.4.0-2.el7ost will be an update
---> Package puppet-ironic.noarch 0:9.4.0-1.el7ost will be updated
---> Package puppet-ironic.noarch 0:9.4.0-1.1.el7ost will be an update
---> Package puppet-kafka.noarch 0:2.1.0-3.061ef74git.el7ost will be updated
---> Package puppet-kafka.noarch 0:2.1.0-4.061ef74git.el7ost will be an update
---> Package puppet-swift.noarch 0:9.4.1-3.el7ost will be updated
---> Package puppet-swift.noarch 0:9.4.2-1.el7ost will be an update
---> Package puppet-tripleo.noarch 0:5.3.0-1.el7ost will be updated
---> Package puppet-tripleo.noarch 0:5.3.0-3.el7ost will be an update
---> Package python-ceilometerclient.noarch 0:2.6.1-1.el7ost will be updated
---> Package python-ceilometerclient.noarch 0:2.6.1-2.el7ost will be an update
---> Package python-cinder.noarch 1:9.0.0-2.el7ost will be updated
---> Package python-cinder.noarch 1:9.0.0-3.el7ost will be an update
---> Package python-django-horizon.noarch 1:10.0.0-2.el7ost will be updated
---> Package python-django-horizon.noarch 1:10.0.0-5.el7ost will be an update
---> Package python-gnocchi.noarch 0:3.0.1-0.20161017182629.a44d88f.el7ost will be updated
---> Package python-gnocchi.noarch 0:3.0.1-1.1.el7ost will be an update
---> Package python-keystoneauth1.noarch 0:2.12.1-1.el7ost will be updated
---> Package python-keystoneauth1.noarch 0:2.12.2-1.el7ost will be an update
---> Package python-muranoclient.noarch 0:0.11.0-1.el7ost will be updated
---> Package python-muranoclient.noarch 0:0.11.1-1.el7ost will be an update
---> Package python-neutron.noarch 1:9.0.0-1.4.el7ost will be updated
---> Package python-neutron.noarch 1:9.0.0-1.5.el7ost will be an update
---> Package python-neutron-tests.noarch 1:9.0.0-1.4.el7ost will be updated
---> Package python-neutron-tests.noarch 1:9.0.0-1.5.el7ost will be an update
---> Package python-nova.noarch 1:14.0.1-3.el7ost will be updated
---> Package python-nova.noarch 1:14.0.1-4.el7ost will be an update
---> Package python-openstack-mistral.noarch 0:3.0.1-0.20161013000829.6356bce.el7ost will be updated
---> Package python-openstack-mistral.noarch 0:3.0.2-2.el7ost will be an update
---> Package python-perf.x86_64 0:3.10.0-513.el7 will be updated
---> Package python-perf.x86_64 0:3.10.0-514.el7 will be an update
--> Running transaction check
---> Package openstack-dashboard.noarch 1:10.0.0-5.el7ost will be installed
--> Finished Dependency Resolution
^CKilled by signal 2.



After checking yum.log on OC nodes, I see that the ceph nodes weren't updated.

Comment 2 Marios Andreou 2016-10-27 15:12:00 UTC

seems similar/related to https://bugzilla.redhat.com/show_bug.cgi?id=1384640

Comment 3 Alexander Chuzhoy 2016-10-31 22:49:25 UTC

Reproduced.

Only the ceph nodes (the compute and swift nodes were updated).

Comment 4 Marios Andreou 2016-11-07 18:17:24 UTC

pointing to the stable/newton review and moving to POST as it has merged at https://review.openstack.org/#/c/392259/4

Comment 6 Alexander Chuzhoy 2016-11-16 03:21:28 UTC

Verified:
Environment:
openstack-tripleo-heat-templates-5.0.0-1.7.el7ost.noarch

The reported issue doesn't reproduce.

Comment 8 errata-xmlrpc 2016-12-14 16:25:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html