Bug 1394587

Summary: rhel-osp-director: OSP10 minor update fails: Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install yum-plugin-priorities' returned 1: Error: Nothing to do
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: John Fulton <johfulto>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: urgent Docs Contact: Derek <dcadzow>
Priority: urgent    
Version: 10.0 (Newton)CC: dbecker, gfidente, jcoufal, johfulto, jomurphy, jschluet, jslagle, kschinck, mandreou, mburns, mcornea, morazi, pgrist, rhel-osp-director-maint, sasha, scohen
Target Milestone: rcKeywords: Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-5.1.0-3.el7ost Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 16:32:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
enabled repos on nodes. none

Description Alexander Chuzhoy 2016-11-13 23:43:42 UTC
rhel-osp-director:   OSP10 minor update fails: Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install yum-plugin-priorities' returned 1: Error: Nothing to do 

Environment:
openstack-puppet-modules-9.3.0-1.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-1.6.el7ost.noarch
instack-undercloud-5.0.0-4.el7ost.noarch



Steps to reproduce:
1. Deploy a setup with external ceph.
2. Update the undercloud and attempt to update the overcloud.
Result:
[stack@undercloud-0 ~]$ heat stack-list 
+--------------------------------------+------------+---------------+----------------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        | updated_time         |     
+--------------------------------------+------------+---------------+----------------------+----------------------+     
| b065e928-307a-4a81-9cb2-91a13436b45a | overcloud  | UPDATE_FAILED | 2016-11-13T18:47:20Z | 2016-11-13T19:35:44Z |     
+--------------------------------------+------------+---------------+----------------------+----------------------+     
                                                                   
[stack@undercloud-0 ~]$ heat resource-list -n5 overcloud |grep -v COMPLE                             
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
heat deployment-show +-------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------+
| resource_name|physical_resource_id| resource_type| resource_status | updated_time | stack_name |+-------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------+                     
| AllNodesDeploySteps                       | dd0fad74-b976-455e-b7ac-35043575d785                                            | OS::TripleO::PostDeploySteps                                                                                        | UPDATE_FAILED   | 2016-11-13T19:56:44Z | overcloud                                                                                                            |                     
| 1                                         | b0d329f1-c62a-4506-9966-d60ceb23afd3                                            | OS::Heat::StructuredDeployment                                                                                      | UPDATE_FAILED   | 2016-11-13T19:58:46Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ControllerDeployment_Step2-w57753pd7fcj                                   |                     
| ControllerDeployment_Step2                | 72a34e44-e4a4-4fec-b1db-0ccbaec1b79e                                            | OS::Heat::StructuredDeploymentGroup                                                                                 | UPDATE_FAILED   | 2016-11-13T19:58:46Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox                                                                           |                     
| 1                                         | e4ed1f61-88cb-4413-95bd-e151923856f1                                            | OS::Heat::StructuredDeployment                                                                                      | UPDATE_FAILED   | 2016-11-13T19:58:47Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ComputeDeployment_Step2-inigzo63jrs4                                      |                     
| ComputeDeployment_Step2                   | dd4e57e8-bae4-4b8c-8d95-9d5c3a6e820f                                            | OS::Heat::StructuredDeploymentGroup                                                                                 | UPDATE_FAILED   | 2016-11-13T19:58:47Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox                                                                           |                     
| 0                                         | ef0d2f1c-d9c8-4dc2-ac31-0eeb53f4dff0                                            | OS::Heat::StructuredDeployment                                                                                      | UPDATE_FAILED   | 2016-11-13T19:58:49Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ComputeDeployment_Step2-inigzo63jrs4                                      |                     
| 0                                         | fddef5ea-0d20-4066-9b9c-7321f631d46b                                            | OS::Heat::StructuredDeployment                                                                                      | UPDATE_FAILED   | 2016-11-13T19:58:49Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ControllerDeployment_Step2-w57753pd7fcj                                   |                     
| 2                                         | 9b4add07-2d5e-43e4-a243-9723a4472250                                            | OS::Heat::StructuredDeployment                                                                                      | UPDATE_FAILED   | 2016-11-13T19:58:49Z | overcloud-AllNodesDeploySteps-m44wyck6s5ox-ControllerDeployment_Step2-w57753pd7fcj                                   |                     
+-------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------+                     



Trancated output is below:

Notice: /Firewall[999 drop all]: Dependency Package[yum-plugin-priorities] has failures: true                                                                                                                        
Notice: Finished catalog run in 3.69 seconds                                                                                                                                                                         
", "deploy_stderr": "exception: connect failed                                                                                                                                                                       
Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install yum-plugin-priorities' returned 1: Error: Nothing to do                                                                                     
Error: /Stage[main]/Ceph::Repo/Package[yum-plugin-priorities]/ensure: change from absent to latest failed: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install yum-plugin-priorities' returned 1: Error: Nothing to do                                                                                                                                                                                                      


Note: Checked all the overcloud nodes - yum update took place on all and no updates are available for any node.

Comment 5 Alexander Chuzhoy 2016-11-14 14:45:28 UTC
1)
Deploy command:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 2 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 0 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml -e /home/stack/extceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log



Update command:
yes ""| openstack overcloud update stack  -i overcloud


2)
I only see this issue with external ceph.
I was able to deploy with external ceph and to launch an instance.
The update fails. Here's the status from ceph:
    cluster 1c50eddf-56fb-470f-a72d-47a960f520a0
     health HEALTH_OK
     monmap e3: 3 mons at {ceph-node1=10.8.29.226:6789/0,ceph-node2=10.8.30.200:6789/0,ceph-node3=10.8.29.1:6789/0}
            election epoch 80, quorum 0,1,2 ceph-node3,ceph-node1,ceph-node2
     osdmap e882: 6 osds: 6 up, 6 in
      pgmap v605366: 1344 pgs, 8 pools, 109 MB data, 1484 objects
            605 MB used, 269 GB / 269 GB avail
                1344 active+clean


3)

This is how the update look on the console:
20:01:48 stdout: starting package update on stack overcloud
20:01:48 IN_PROGRESS
20:01:48 WAITING
20:01:48 on_breakpoint: [u'controller-2', u'controller-1', u'compute-0', u'controller-0', u'compute-1']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear d84a7955-43dd-4cfc-9260-b110b8d5458b), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
20:01:48 WAITING
20:01:48 completed: [u'compute-1']
20:01:48 on_breakpoint: [u'controller-2', u'controller-1', u'compute-0', u'controller-0']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear d2c056c4-d5fd-465d-b917-241a0769f425), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 WAITING
20:01:48 completed: [u'compute-1', u'controller-0']
20:01:48 on_breakpoint: [u'controller-2', u'controller-1', u'compute-0']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear 39145360-475b-45f6-b267-f40a57172a54), no=cancel update, C-c=quit interactive mode: WAITING
20:01:48 completed: [u'compute-1', u'compute-0', u'controller-0']
20:01:48 on_breakpoint: [u'controller-2', u'controller-1']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear 7917f829-1853-4d40-ba5a-513e0b01e3cb), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 WAITING
20:01:48 completed: [u'controller-1', u'compute-0', u'controller-0', u'compute-1']
20:01:48 on_breakpoint: [u'controller-2']
20:01:48 Breakpoint reached, continue? Regexp or Enter=proceed (will clear 8e57bd1a-ee3e-4da5-96a4-cff927dc33c6), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 IN_PROGRESS
20:01:48 FAILED
20:01:48 update finished with status FAILED




When I try to run "yum update" on any node now (I left the setup running over night) - I get:
---> Package policycoreutils-python.x86_64 0:2.5-8.el7 will be updated
---> Package policycoreutils-python.x86_64 0:2.5-9.el7 will be an update



[stack@undercloud-0 ~]$ for i in `nova list|awk -F"|" '/Running/ {print $(NF-1)}'|awk -F"=" '{print $NF}'`; do echo "####################################"; ssh heat-admin@$i "hostname; sudo grep -i update /var/log/yum.log";  done                                                                                                                                                                                                 
####################################                                                                                                                                                                             
compute-0.localdomain                                                                                                                                                                                            
Nov 13 19:57:52 Updated: 1:librados2-10.2.3-0.el7.x86_64                                                                                                                                                         
Nov 13 19:57:52 Updated: 1:python-rados-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:57:53 Updated: 1:librbd1-10.2.3-0.el7.x86_64                                                                                                                                                           
Nov 13 19:57:53 Updated: 1:librgw2-10.2.3-0.el7.x86_64                                                                                                                                                           
Nov 13 19:57:53 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64                                                                                                                                                        
Nov 13 19:57:54 Updated: 1:v8-3.14.5.10-25.el7.x86_64                                                                                                                                                            
Nov 13 19:57:54 Updated: leveldb-1.12.0-11.el7.x86_64                                                                                                                                                            
Nov 13 19:57:54 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64                                                                                                                                                     
Nov 13 19:57:54 Updated: 1:python-rbd-10.2.3-0.el7.x86_64                                                                                                                                                        
Nov 13 19:57:57 Updated: 1:ceph-common-10.2.3-0.el7.x86_64                                                                                                                                                       
Nov 13 19:58:04 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:58:05 Updated: 1:ceph-base-10.2.3-0.el7.x86_64                                                                                                                                                         
Nov 13 19:58:05 Updated: python-tables-3.2.0-1.el7.x86_64                                                                                                                                                        
Nov 13 19:58:05 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch                                                                                                                                                       
Nov 13 19:58:05 Updated: python-Bottleneck-0.7.0-1.el7.x86_64                                                                                                                                                    
Nov 13 19:58:06 Updated: rubygem-daemons-1.1.9-4.el7.noarch                                                                                                                                                      
Nov 13 19:58:06 Updated: rubygem-thin-1.6.4-3.el7.x86_64                                                                                                                                                         
Nov 13 19:58:06 Updated: jq-1.5-1.el7.x86_64                                                                                                                                                                     
Nov 13 19:58:07 Updated: python-pandas-0.17.1-1.el7.x86_64                                                                                                                                                       
Nov 13 19:58:09 Updated: python-unittest2-1.1.0-4.el7.noarch                                                                                                                                                     
Nov 13 19:58:09 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64                                                                                                                                                          
Nov 13 19:58:11 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64                                                                                                                                                          
Nov 13 19:58:11 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:58:17 Updated: mongodb-2.6.12-3.el7.x86_64                                                                                                                                                             
Nov 13 19:58:18 Updated: mongodb-server-2.6.12-3.el7.x86_64                                                                                                                                                      
Nov 13 19:58:18 Updated: python-unicodecsv-0.14.1-4.el7.noarch                                                                                                                                                   
Nov 13 19:58:18 Updated: 1:hiera-1.3.4-5.el7.noarch                                                                                                                                                              
Nov 13 19:58:18 Updated: galera-25.3.12-2.el7.x86_64                                                                                                                                                             
Nov 13 19:58:18 Updated: ruby-shadow-1.4.1-23.el7.x86_64                                                                                                                                                         
Nov 13 19:58:18 Updated: python-psutil-2.2.1-1.el7.x86_64                                                                                                                                                        
Nov 13 19:58:19 Updated: pysnmp-4.2.5-2.el7.noarch                                                                                                                                                               
Nov 13 19:58:19 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch                                                                                                                                                  
Nov 13 19:58:19 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64                                                                                                                                                   
Nov 13 19:58:19 Updated: python-contextlib2-0.5.1-2.el7.noarch                                                                                                                                                   
Nov 13 19:58:19 Updated: userspace-rcu-0.7.16-1.el7.x86_64                                                                                                                                                       
Nov 13 19:58:19 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch                                                                                                                                              
####################################                                                                                                                                                                             
compute-1.localdomain                                                                                                                                                                                            
Nov 13 19:57:39 Updated: 1:librados2-10.2.3-0.el7.x86_64                                                                                                                                                         
Nov 13 19:57:39 Updated: 1:python-rados-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:57:39 Updated: 1:librbd1-10.2.3-0.el7.x86_64                                                                                                                                                           
Nov 13 19:57:39 Updated: 1:librgw2-10.2.3-0.el7.x86_64                                                                                                                                                           
Nov 13 19:57:40 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64                                                                                                                                                        
Nov 13 19:57:40 Updated: 1:v8-3.14.5.10-25.el7.x86_64                                                                                                                                                            
Nov 13 19:57:40 Updated: leveldb-1.12.0-11.el7.x86_64                                                                                                                                                            
Nov 13 19:57:40 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64                                                                                                                                                     
Nov 13 19:57:40 Updated: 1:python-rbd-10.2.3-0.el7.x86_64                                                                                                                                                        
Nov 13 19:57:43 Updated: 1:ceph-common-10.2.3-0.el7.x86_64                                                                                                                                                       
Nov 13 19:57:50 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:57:51 Updated: 1:ceph-base-10.2.3-0.el7.x86_64                                                                                                                                                         
Nov 13 19:57:52 Updated: python-tables-3.2.0-1.el7.x86_64                                                                                                                                                        
Nov 13 19:57:52 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch                                                                                                                                                       
Nov 13 19:57:52 Updated: python-Bottleneck-0.7.0-1.el7.x86_64                                                                                                                                                    
Nov 13 19:57:52 Updated: rubygem-daemons-1.1.9-4.el7.noarch                                                                                                                                                      
Nov 13 19:57:52 Updated: rubygem-thin-1.6.4-3.el7.x86_64                                                                                                                                                         
Nov 13 19:57:52 Updated: jq-1.5-1.el7.x86_64                                                                                                                                                                     
Nov 13 19:57:54 Updated: python-pandas-0.17.1-1.el7.x86_64                                                                                                                                                       
Nov 13 19:57:55 Updated: python-unittest2-1.1.0-4.el7.noarch                                                                                                                                                     
Nov 13 19:57:55 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64                                                                                                                                                          
Nov 13 19:57:57 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64                                                                                                                                                          
Nov 13 19:57:57 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:58:03 Updated: mongodb-2.6.12-3.el7.x86_64                                                                                                                                                             
Nov 13 19:58:05 Updated: mongodb-server-2.6.12-3.el7.x86_64                                                                                                                                                      
Nov 13 19:58:05 Updated: python-unicodecsv-0.14.1-4.el7.noarch                                                                                                                                                   
Nov 13 19:58:05 Updated: 1:hiera-1.3.4-5.el7.noarch                                                                                                                                                              
Nov 13 19:58:05 Updated: galera-25.3.12-2.el7.x86_64                                                                                                                                                             
Nov 13 19:58:05 Updated: ruby-shadow-1.4.1-23.el7.x86_64                                                                                                                                                         
Nov 13 19:58:05 Updated: python-psutil-2.2.1-1.el7.x86_64                                                                                                                                                        
Nov 13 19:58:05 Updated: pysnmp-4.2.5-2.el7.noarch                                                                                                                                                               
Nov 13 19:58:05 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch                                                                                                                                                  
Nov 13 19:58:05 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64                                                                                                                                                   
Nov 13 19:58:05 Updated: python-contextlib2-0.5.1-2.el7.noarch                                                                                                                                                   
Nov 13 19:58:05 Updated: userspace-rcu-0.7.16-1.el7.x86_64                                                                                                                                                       
Nov 13 19:58:05 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch                                                                                                                                              
####################################                                                                                                                                                                             
controller-0.localdomain                                                                                                                                                                                         
Nov 13 19:45:43 Updated: 1:librados2-10.2.3-0.el7.x86_64                                                                                                                                                         
Nov 13 19:45:43 Updated: 1:python-rados-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:45:44 Updated: 1:librbd1-10.2.3-0.el7.x86_64                                                                                                                                                           
Nov 13 19:45:44 Updated: 1:librgw2-10.2.3-0.el7.x86_64                                                                                                                                                           
Nov 13 19:45:44 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64                                                                                                                                                        
Nov 13 19:45:45 Updated: 1:v8-3.14.5.10-25.el7.x86_64                                                                                                                                                            
Nov 13 19:45:45 Updated: leveldb-1.12.0-11.el7.x86_64                                                                                                                                                            
Nov 13 19:45:45 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64                                                                                                                                                     
Nov 13 19:45:45 Updated: 1:python-rbd-10.2.3-0.el7.x86_64                                                                                                                                                        
Nov 13 19:45:48 Updated: 1:ceph-common-10.2.3-0.el7.x86_64                                                                                                                                                       
Nov 13 19:45:54 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:45:55 Updated: 1:ceph-base-10.2.3-0.el7.x86_64                                                                                                                                                         
Nov 13 19:45:56 Updated: python-tables-3.2.0-1.el7.x86_64                                                                                                                                                        
Nov 13 19:45:56 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch                                                                                                                                                       
Nov 13 19:45:56 Updated: python-Bottleneck-0.7.0-1.el7.x86_64                                                                                                                                                    
Nov 13 19:45:56 Updated: rubygem-daemons-1.1.9-4.el7.noarch                                                                                                                                                      
Nov 13 19:45:56 Updated: rubygem-thin-1.6.4-3.el7.x86_64                                                                                                                                                         
Nov 13 19:45:56 Updated: jq-1.5-1.el7.x86_64                                                                                                                                                                     
Nov 13 19:45:58 Updated: python-pandas-0.17.1-1.el7.x86_64                                                                                                                                                       
Nov 13 19:45:59 Updated: python-unittest2-1.1.0-4.el7.noarch                                                                                                                                                     
Nov 13 19:46:00 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64                                                                                                                                                          
Nov 13 19:46:01 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64                                                                                                                                                          
Nov 13 19:46:01 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:46:07 Updated: mongodb-2.6.12-3.el7.x86_64                                                                                                                                                             
Nov 13 19:46:08 Updated: mongodb-server-2.6.12-3.el7.x86_64                                                                                                                                                      
Nov 13 19:46:08 Updated: python-unicodecsv-0.14.1-4.el7.noarch                                                                                                                                                   
Nov 13 19:46:08 Updated: 1:hiera-1.3.4-5.el7.noarch                                                                                                                                                              
Nov 13 19:46:09 Updated: galera-25.3.12-2.el7.x86_64                                                                                                                                                             
Nov 13 19:46:09 Updated: ruby-shadow-1.4.1-23.el7.x86_64                                                                                                                                                         
Nov 13 19:46:09 Updated: python-psutil-2.2.1-1.el7.x86_64                                                                                                                                                        
Nov 13 19:46:09 Updated: pysnmp-4.2.5-2.el7.noarch                                                                                                                                                               
Nov 13 19:46:09 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch                                                                                                                                                  
Nov 13 19:46:09 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64                                                                                                                                                   
Nov 13 19:46:09 Updated: python-contextlib2-0.5.1-2.el7.noarch                                                                                                                                                   
Nov 13 19:46:09 Updated: userspace-rcu-0.7.16-1.el7.x86_64                                                                                                                                                       
Nov 13 19:46:09 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch                                                                                                                                              
####################################                                                                                                                                                                             
controller-1.localdomain                                                                                                                                                                                         
Nov 13 19:50:31 Updated: 1:librados2-10.2.3-0.el7.x86_64                                                                                                                                                         
Nov 13 19:50:31 Updated: 1:python-rados-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:50:32 Updated: 1:librbd1-10.2.3-0.el7.x86_64                                                                                                                                                           
Nov 13 19:50:32 Updated: 1:librgw2-10.2.3-0.el7.x86_64                                                                                                                                                           
Nov 13 19:50:33 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64                                                                                                                                                        
Nov 13 19:50:33 Updated: 1:v8-3.14.5.10-25.el7.x86_64                                                                                                                                                            
Nov 13 19:50:33 Updated: leveldb-1.12.0-11.el7.x86_64                                                                                                                                                            
Nov 13 19:50:33 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64                                                                                                                                                     
Nov 13 19:50:33 Updated: 1:python-rbd-10.2.3-0.el7.x86_64                                                                                                                                                        
Nov 13 19:50:36 Updated: 1:ceph-common-10.2.3-0.el7.x86_64                                                                                                                                                       
Nov 13 19:50:43 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:50:44 Updated: 1:ceph-base-10.2.3-0.el7.x86_64                                                                                                                                                         
Nov 13 19:50:45 Updated: python-tables-3.2.0-1.el7.x86_64                                                                                                                                                        
Nov 13 19:50:45 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch                                                                                                                                                       
Nov 13 19:50:45 Updated: python-Bottleneck-0.7.0-1.el7.x86_64                                                                                                                                                    
Nov 13 19:50:45 Updated: rubygem-daemons-1.1.9-4.el7.noarch                                                                                                                                                      
Nov 13 19:50:45 Updated: rubygem-thin-1.6.4-3.el7.x86_64                                                                                                                                                         
Nov 13 19:50:45 Updated: jq-1.5-1.el7.x86_64                                                                                                                                                                     
Nov 13 19:50:47 Updated: python-pandas-0.17.1-1.el7.x86_64                                                                                                                                                       
Nov 13 19:50:48 Updated: python-unittest2-1.1.0-4.el7.noarch                                                                                                                                                     
Nov 13 19:50:48 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64                                                                                                                                                          
Nov 13 19:50:50 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64                                                                                                                                                          
Nov 13 19:50:50 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64                                                                                                                                                      
Nov 13 19:50:56 Updated: mongodb-2.6.12-3.el7.x86_64                                                                                                                                                             
Nov 13 19:50:57 Updated: mongodb-server-2.6.12-3.el7.x86_64
Nov 13 19:50:57 Updated: python-unicodecsv-0.14.1-4.el7.noarch
Nov 13 19:50:57 Updated: 1:hiera-1.3.4-5.el7.noarch
Nov 13 19:50:57 Updated: galera-25.3.12-2.el7.x86_64     
Nov 13 19:50:57 Updated: ruby-shadow-1.4.1-23.el7.x86_64                   
Nov 13 19:50:57 Updated: python-psutil-2.2.1-1.el7.x86_64               
Nov 13 19:50:57 Updated: pysnmp-4.2.5-2.el7.noarch
Nov 13 19:50:57 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch
Nov 13 19:50:57 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64
Nov 13 19:50:57 Updated: python-contextlib2-0.5.1-2.el7.noarch
Nov 13 19:50:58 Updated: userspace-rcu-0.7.16-1.el7.x86_64
Nov 13 19:50:58 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch
####################################
controller-2.localdomain
Nov 13 19:54:23 Updated: 1:librados2-10.2.3-0.el7.x86_64
Nov 13 19:54:23 Updated: 1:python-rados-10.2.3-0.el7.x86_64
Nov 13 19:54:24 Updated: 1:librbd1-10.2.3-0.el7.x86_64
Nov 13 19:54:24 Updated: 1:librgw2-10.2.3-0.el7.x86_64
Nov 13 19:54:25 Updated: 1:libcephfs1-10.2.3-0.el7.x86_64
Nov 13 19:54:25 Updated: 1:v8-3.14.5.10-25.el7.x86_64
Nov 13 19:54:25 Updated: leveldb-1.12.0-11.el7.x86_64
Nov 13 19:54:25 Updated: 1:python-cephfs-10.2.3-0.el7.x86_64
Nov 13 19:54:25 Updated: 1:python-rbd-10.2.3-0.el7.x86_64
Nov 13 19:54:28 Updated: 1:ceph-common-10.2.3-0.el7.x86_64
Nov 13 19:54:35 Updated: 1:ceph-selinux-10.2.3-0.el7.x86_64
Nov 13 19:54:36 Updated: 1:ceph-base-10.2.3-0.el7.x86_64
Nov 13 19:54:36 Updated: python-tables-3.2.0-1.el7.x86_64
Nov 13 19:54:36 Updated: 1:rubygem-rack-1.6.4-2.el7.noarch
Nov 13 19:54:36 Updated: python-Bottleneck-0.7.0-1.el7.x86_64
Nov 13 19:54:37 Updated: rubygem-daemons-1.1.9-4.el7.noarch
Nov 13 19:54:37 Updated: rubygem-thin-1.6.4-3.el7.x86_64
Nov 13 19:54:37 Updated: jq-1.5-1.el7.x86_64
Nov 13 19:54:38 Updated: python-pandas-0.17.1-1.el7.x86_64
Nov 13 19:54:40 Updated: python-unittest2-1.1.0-4.el7.noarch
Nov 13 19:54:40 Updated: 1:ceph-mon-10.2.3-0.el7.x86_64
Nov 13 19:54:41 Updated: 1:ceph-osd-10.2.3-0.el7.x86_64
Nov 13 19:54:41 Updated: 1:ceph-radosgw-10.2.3-0.el7.x86_64
Nov 13 19:54:48 Updated: mongodb-2.6.12-3.el7.x86_64
Nov 13 19:54:49 Updated: mongodb-server-2.6.12-3.el7.x86_64
Nov 13 19:54:49 Updated: python-unicodecsv-0.14.1-4.el7.noarch
Nov 13 19:54:49 Updated: 1:hiera-1.3.4-5.el7.noarch
Nov 13 19:54:49 Updated: galera-25.3.12-2.el7.x86_64
Nov 13 19:54:49 Updated: ruby-shadow-1.4.1-23.el7.x86_64
Nov 13 19:54:49 Updated: python-psutil-2.2.1-1.el7.x86_64
Nov 13 19:54:49 Updated: pysnmp-4.2.5-2.el7.noarch
Nov 13 19:54:49 Updated: rubygem-thread_safe-0.3.5-2.el7.noarch
Nov 13 19:54:49 Updated: libnetfilter_queue-1.0.2-2.el7.x86_64
Nov 13 19:54:49 Updated: python-contextlib2-0.5.1-2.el7.noarch
Nov 13 19:54:50 Updated: userspace-rcu-0.7.16-1.el7.x86_64
Nov 13 19:54:50 Updated: rubygem-rack-protection-1.5.3-3.el7.noarch





4) Attaching the file with enabled repos on nodes.

The issue reproduced twice only for ext ceph deployment.

Comment 6 Alexander Chuzhoy 2016-11-14 14:46:05 UTC
Created attachment 1220464 [details]
enabled repos on nodes.

Comment 10 James Slagle 2016-11-14 16:54:23 UTC
the issue here is with using puppet-ceph/manifests/repo.pp where it has:

package {'yum-plugin-priorities':              
  ensure => present
}

we need to either patch that manifest to not do that in cases where we are using RHEL or we should not be using this class from repo.pp.

We really don't want repo management done by puppet-ceph, as it is configured externally via subscription-manager or rhos-release for internal testing.

Comment 11 James Slagle 2016-11-14 16:55:25 UTC
this should be DFG:Ceph

Comment 12 Keith Schincke 2016-11-14 20:42:11 UTC
Can you attach your below two files?

/usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml

/home/stack/extceph.yaml

Comment 14 Alexander Chuzhoy 2016-11-14 22:24:27 UTC
[stack@undercloud-0 ~]$ cat /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml
# A Heat environment file which can be used to enable the
# use of an externally managed Ceph cluster.
resource_registry:
  OS::TripleO::Services::CephExternal: ../puppet/services/ceph-external.yaml
  OS::TripleO::Services::CephMon: OS::Heat::None
  OS::TripleO::Services::CephClient: OS::Heat::None
  OS::TripleO::Services::CephOSD: OS::Heat::None

parameter_defaults:
  # NOTE: These example parameters are required when using CephExternal
  #CephClusterFSID: '4b5c8c0a-ff60-454b-a1b4-9747aa737d19'
  CephClusterFSID: '1c50eddf-56fb-470f-a72d-47a960f520a0'
  #CephClientKey: 'AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ=='
  CephClientKey: 'AQBeskBWuKG8HhAA0iscCNXPa1yZkIb3y62Q7w=='
  #CephExternalMonHost: '172.16.1.7, 172.16.1.8'
  CephExternalMonHost: '10.8.29.226, 10.8.30.200, 10.8.29.1'

  # the following parameters enable Ceph backends for Cinder, Glance, Gnocchi and Nova
  NovaEnableRbdBackend: true
  CinderEnableRbdBackend: true
  CinderBackupBackend: ceph
  GlanceBackend: rbd
  GnocchiBackend: rbd
  # If the Ceph pools which host VMs, Volumes and Images do not match these
  # names OR the client keyring to use is not named 'openstack',  edit the
  # following as needed.
  NovaRbdPoolName: vms
  CinderRbdPoolName: volumes
  GlanceRbdPoolName: images
  GnocchiRbdPoolName: metrics
  CephClientUserName: openstack

  # finally we disable the Cinder LVM backend
  CinderEnableIscsiBackend: false

  # Backward compatibility setting, will be removed in the future
  CephAdminKey: ''

Comment 15 Alexander Chuzhoy 2016-11-14 22:24:49 UTC
[stack@undercloud-0 ~]$ cat extceph.yaml
parameter_defaults:
  ExtraConfig:
    ceph::conf::args:
      client/rbd_default_features:
        value: "1"

Comment 16 John Fulton 2016-11-15 01:39:45 UTC
Sasha,

Thanks for building the env for me to troubleshoot. Would you
please try to reproduce this error but make the following
changes:

a. Use the following extceph.yaml:

[stack@undercloud-0 ~]$ cat extceph.yaml 
parameter_defaults:
  ExtraConfig:
    ceph::conf::args:
      client/rbd_default_features:
        value: "1"
    ceph::profile::params::manage_repo: false
[stack@undercloud-0 ~]$

Note that I'm passing the manage_repo flag set to false to keep
repo.pp affecting your repo. 

b. Use the following deploy command:

openstack overcloud deploy --debug --templates --libvirt-type kvm \
--ntp-server clock.redhat.com \
--neutron-network-type vxlan \
--neutron-tunnel-types vxlan \
--control-scale 3 --control-flavor controller-d75f3dec-c7-5f88-9d4c-3fea1bf9c484 \
--compute-scale 2 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppeceph-external.yaml \
-e /home/stack/extceph.yaml \
-e virt/hostnames.yml \
-e virt/network/network-environment.yaml \
--log-file overcloud_deployment_48.log

The change in the above is to not include storage-environment.yaml.
Since you're not installing ceph on your overcloud it shouldn't be
included. I also removed the ceph-storage scale and flavor simply
because they're not applicable in an external ceph situation. 
Otherwise I just added some new lines for readability.

Comment 20 Alexander Chuzhoy 2016-11-15 17:52:46 UTC
So I tried the following:

deploy_command:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 2 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml -e /home/stack/extceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log


[stack@undercloud-0 ~]$ cat extceph.yaml
parameter_defaults:
  ExtraConfig:
    ceph::conf::args:
      client/rbd_default_features:
        value: "1"
    ceph::profile::params::manage_repo: false


The update failed with:
Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install ceph' returned 1: Error: Nothing to do                                                                                                      
Error: /Stage[main]/Ceph/Package[ceph]/ensure: change from absent to latest failed: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install ceph' returned 1: Error: Nothing to do

Comment 21 Giulio Fidente 2016-11-15 18:26:45 UTC
Alexander, yep that is because we also nees to customize the list of packages to be installed, as per comment #19.

That has been added to the upstream patch https://review.openstack.org/397819

Comment 22 John Fulton 2016-11-16 12:32:17 UTC
Alexander, the patch has had another change (sorry but there was an indentation problem that may explain the most recent problem you had yesterday). Would you mind running the test again with the following (make sure it's patchset 5)? 

 https://review.openstack.org/#/c/397819/5/puppet/services/ceph-external.yaml

Comment 23 John Fulton 2016-11-16 20:23:13 UTC
Alexander, as per our conversation, I understand that dropping in the new ceph-external.yaml and re-running your test resolved the issue. The fix merged into Octa and the backport needs to merge into Newton next. Thanks. --John

 https://review.openstack.org/#/c/398475/

Comment 24 John Fulton 2016-11-17 14:36:37 UTC
The newton backport merged. Updating BZ to POST.

Comment 28 Alexander Chuzhoy 2016-11-29 19:46:57 UTC
Verified:
Environment:
openstack-tripleo-heat-templates-5.1.0-6.el7ost.noarch

The reported issue doesn't reproduce.

Comment 30 errata-xmlrpc 2016-12-14 16:32:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html