Bug 1334047

Summary: rhel-osp-director: Upgrading undercloud with SSL 7.3->8.0 fails with: Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: undefined method `empty?' for nil:NilClass
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: dbecker, mburns, mcornea, morazi, rhel-osp-director-maint
Target Milestone: ga   
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-09 17:43:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
install-undercloud.log none

Description Alexander Chuzhoy 2016-05-07 15:20:47 UTC 
rhel-osp-director: Upgrading undercloud with SSL 7.3->8.0 fails with:  Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: undefined method `empty?' for nil:NilClass 


Environment:
openstack-tripleo-heat-templates-kilo-0.8.14-9.el7ost.noarch
instack-undercloud-2.2.7-4.el7ost.noarch
openstack-puppet-modules-7.0.17-1.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-9.el7ost.noarch

Steps to reproduce:
1. Deploy 7.3 with SSL.
2. Attempt to upgrade the undercloud to 8.0 (run "openstack undercloud upgrade").


Result:
2016-05-06 22:05:35,560 INFO: Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Dependency Keystone_user[nova] has failures: true                                                                        
2016-05-06 22:05:35,562 INFO: Warning: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Skipping because of failed dependencies                                                                                 
2016-05-06 22:05:37,513 INFO: Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Triggered 'refresh' from 16 events                                                                                       
2016-05-06 22:05:49,295 INFO: Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: undefined method `empty?' for nil:NilClass             
2016-05-06 22:05:51,941 INFO: Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/description: description changed '' to 'Tenant for the openstack services'                                        
2016-05-06 22:05:52,818 INFO: Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user_role[nova@service]: Dependency Keystone_user[nova] has failures: true                                                                                                                                                                                                                 
2016-05-06 22:05:52,818 INFO: Warning: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user_role[nova@service]: Skipping because of failed dependencies     
2016-05-06 22:05:52,819 INFO: Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user_role[heat@service]: Dependency Keystone_user[heat] has failures: true               
2016-05-06 22:05:52,819 INFO: Warning: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user_role[heat@service]: Skipping because of failed dependencies                        
2016-05-06 22:05:52,819 INFO: Notice: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user_role[swift@service]: Dependency Keystone_user[swift] has failures: true           
2016-05-06 22:05:52,819 INFO: Warning: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user_role[swift@service]: Skipping because of failed dependencies                     
2016-05-06 22:05:52,834 INFO: Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user_role[ceilometer@service]: Dependency Keystone_user[ceilometer] has failures: true                                                                                                                                                                                                            
2016-05-06 22:05:52,835 INFO: Warning: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user_role[ceilometer@service]: Skipping because of failed dependencies      
2016-05-06 22:05:52,842 INFO: Notice: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user_role[ironic@service]: Dependency Keystone_user[ironic] has failures: true       
2016-05-06 22:05:52,842 INFO: Warning: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user_role[ironic@service]: Skipping because of failed dependencies                  
2016-05-06 22:06:11,299 INFO: Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: undefined method `empty?' for nil:NilClass                                                        
2016-05-06 22:06:11,493 INFO: Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Dependency Keystone_user[admin] has failures: true                                                        
2016-05-06 22:06:11,495 INFO: Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Skipping because of failed dependencies                                                                  
2016-05-06 22:06:11,499 INFO: Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_user_domain_name]: Dependency Keystone_user[admin] has failures: true                                            
2016-05-06 22:06:11,500 INFO: Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_user_domain_name]: Skipping because of failed dependencies                                                      
2016-05-06 22:06:11,501 INFO: Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]: Dependency Keystone_user[admin] has failures: true                                       
2016-05-06 22:06:11,501 INFO: Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]: Skipping because of failed dependencies                                                 
2016-05-06 22:06:11,505 INFO: Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]: Dependency Keystone_user[admin] has failures: true                                                
2016-05-06 22:06:11,505 INFO: Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]: Skipping because of failed dependencies                                                          
2016-05-06 22:06:12,501 INFO: Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[heat_stack]: Dependency Keystone_user[admin] has failures: true                                                            
2016-05-06 22:06:12,502 INFO: Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[heat_stack]: Skipping because of failed dependencies                                                                      
2016-05-06 22:06:12,504 INFO: Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_user[heat_admin::heat_stack]: Dependency Keystone_user[admin] has failures: true                                                  
2016-05-06 22:06:12,504 INFO: Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_user[heat_admin::heat_stack]: Skipping because of failed dependencies                                                            
2016-05-06 22:06:12,507 INFO: Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[heat_admin::heat_stack@::heat_stack]: Dependency Keystone_user[admin] has failures: true                                
2016-05-06 22:06:12,508 INFO: Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[heat_admin::heat_stack@::heat_stack]: Skipping because of failed dependencies                                          
2016-05-06 22:06:12,514 INFO: Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user_role[glance@service]: Dependency Keystone_user[glance] has failures: true       
2016-05-06 22:06:12,514 INFO: Warning: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user_role[glance@service]: Skipping because of failed dependencies                  
2016-05-06 22:06:12,520 INFO: Notice: /Stage[main]/Glance::Registry/Service[glance-registry]: Dependency Keystone_user[glance] has failures: true                                                                    
2016-05-06 22:06:12,521 INFO: Warning: /Stage[main]/Glance::Registry/Service[glance-registry]: Skipping because of failed dependencies                                                                               
2016-05-06 22:06:13,849 INFO: Notice: /Stage[main]/Glance::Registry/Service[glance-registry]: Triggered 'refresh' from 10 events                                                                                     
2016-05-06 22:06:13,857 INFO: Notice: /Stage[main]/Glance::Api/Service[glance-api]: Dependency Keystone_user[glance] has failures: true                                                                              
2016-05-06 22:06:13,858 INFO: Warning: /Stage[main]/Glance::Api/Service[glance-api]: Skipping because of failed dependencies                                                                                         
2016-05-06 22:06:15,916 INFO: Notice: /Stage[main]/Glance::Api/Service[glance-api]: Triggered 'refresh' from 26 events                                                                                               
2016-05-06 22:06:15,926 INFO: Notice: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user_role[neutron@service]: Dependency Keystone_user[neutron] has failures: true   
2016-05-06 22:06:15,926 INFO: Warning: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user_role[neutron@service]: Skipping because of failed dependencies               
2016-05-06 22:06:15,931 INFO: Notice: /Stage[main]/Neutron::Server/Service[neutron-server]: Dependency Keystone_user[nova] has failures: true                                                                        
2016-05-06 22:06:15,932 INFO: Notice: /Stage[main]/Neutron::Server/Service[neutron-server]: Dependency Keystone_user[neutron] has failures: true                                                                     
2016-05-06 22:06:15,933 INFO: Warning: /Stage[main]/Neutron::Server/Service[neutron-server]: Skipping because of failed dependencies                                                                                 
2016-05-06 22:06:21,141 INFO: Notice: /Stage[main]/Neutron::Server/Service[neutron-server]: Triggered 'refresh' from 24 events                                                                                       
2016-05-06 22:06:23,454 INFO: Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[neutron-ovs-agent-service]: Triggered 'refresh' from 3 events                                                                   
2016-05-06 22:06:23,492 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::config::end]: Dependency Keystone_user[admin] has failures: true                                                                          
2016-05-06 22:06:23,492 INFO: Warning: /Stage[main]/Heat::Deps/Anchor[heat::config::end]: Skipping because of failed dependencies                                                                                    
2016-05-06 22:06:23,493 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::config::end]: Triggered 'refresh' from 21 events                                                                                          
2016-05-06 22:06:23,496 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::db::begin]: Dependency Keystone_user[admin] has failures: true                                                                            
2016-05-06 22:06:23,497 INFO: Warning: /Stage[main]/Heat::Deps/Anchor[heat::db::begin]: Skipping because of failed dependencies                                                                                      
2016-05-06 22:06:23,521 INFO: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Dependency Keystone_user[admin] has failures: true                                            
2016-05-06 22:06:23,522 INFO: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies                                                      
2016-05-06 22:06:23,522 INFO: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Dependency Keystone_user[admin] has failures: true 
2016-05-06 22:06:23,522 INFO: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies           
2016-05-06 22:06:23,522 INFO: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Dependency Keystone_user[admin] has failures: true                                                                                                                                                                                                              
2016-05-06 22:06:23,523 INFO: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies   
2016-05-06 22:06:23,559 INFO: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.1]/Mysql_user[heat.2.1]: Dependency Keystone_user[admin] has failures: true                                                                                                                                                                                                      
2016-05-06 22:06:23,560 INFO: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.1]/Mysql_user[heat.2.1]: Skipping because of failed dependencies                                                                                                                                                                                                                
2016-05-06 22:06:23,560 INFO: Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.1]/Mysql_grant[heat.2.1/heat.*]: Dependency Keystone_user[admin] has failures: true                                                                                                                                                                                              
2016-05-06 22:06:23,560 INFO: Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.1]/Mysql_grant[heat.2.1/heat.*]: Skipping because of failed dependencies                                                                                                                                                                                                        
2016-05-06 22:06:23,589 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::db::end]: Dependency Keystone_user[admin] has failures: true                                                                              
2016-05-06 22:06:23,590 INFO: Warning: /Stage[main]/Heat::Deps/Anchor[heat::db::end]: Skipping because of failed dependencies                                                                                        
2016-05-06 22:06:23,590 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::begin]: Dependency Keystone_user[admin] has failures: true
2016-05-06 22:06:23,590 INFO: Warning: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::begin]: Skipping because of failed dependencies
2016-05-06 22:06:23,590 INFO: Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Dependency Keystone_user[admin] has failures: true
2016-05-06 22:06:23,590 INFO: Warning: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Skipping because of failed dependencies
2016-05-06 22:06:30,618 INFO: Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Triggered 'refresh' from 1 events
2016-05-06 22:06:30,619 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Dependency Keystone_user[admin] has failures: true
2016-05-06 22:06:30,619 INFO: Warning: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Skipping because of failed dependencies
2016-05-06 22:06:30,619 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Triggered 'refresh' from 1 events
2016-05-06 22:06:30,621 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Dependency Keystone_user[admin] has failures: true
2016-05-06 22:06:30,622 INFO: Warning: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Skipping because of failed dependencies
2016-05-06 22:06:30,622 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Triggered 'refresh' from 2 events
2016-05-06 22:06:30,624 INFO: Notice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Dependency Keystone_user[admin] has failures: true
2016-05-06 22:06:30,624 INFO: Warning: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Skipping because of failed dependencies
2016-05-06 22:06:30,701 INFO: Notice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Triggered 'refresh' from 1 events
2016-05-06 22:06:30,701 INFO: Notice: /Stage[main]/Heat::Engine/Service[heat-engine]: Dependency Keystone_user[admin] has failures: true
2016-05-06 22:06:30,702 INFO: Warning: /Stage[main]/Heat::Engine/Service[heat-engine]: Skipping because of failed dependencies
2016-05-06 22:06:30,937 INFO: Notice: /Stage[main]/Heat::Engine/Service[heat-engine]: Triggered 'refresh' from 1 events
2016-05-06 22:06:30,938 INFO: Notice: /Stage[main]/Heat::Api/Service[heat-api]: Dependency Keystone_user[admin] has failures: true
2016-05-06 22:06:30,938 INFO: Warning: /Stage[main]/Heat::Api/Service[heat-api]: Skipping because of failed dependencies
2016-05-06 22:06:31,046 INFO: Notice: /Stage[main]/Heat::Api/Service[heat-api]: Triggered 'refresh' from 1 events
2016-05-06 22:06:31,046 INFO: Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Dependency Keystone_user[admin] has failures: true
2016-05-06 22:06:31,046 INFO: Warning: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Skipping because of failed dependencies
2016-05-06 22:06:31,242 INFO: Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Triggered 'refresh' from 1 events
2016-05-06 22:06:31,242 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Dependency Keystone_user[admin] has failures: true
2016-05-06 22:06:31,243 INFO: Warning: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Skipping because of failed dependencies
2016-05-06 22:06:31,243 INFO: Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 4 events
2016-05-06 22:06:32,778 INFO: Notice: Finished catalog run in 411.83 seconds
2016-05-06 22:06:37,953 INFO: + rc=6
2016-05-06 22:06:37,953 INFO: + set -e
2016-05-06 22:06:37,953 INFO: + echo 'puppet apply exited with exit code 6'
2016-05-06 22:06:37,953 INFO: puppet apply exited with exit code 6
2016-05-06 22:06:37,953 INFO: + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
2016-05-06 22:06:37,954 INFO: + exit 6
2016-05-06 22:06:37,954 INFO: [2016-05-06 22:06:37,952] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6]
2016-05-06 22:06:37,954 INFO:
2016-05-06 22:06:37,954 INFO: [2016-05-06 22:06:37,952] (os-refresh-config) [ERROR] Aborting...


Expected result:
Undercloud successfully upgraded.

The issue is reproducible.
Comment 2 Alexander Chuzhoy 2016-05-07 15:23:26 UTC 
Created attachment 1154875 [details]
install-undercloud.log
Comment 3 Alexander Chuzhoy 2016-05-09 17:43:28 UTC 

*** This bug has been marked as a duplicate of bug 1326644 ***