Bug 1382112

Summary: rhel-osp-director: Deployment of undercloud fails with puppet errors.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: James Slagle <jslagle>
Status: CLOSED WORKSFORME QA Contact: Omri Hochman <ohochman>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: bnemec, dbecker, jslagle, mburns, morazi, rhel-osp-director-maint, sasha
Target Milestone: ga   
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-07 17:23:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
install-undercloud.log none

Description Alexander Chuzhoy 2016-10-05 18:53:43 UTC 
rhel-osp-director:   Deployment of undercloud+ssl fails with puppet errors
Environment:
instack-undercloud-5.0.0-0.20160929132429.56e65af.el7ost.noarch


Steps to reproduce:
1. Configure undercloud.conf to use ssl
2. run openstack undercloud install


Result:
2016-10-05 11:23:41,139 INFO: 2016-10-05 11:23:41 - Error: Could not start Service[openvswitch]: Execution of '/bin/systemctl start openvswitch' returned 1: A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.
2016-10-05 11:23:41,139 INFO: 2016-10-05 11:23:41 - Error: /Stage[setup]/Vswitch::Ovs/Service[openvswitch]/ensure: change from stopped to running failed: Could not start Service[openvswitch]: Execution of '/bin/systemctl start openvswitch' returned 1: A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.
2016-10-05 11:23:43,188 INFO: 2016-10-05 11:23:43 - Notice: /Stage[main]/Mistral::Logging/Oslo::Log[mistral_config]/Mistral_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:43,188 INFO: 2016-10-05 11:23:43 - Warning: /Stage[main]/Mistral::Logging/Oslo::Log[mistral_config]/Mistral_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:43,474 INFO: 2016-10-05 11:23:43 - Notice: /Stage[main]/Glance::Api::Logging/Oslo::Log[glance_api_config]/Glance_api_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:43,474 INFO: 2016-10-05 11:23:43 - Warning: /Stage[main]/Glance::Api::Logging/Oslo::Log[glance_api_config]/Glance_api_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:43,534 INFO: 2016-10-05 11:23:43 - Notice: /Stage[main]/Glance::Registry::Logging/Oslo::Log[glance_registry_config]/Glance_registry_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:43,534 INFO: 2016-10-05 11:23:43 - Warning: /Stage[main]/Glance::Registry::Logging/Oslo::Log[glance_registry_config]/Glance_registry_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:43,809 INFO: 2016-10-05 11:23:43 - Notice: /Stage[main]/Neutron::Logging/Oslo::Log[neutron_config]/Neutron_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:43,809 INFO: 2016-10-05 11:23:43 - Warning: /Stage[main]/Neutron::Logging/Oslo::Log[neutron_config]/Neutron_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:43,913 INFO: 2016-10-05 11:23:43 - Notice: /Stage[main]/Glance::Cache::Logging/Oslo::Log[glance_cache_config]/Glance_cache_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:43,913 INFO: 2016-10-05 11:23:43 - Warning: /Stage[main]/Glance::Cache::Logging/Oslo::Log[glance_cache_config]/Glance_cache_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:44,331 INFO: 2016-10-05 11:23:44 - Notice: /Stage[main]/Heat::Logging/Oslo::Log[heat_config]/Heat_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:44,332 INFO: 2016-10-05 11:23:44 - Warning: /Stage[main]/Heat::Logging/Oslo::Log[heat_config]/Heat_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:44,643 INFO: 2016-10-05 11:23:44 - Notice: /Stage[main]/Swift::Proxy::Catch_errors/Concat::Fragment[swift_catch_errors]/Concat_fragment[swift_catch_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:44,643 INFO: 2016-10-05 11:23:44 - Warning: /Stage[main]/Swift::Proxy::Catch_errors/Concat::Fragment[swift_catch_errors]/Concat_fragment[swift_catch_errors]: Skipping because of failed dependencies
2016-10-05 11:23:45,210 INFO: 2016-10-05 11:23:45 - Notice: /Stage[main]/Ironic::Inspector::Logging/Ironic_inspector_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:45,210 INFO: 2016-10-05 11:23:45 - Warning: /Stage[main]/Ironic::Inspector::Logging/Ironic_inspector_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:45,256 INFO: 2016-10-05 11:23:45 - Notice: /Stage[main]/Ironic::Logging/Oslo::Log[ironic_config]/Ironic_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:45,256 INFO: 2016-10-05 11:23:45 - Warning: /Stage[main]/Ironic::Logging/Oslo::Log[ironic_config]/Ironic_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:45,950 INFO: 2016-10-05 11:23:45 - Notice: /Stage[main]/Keystone::Logging/Oslo::Log[keystone_config]/Keystone_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:45,950 INFO: 2016-10-05 11:23:45 - Warning: /Stage[main]/Keystone::Logging/Oslo::Log[keystone_config]/Keystone_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:48,071 INFO: 2016-10-05 11:23:48 - Notice: /Stage[main]/Nova::Logging/Oslo::Log[nova_config]/Nova_config[DEFAULT/publish_errors]: Dependency Service[openvswitch] has failures: true
2016-10-05 11:23:48,071 INFO: 2016-10-05 11:23:48 - Warning: /Stage[main]/Nova::Logging/Oslo::Log[nova_config]/Nova_config[DEFAULT/publish_errors]: Skipping because of failed dependencies
2016-10-05 11:23:48,100 INFO: 2016-10-05 11:23:48 - Error: /Stage[main]/Rabbitmq/Rabbitmq_plugin[rabbitmq_management]: Provider rabbitmqplugins is not functional on this host
2016-10-05 11:23:48,121 INFO: 2016-10-05 11:23:48 - Error: Could not prefetch mysql_grant provider 'mysql': Command mysql is missing
2016-10-05 11:23:48,404 INFO: 2016-10-05 11:23:48 - Error: Could not prefetch keystone_service provider 'openstack': Insufficient credentials to authenticate
2016-10-05 11:23:48,433 INFO: 2016-10-05 11:23:48 - Error: Could not prefetch keystone_role provider 'openstack': Insufficient credentials to authenticate
2016-10-05 11:23:48,753 INFO: 2016-10-05 11:23:48 - Error: Could not prefetch keystone_endpoint provider 'openstack': Insufficient credentials to authenticate
2016-10-05 11:23:49,428 INFO: 2016-10-05 11:23:49 - Error: Could not prefetch keystone_tenant provider 'openstack': Insufficient credentials to authenticate
2016-10-05 11:23:49,749 INFO: 2016-10-05 11:23:49 - Error: Could not prefetch keystone_domain provider 'openstack': Insufficient credentials to authenticate
2016-10-05 11:23:50,921 INFO: 2016-10-05 11:23:50 - Error: Could not find a suitable provider for mysql_datadir
2016-10-05 11:23:50,921 INFO: 2016-10-05 11:23:50 - Error: Could not find a suitable provider for mysql_database
2016-10-05 11:23:50,921 INFO: 2016-10-05 11:23:50 - Error: Could not find a suitable provider for mysql_user
2016-10-05 11:23:58,273 INFO: [2016-10-05 11:23:58,272] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2016-10-05 11:23:58,274 INFO: [2016-10-05 11:23:58,273] (os-refresh-config) [ERROR] Aborting...


Expected result:
The undercloud deployment should succeed.
Comment 1 Alexander Chuzhoy 2016-10-05 18:55:22 UTC 
Created attachment 1207666 [details]
install-undercloud.log
Comment 2 James Slagle 2016-10-06 13:20:37 UTC 
ben, can you have a look at this one?
Comment 3 Alexander Chuzhoy 2016-10-06 14:19:42 UTC 
Just saw the same error on a setup without ssl.
Comment 4 Ben Nemec 2016-10-06 15:19:15 UTC 
Error: Could not start Service[openvswitch]: Execution of '/bin/systemctl start openvswitch' returned 1: A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.

That makes it sound like something is pretty wrong on the system.  Possibly this is rhel 7.3 related since we aren't seeing the same thing upstream on centos?  Can you let me know what image you're basing these underclouds on?
Comment 6 James Slagle 2016-10-06 17:01:03 UTC 
i'm trying to reproduce this using today's puddle:
http://openstack.etherpad.corp.redhat.com/rhel-osp-10-puddle-2016-10-06

will report back on the result
Comment 7 James Slagle 2016-10-06 18:10:36 UTC 
(In reply to James Slagle from comment #6)
> i'm trying to reproduce this using today's puddle:
> http://openstack.etherpad.corp.redhat.com/rhel-osp-10-puddle-2016-10-06
> 
> will report back on the result

could not reproduce either with ssl or non-ssl using the 2016-10-06 puddle. sasha, please see if it still reproduces for you using the newer puddle.
Comment 8 Ben Nemec 2016-10-06 21:06:33 UTC 
(In reply to James Slagle from comment #7)
> could not reproduce either with ssl or non-ssl using the 2016-10-06 puddle.
> sasha, please see if it still reproduces for you using the newer puddle.

Just a quick note that I could not reproduce this locally either using the same puddle.  If it happens again, it would be good to check the journal for things like openvswitch that have failed to start and see if there are log messages that might point to what is wrong.
Comment 9 Alexander Chuzhoy 2016-10-07 13:28:39 UTC 
I don't reproduce it now neither.
Comment 10 James Slagle 2016-10-07 17:23:36 UTC 
(In reply to Alexander Chuzhoy from comment #9)
> I don't reproduce it now neither.

thanks for checking, closing this one out then