Bug 1316588
| Summary: | Upgrade undercloud fails on keystone error | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Adriano Petrich <apetrich> | ||||||
| Component: | instack-undercloud | Assignee: | Brad P. Crochet <brad> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Udi Kalifon <ukalifon> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 8.0 (Liberty) | CC: | achernet, brad, emacchi, hbrock, jslagle, mandreou, mburns, rhel-osp-director-maint, sasha | ||||||
| Target Milestone: | --- | Keywords: | Automation, AutomationBlocker | ||||||
| Target Release: | 8.0 (Liberty) | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | instack-undercloud-2.2.6-1.el7ost | Doc Type: | Bug Fix | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2016-04-07 21:49:13 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
FYI the Warning Scope(Class[Keystone::Endpoint]) is not relevant. mmagr added this warning in puppet-keystone. # systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/httpd.service.d
└─openstack-dashboard.conf
Active: failed (Result: exit-code) since Thu 2016-03-10 09:18:19 EST; 2h 16min ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 10502 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
Process: 10499 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
Process: 10399 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py compress --force (code=exited, status=0/SUCCESS)
Process: 10370 ExecStartPre=/usr/bin/python /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear (code=exited, status=0/SUCCESS)
Main PID: 10499 (code=exited, status=1/FAILURE)
Mar 10 09:18:19 instack.localdomain httpd[10499]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:35357
Mar 10 09:18:19 instack.localdomain httpd[10499]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:35357
Mar 10 09:18:19 instack.localdomain httpd[10499]: no listening sockets available, shutting down
Mar 10 09:18:19 instack.localdomain httpd[10499]: AH00015: Unable to open logs
Mar 10 09:18:19 instack.localdomain systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Mar 10 09:18:19 instack.localdomain kill[10502]: kill: cannot find process ""
Mar 10 09:18:19 instack.localdomain systemd[1]: httpd.service: control process exited, code=exited status=1
Mar 10 09:18:19 instack.localdomain systemd[1]: Failed to start The Apache HTTP Server.
Mar 10 09:18:19 instack.localdomain systemd[1]: Unit httpd.service entered failed state.
Mar 10 09:18:19 instack.localdomain systemd[1]: httpd.service failed.
# cat error_log
[Thu Mar 10 05:18:05.876608 2016] [core:notice] [pid 32026] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Mar 10 05:18:05.877725 2016] [suexec:notice] [pid 32026] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 10 05:18:05.886882 2016] [auth_digest:notice] [pid 32026] AH01757: generating secret for digest authentication ...
[Thu Mar 10 05:18:05.892957 2016] [mpm_prefork:notice] [pid 32026] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Mar 10 05:18:05.892999 2016] [core:notice] [pid 32026] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Mar 10 05:18:19.391863 2016] [mpm_prefork:notice] [pid 32026] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Mar 10 05:18:31.567266 2016] [core:notice] [pid 32548] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Mar 10 05:18:31.568935 2016] [suexec:notice] [pid 32548] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 10 05:18:31.578477 2016] [auth_digest:notice] [pid 32548] AH01757: generating secret for digest authentication ...
[Thu Mar 10 05:18:31.583899 2016] [mpm_prefork:notice] [pid 32548] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Mar 10 05:18:31.583923 2016] [core:notice] [pid 32548] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Mar 10 05:25:12.616067 2016] [mpm_prefork:notice] [pid 32548] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Mar 10 05:25:24.030143 2016] [core:notice] [pid 6335] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Mar 10 05:25:24.031137 2016] [suexec:notice] [pid 6335] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 10 05:25:24.040256 2016] [auth_digest:notice] [pid 6335] AH01757: generating secret for digest authentication ...
[Thu Mar 10 05:25:24.045257 2016] [mpm_prefork:notice] [pid 6335] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Mar 10 05:25:24.045287 2016] [core:notice] [pid 6335] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Mar 10 05:25:27.576710 2016] [mpm_prefork:notice] [pid 6335] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Mar 10 05:25:39.310434 2016] [core:notice] [pid 6566] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Mar 10 05:25:39.312170 2016] [suexec:notice] [pid 6566] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 10 05:25:39.326976 2016] [auth_digest:notice] [pid 6566] AH01757: generating secret for digest authentication ...
[Thu Mar 10 05:25:39.334254 2016] [mpm_prefork:notice] [pid 6566] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Thu Mar 10 05:25:39.334288 2016] [core:notice] [pid 6566] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu Mar 10 06:58:19.462589 2016] [mpm_prefork:notice] [pid 6566] AH00170: caught SIGWINCH, shutting down gracefully
Created attachment 1134956 [details]
httpd logs
Created attachment 1134957 [details]
/var/log/messages
added httpd logs and /var/log/messages on request from irc Reproduced the issue: Environment: openstack-tripleo-heat-templates-0.8.10-2.el7ost.noarch instack-undercloud-2.2.5-1.el7ost.noarch openstack-puppet-modules-7.0.14-1.el7ost.noarch *** Bug 1318029 has been marked as a duplicate of this bug. *** With the latest puddles the upgrade is successful. Verified in python-tripleoclient-0.3.4-1.el7ost.noarch. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0604.html |
Description of problem: When doing an upgrade from 7.3 to 8 the undercloud upgrade fails Version-Release number of selected component (if applicable): upgrade 7.3 to 8 How reproducible: very Steps to Reproduce: 1. do a full install on 7.3 (undercloud and overcloud 2. do a rhos-release 8 -d 3. update the packages with yum upgrade -y 4. source stackrc 5. openstack undercloud install #upgrade the undercloud Actual results: It fails Notice: /File[/etc/ironic/ironic.conf]/seluser: seluser changed 'unconfined_u' to 'system_u' Notice: /Stage[main]/Keystone::Service/Service[keystone]/ensure: ensure changed 'running' to 'stopped' Broadcast message from systemd-journald (Thu 2016-03-10 09:18:09 EST): haproxy[9505]: proxy keystone_admin has no server available! Broadcast message from systemd-journald (Thu 2016-03-10 09:18:09 EST): haproxy[9505]: proxy keystone_public has no server available! Error: Could not start Service[httpd]: Execution of '/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details. Wrapped exception: Execution of '/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details. Error: /Stage[main]/Apache::Service/Service[httpd]/ensure: change from stopped to running failed: Could not start Service[httpd]: Execution of '/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details. Notice: /Stage[main]/Apache::Service/Service[httpd]: Triggered 'refresh' from 3 events Notice: /Stage[main]/Keystone/Anchor[keystone_started]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Keystone/Anchor[keystone_started]: Skipping because of failed dependencies Error: Could not prefetch keystone_user provider 'openstack': undefined method `collect' for nil:NilClass Notice: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Skipping because of failed dependencies Error: Could not prefetch keystone_service provider 'openstack': undefined method `collect' for nil:NilClass Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]: Skipping because of failed dependencies Error: Could not prefetch keystone_role provider 'openstack': undefined method `collect' for nil:NilClass Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]: Skipping because of failed dependencies Notice: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]: Skipping because of failed dependencies Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_user[aodh]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_user[aodh]: Skipping because of failed dependencies Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Skipping because of failed dependencies Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_service[aodh::alarming]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_service[aodh::alarming]: Skipping because of failed dependencies Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Skipping because of failed dependencies Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]: Skipping because of failed dependencies Notice: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Skipping because of failed dependencies Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]: Skipping because of failed dependencies Notice: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]: Skipping because of failed dependencies Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]: Skipping because of failed dependencies Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/notify_nova_on_port_status_changes]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/notify_nova_on_port_status_changes]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/region_name]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/region_name]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/project_name]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/project_name]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/auth_plugin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/auth_plugin]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/send_events_interval]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/send_events_interval]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/username]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/username]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/password]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/password]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/nova_region_name]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/nova_region_name]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/project_domain_id]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/project_domain_id]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/notify_nova_on_port_data_changes]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/notify_nova_on_port_data_changes]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/tenant_name]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/tenant_name]: Skipping because of failed dependencies Notice: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]: Skipping because of failed dependencies Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/user_domain_id]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/user_domain_id]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/auth_url]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[nova/auth_url]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/nova_url]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server::Notifications/Neutron_config[DEFAULT/nova_url]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Agents::Dhcp/Service[neutron-dhcp-service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Agents::Dhcp/Service[neutron-dhcp-service]: Skipping because of failed dependencies Notice: /Stage[main]/Main/Keystone_service[nova::computev3]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Main/Keystone_service[nova::computev3]: Skipping because of failed dependencies Notice: Puppet::Provider::Openstack: endpoint service is unavailable. Will retry for up to 9 seconds. Error: Could not prefetch keystone_endpoint provider 'openstack': undefined method `each' for nil:NilClass Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_endpoint[regionOne/novav3::computev3]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_endpoint[regionOne/novav3::computev3]: Skipping because of failed dependencies Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_endpoint[regionOne/aodh::alarming]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_endpoint[regionOne/aodh::alarming]: Skipping because of failed dependencies Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity]: Skipping because of failed dependencies Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_endpoint[regionOne/nova::compute]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_endpoint[regionOne/nova::compute]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_endpoint[regionOne/heat::orchestration]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_endpoint[regionOne/heat::orchestration]: Skipping because of failed dependencies Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_endpoint[regionOne/glance::image]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_endpoint[regionOne/glance::image]: Skipping because of failed dependencies Notice: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_endpoint[regionOne/swift::object-store]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_endpoint[regionOne/swift::object-store]: Skipping because of failed dependencies Notice: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_endpoint[regionOne/ironic::baremetal]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_endpoint[regionOne/ironic::baremetal]: Skipping because of failed dependencies Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_endpoint[regionOne/ceilometer::metering]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_endpoint[regionOne/ceilometer::metering]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_endpoint[regionOne/neutron::network]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_endpoint[regionOne/neutron::network]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Skipping because of failed dependencies Notice: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Skipping because of failed dependencies Notice: Puppet::Provider::Openstack: project service is unavailable. Will retry for up to 9 seconds. Error: Could not prefetch keystone_tenant provider 'openstack': undefined method `collect' for nil:NilClass Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]: Skipping because of failed dependencies Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]: Skipping because of failed dependencies Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]: Skipping because of failed dependencies Notice: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_user_role[aodh@service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_user_role[aodh@service]: Skipping because of failed dependencies Notice: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user_role[nova@service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user_role[nova@service]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user_role[heat@service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user_role[heat@service]: Skipping because of failed dependencies Notice: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user_role[swift@service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user_role[swift@service]: Skipping because of failed dependencies Notice: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user_role[ceilometer@service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user_role[ceilometer@service]: Skipping because of failed dependencies Notice: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user_role[ironic@service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user_role[ironic@service]: Skipping because of failed dependencies Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Skipping because of failed dependencies Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_user_domain_name]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_user_domain_name]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]: Skipping because of failed dependencies Notice: Puppet::Provider::Openstack: domain service is unavailable. Will retry for up to 9 seconds. Error: Could not prefetch keystone_domain provider 'openstack': undefined method `collect' for nil:NilClass Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[heat_stack]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[heat_stack]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_user[heat_admin::heat_stack]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_user[heat_admin::heat_stack]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[heat_admin::heat_stack@::heat_stack]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[heat_admin::heat_stack@::heat_stack]: Skipping because of failed dependencies Notice: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user_role[glance@service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user_role[glance@service]: Skipping because of failed dependencies Notice: /Stage[main]/Glance::Registry/Service[glance-registry]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Glance::Registry/Service[glance-registry]: Skipping because of failed dependencies Notice: /Stage[main]/Glance::Api/Service[glance-api]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Glance::Api/Service[glance-api]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user_role[neutron@service]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user_role[neutron@service]: Skipping because of failed dependencies Notice: /Stage[main]/Neutron::Server/Service[neutron-server]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Neutron::Server/Service[neutron-server]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Deps/Anchor[heat::config::end]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Deps/Anchor[heat::config::end]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Deps/Anchor[heat::db::begin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Deps/Anchor[heat::db::begin]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.1]/Mysql_user[heat.2.1]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.1]/Mysql_user[heat.2.1]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.1]/Mysql_grant[heat.2.1/heat.*]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.0.2.1]/Mysql_grant[heat.2.1/heat.*]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Deps/Anchor[heat::db::end]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Deps/Anchor[heat::db::end]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::begin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::begin]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Engine/Service[heat-engine]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Engine/Service[heat-engine]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Api/Service[heat-api]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Api/Service[heat-api]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Skipping because of failed dependencies Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Dependency Service[httpd] has failures: true Warning: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Skipping because of failed dependencies Notice: Finished catalog run in 118.21 seconds + rc=6 + set -e + echo 'puppet apply exited with exit code 6' puppet apply exited with exit code 6 + '[' 6 '!=' 2 -a 6 '!=' 0 ']' + exit 6 [2016-03-10 09:19:29,913] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6] [2016-03-10 09:19:29,913] (os-refresh-config) [ERROR] Aborting... Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 807, in install _run_orc(instack_env) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 691, in _run_orc _run_live_command(args, instack_env, 'os-refresh-config') File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 366, in _run_live_command raise RuntimeError('%s failed. See log for details.' % name) RuntimeError: os-refresh-config failed. See log for details. Command 'instack-install-undercloud' returned non-zero exit status 1 Expected results: upgrade works Additional info: there's an warning up the install process that is Warning: Scope(Class[Keystone::Endpoint]): In Mitaka, the default value of $keystone::endpoint::version will change to ''. To avoid this warning, please set the version parameter. Not sure if that is relevant or not