Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2; Unable to find image /rhosp12/openstack-ceilometer-notification-docker:2017-09-22.5' locally Environment: openstack-tripleo-image-elements-7.0.0-0.20170914203419.526772d.el7ost.noarch openstack-tripleo-heat-templates-7.0.1-0.20170919183703.el7ost.noarch openstack-tripleo-common-7.6.1-0.20170913110336.el7ost.noarch python-tripleoclient-7.3.1-0.20170919185615.331e481.el7ost.noarch puppet-tripleo-7.4.1-0.20170916161117.4a16f48.el7ost.noarch openstack-tripleo-common-containers-7.6.1-0.20170913110336.el7ost.noarch openstack-tripleo-validations-7.4.0-0.20170918153806.efe8a72.el7ost.noarch openstack-tripleo-puppet-elements-7.0.0-0.20170914203705.2094778.el7ost.noarch openstack-tripleo-ui-7.4.1-0.20170915125211.1031ea6.el7ost.noarch instack-undercloud-7.4.1-0.20170912115418.el7ost.noarch instack-7.0.1-0.20170911221228.d0b15e2.el7ost.noarch container-selinux-2.21-2.gitba103ac.el7.noarch python-heat-agent-1.4.1-0.20170824121708.ff31eb2.el7ost.noarch puppet-heat-11.3.1-0.20170825182847.1700009.el7ost.noarch openstack-heat-engine-9.0.1-0.20170913170444.0c64134.el7ost.noarch openstack-heat-common-9.0.1-0.20170913170444.0c64134.el7ost.noarch Steps to reproduce: 1. Deploy UC: Successfully deploy UC using latest puddle 2017-09-22.5 2. Deploy OC - Failed cat overcloud_deploy.sh openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/rhos12.yaml \ -e /home/stack/templates/ironic.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/ironic.yaml \ --log-file overcloud_deployment_0.log cat /home/stack/templates/nodes_data.yaml parameter_defaults: ControllerCount: '3' OvercloudControlFlavor: 'controller' ComputeCount: '2' OvercloudComputeFlavor: 'compute' NtpServer: ["clock.redhat.com","clock2.redhat.com"] Result: (undercloud) [stack@undercloud-0 ~]$ openstack stack failures list overcloud overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: d9aeb51d-9008-4097-b187-a7c883bdf50b status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 deploy_stdout: | ... ], "changed": false, "failed": true, "failed_when_result": true } to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/0181074a-08f5-4b6f-8280-16886194b53f_playbook.retry PLAY RECAP ********************************************************************* localhost : ok=7 changed=1 unreachable=0 failed=1 (truncated, view all with --long) deploy_stderr: | (undercloud) [stack@undercloud-0 ~]$ heat stack-list --show-nested -f "status=FAILED" WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead +--------------------------------------+------------------------------------------------------------------------------------+---------------+----------------------+--------------+--------------------------------------+----------------------------------+ | id | stack_name | stack_status | creation_time | updated_time | parent | project | +--------------------------------------+------------------------------------------------------------------------------------+---------------+----------------------+--------------+--------------------------------------+----------------------------------+ | 7efb8529-3e6a-4b78-a5c1-14680b752caa | overcloud | CREATE_FAILED | 2017-09-25T19:32:32Z | None | None | 89ebc78384564867af9309816ee82e22 | | 51ccb8fa-cf82-4caa-9ac8-8d33c9ed9697 | overcloud-AllNodesDeploySteps-jjd2ujwjevnb | CREATE_FAILED | 2017-09-25T19:44:34Z | None | 7efb8529-3e6a-4b78-a5c1-14680b752caa | 89ebc78384564867af9309816ee82e22 | | 7d7b3e9b-7158-47bc-9c03-6502aa9ffc0b | overcloud-AllNodesDeploySteps-jjd2ujwjevnb-ControllerDeployment_Step3-6xcvc6mbtp6j | CREATE_FAILED | 2017-09-25T19:57:03Z | None | 51ccb8fa-cf82-4caa-9ac8-8d33c9ed9697 | 89ebc78384564867af9309816ee82e22 | +--------------------------------------+------------------------------------------------------------------------------------+---------------+----------------------+--------------+--------------------------------------+----------------------------------+ (undercloud) [stack@undercloud-0 ~]$ heat resource-list --nested-depth 5 overcloud | grep FAILED WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead | AllNodesDeploySteps | 51ccb8fa-cf82-4caa-9ac8-8d33c9ed9697 | OS::TripleO::PostDeploySteps | CREATE_FAILED | 2017-09-25T19:32:33Z | overcloud | | ControllerDeployment_Step3 | 7d7b3e9b-7158-47bc-9c03-6502aa9ffc0b | OS::TripleO::DeploymentSteps | CREATE_FAILED | 2017-09-25T19:44:35Z | overcloud-AllNodesDeploySteps-jjd2ujwjevnb | | 0 | d9aeb51d-9008-4097-b187-a7c883bdf50b | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-09-25T19:57:04Z | overcloud-AllNodesDeploySteps-jjd2ujwjevnb-ControllerDeployment_Step3-6xcvc6mbtp6j | (undercloud) [stack@undercloud-0 ~]$ heat resource-show 7d7b3e9b-7158-47bc-9c03-6502aa9ffc0b 0 WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead +------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attributes | { | | | "deploy_stdout": null, | | | "deploy_stderr": null, | | | "deploy_status_code": null | | | } | | creation_time | 2017-09-25T19:57:04Z | | description | | | links | http://192.168.24.1:8004/v1/89ebc78384564867af9309816ee82e22/stacks/overcloud-AllNodesDeploySteps-jjd2ujwjevnb-ControllerDeployment_Step3-6xcvc6mbtp6j/7d7b3e9b-7158-47bc-9c03-6502aa9ffc0b/resources/0 (self) | | | http://192.168.24.1:8004/v1/89ebc78384564867af9309816ee82e22/stacks/overcloud-AllNodesDeploySteps-jjd2ujwjevnb-ControllerDeployment_Step3-6xcvc6mbtp6j/7d7b3e9b-7158-47bc-9c03-6502aa9ffc0b (stack) | | logical_resource_id | 0 | | parent_resource | ControllerDeployment_Step3 | | physical_resource_id | d9aeb51d-9008-4097-b187-a7c883bdf50b | | required_by | | | resource_name | 0 | | resource_status | CREATE_FAILED | | resource_status_reason | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 | | resource_type | OS::Heat::StructuredDeployment | | updated_time | 2017-09-25T19:57:04Z | +------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (undercloud) [stack@undercloud-0 ~]$ het deployment-show d9aeb51d-9008-4097-b187-a7c883bdf50b -bash: het: command not found (undercloud) [stack@undercloud-0 ~]$ heat deployment-show d9aeb51d-9008-4097-b187-a7c883bdf50b WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "28d741cc-952c-4007-80d5-80883c3eef89", "config_id": "0181074a-08f5-4b6f-8280-16886194b53f", "output_values": { "deploy_stdout": "\nPLAY [localhost] ***************************************************************\n\nTASK [Gathering Facts] *********************************************************\nok: [localhost]\n\nTASK [Write the config_step hieradata] *****************************************\nchanged: [localhost]\n\nTASK [Run puppet host configuration for step 3] ********************************\nok: [localhost]\n\nTASK [debug] *******************************************************************\nok: [localhost] => {\n \"(outputs.stderr|default('')).split('\\n')|union(outputs.stdout_lines|default([]))\": [\n \"Warning: Facter: Could not retrieve fact='rabbitmq_nodename', resolution='<anonymous>': undefined method `[]' for nil:NilClass\", \n \"Warning: Undefined variable 'deploy_config_name'; \", \n \" (file & line not available)\", \n \"Warning: ModuleLoader: module 'cinder' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n \"Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/cinder/manifests/db.pp\\\", 64]:[\\\"/etc/puppet/modules/cinder/manifests/init.pp\\\", 385]\", \n \" (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')\", \n \"Warning: Scope(Class[Cinder]): host is deprecated, has no effect and will be removed in a future release, use backend_host instead\", \n \"Warning: Scope(Class[Cinder]): cinder::rabbit_host, cinder::rabbit_hosts, cinder::rabbit_password, cinder::rabbit_port, cinder::rabbit_userid, cinder::rabbit_virtual_host and cinder::rpc_backend are deprecated. Please use cinder::default_transport_url instead.\", \n \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/cinder/manifests/config.pp\\\", 38]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/cinder.pp\\\", 121]\", \n \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/cinder/manifests/api.pp\\\", 214]:[\\\"/var/lib/tripleo-config/puppet_step_config.pp\\\", 2]\", \n \"Warning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n \"Warning: Scope(Class[Heat]): heat::rabbit_host, heat::rabbit_hosts, heat::rabbit_password, heat::rabbit_port, heat::rabbit_userid, heat::rabbit_virtual_host and heat::rpc_backend are deprecated. Please use heat::default_transport_url instead.\", \n \"Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n \"Warning: Scope(Class[Neutron]): neutron::rabbit_host, neutron::rabbit_hosts, neutron::rabbit_password, neutron::rabbit_port, neutron::rabbit_user, neutron::rabbit_virtual_host and neutron::rpc_backend are deprecated. Please use neutron::default_transport_url instead.\", \n \"Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at [\\\"/etc/puppet/modules/neutron/manifests/init.pp\\\", 524]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/neutron/server.pp\\\", 104]\", \n \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 55]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 56]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 66]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n \"Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at [\\\"/etc/puppet/modules/ntp/manifests/init.pp\\\", 76]:[\\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\\", 29]\", \n \"Warning: ModuleLoader: module 'ssh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n \"Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n \"Warning: ModuleLoader: module 'oslo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n \"Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n \"Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\", \n \"Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend\", \n \"Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.\", \n \"Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 8.50 seconds\", \n \"Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/api_paste_config]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/storage_availability_zone]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/default_availability_zone]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/host]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Cinder_config[DEFAULT/enable_v3_api]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Glance/Cinder_config[DEFAULT/glance_api_servers]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Glance/Cinder_config[DEFAULT/glance_api_version]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Api/Cinder_config[DEFAULT/osapi_volume_listen]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Api/Cinder_config[DEFAULT/osapi_volume_workers]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Api/Cinder_config[DEFAULT/auth_strategy]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Api/Cinder_config[DEFAULT/nova_catalog_info]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Api/Cinder_config[DEFAULT/nova_catalog_admin_info]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Api/Cinder_config[key_manager/api_class]/ensure: created\", \n \"Notice: /Stage[main]/Apache::Mod::Mime/File[mime.conf]/ensure: defined content as '{md5}9da85e58f3bd6c780ce76db603b7f028'\", \n \"Notice: /Stage[main]/Apache::Mod::Mime_magic/File[mime_magic.conf]/ensure: defined content as '{md5}b258529b332429e2ff8344f726a95457'\", \n \"Notice: /Stage[main]/Apache::Mod::Alias/File[alias.conf]/ensure: defined content as '{md5}983e865be85f5e0daaed7433db82995e'\", \n \"Notice: /Stage[main]/Apache::Mod::Autoindex/File[autoindex.conf]/ensure: defined content as '{md5}2421a3c6df32c7e38c2a7a22afdf5728'\", \n \"Notice: /Stage[main]/Apache::Mod::Deflate/File[deflate.conf]/ensure: defined content as '{md5}a045d750d819b1e9dae3fbfb3f20edd5'\", \n \"Notice: /Stage[main]/Apache::Mod::Dir/File[dir.conf]/ensure: defined content as '{md5}c741d8ea840e6eb999d739eed47c69d7'\", \n \"Notice: /Stage[main]/Apache::Mod::Negotiation/File[negotiation.conf]/ensure: defined content as '{md5}47284b5580b986a6ba32580b6ffb9fd7'\", \n \"Notice: /Stage[main]/Apache::Mod::Setenvif/File[setenvif.conf]/ensure: defined content as '{md5}c7ede4173da1915b7ec088201f030c28'\", \n \"Notice: /Stage[main]/Apache::Mod::Prefork/File[/etc/httpd/conf.modules.d/prefork.conf]/ensure: defined content as '{md5}109c4f51dac10fc1b39373855e566d01'\", \n \"Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_user_domain_name]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[trustee/auth_type]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[trustee/auth_url]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[trustee/username]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[trustee/password]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[trustee/project_domain_name]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[trustee/user_domain_name]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[clients_keystone/auth_uri]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[DEFAULT/max_json_body_size]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[ec2authtoken/auth_uri]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[yaql/limit_iterators]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Heat_config[yaql/memory_quota]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Api_cloudwatch/Heat_config[heat_api_cloudwatch/bind_host]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/bind_host]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/auth_strategy]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/core_plugin]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/host]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/dns_domain]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/dhcp_agents_per_network]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/dhcp_agent_notification]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/allow_overlapping_ips]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/global_physnet_mtu]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[agent/root_helper]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Neutron_config[DEFAULT/service_plugins]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/File[/etc/neutron/plugin.ini]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/File[/etc/default/neutron-server]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/type_drivers]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/tenant_network_types]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/mechanism_drivers]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/path_mtu]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/extension_drivers]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/overlay_ip_version]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[securitygroup/firewall_driver]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Db/Oslo::Db[cinder_config]/Cinder_config[database/connection]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Db/Oslo::Db[cinder_config]/Cinder_config[database/max_retries]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Db/Oslo::Db[cinder_config]/Cinder_config[database/db_max_retries]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Logging/Oslo::Log[cinder_config]/Cinder_config[DEFAULT/log_dir]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Oslo::Messaging::Rabbit[cinder_config]/Cinder_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Oslo::Messaging::Rabbit[cinder_config]/Cinder_config[oslo_messaging_rabbit/rabbit_password]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Oslo::Messaging::Rabbit[cinder_config]/Cinder_config[oslo_messaging_rabbit/ssl]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Oslo::Messaging::Rabbit[cinder_config]/Cinder_config[oslo_messaging_rabbit/rabbit_userid]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Oslo::Messaging::Rabbit[cinder_config]/Cinder_config[oslo_messaging_rabbit/rabbit_port]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Oslo::Messaging::Default[cinder_config]/Cinder_config[DEFAULT/transport_url]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Oslo::Messaging::Default[cinder_config]/Cinder_config[DEFAULT/control_exchange]/ensure: created\", \n \"Notice: /Stage[main]/Cinder/Oslo::Concurrency[cinder_config]/Cinder_config[oslo_concurrency/lock_path]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Ceilometer/Oslo::Messaging::Notifications[cinder_config]/Cinder_config[oslo_messaging_notifications/driver]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Ceilometer/Oslo::Messaging::Notifications[cinder_config]/Cinder_config[oslo_messaging_notifications/transport_url]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Policy/Oslo::Policy[cinder_config]/Cinder_config[oslo_policy/policy_file]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Api/Oslo::Middleware[cinder_config]/Cinder_config[oslo_middleware/enable_proxy_headers_parsing]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Keystone::Authtoken/Keystone::Resource::Authtoken[cinder_config]/Cinder_config[keystone_authtoken/auth_uri]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Keystone::Authtoken/Keystone::Resource::Authtoken[cinder_config]/Cinder_config[keystone_authtoken/auth_type]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Keystone::Authtoken/Keystone::Resource::Authtoken[cinder_config]/Cinder_config[keystone_authtoken/auth_url]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Keystone::Authtoken/Keystone::Resource::Authtoken[cinder_config]/Cinder_config[keystone_authtoken/username]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Keystone::Authtoken/Keystone::Resource::Authtoken[cinder_config]/Cinder_config[keystone_authtoken/password]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Keystone::Authtoken/Keystone::Resource::Authtoken[cinder_config]/Cinder_config[keystone_authtoken/user_domain_name]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Keystone::Authtoken/Keystone::Resource::Authtoken[cinder_config]/Cinder_config[keystone_authtoken/project_name]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Keystone::Authtoken/Keystone::Resource::Authtoken[cinder_config]/Cinder_config[keystone_authtoken/project_domain_name]/ensure: created\", \n \"Notice: /Stage[main]/Apache/Concat[/etc/httpd/conf/ports.conf]/File[/etc/httpd/conf/ports.conf]/ensure: defined content as '{md5}6323431ffca83b5ae65685fd4cbf8d3d'\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf/httpd.conf]/content: content changed '{md5}f5e7449c0f17bc856e86011cb5d152ba' to '{md5}f81a6aba90472abbf67ca7f2d7fc6aab'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[log_config]/File[log_config.load]/ensure: defined content as '{md5}785d35cb285e190d589163b45263ca89'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[systemd]/File[systemd.load]/ensure: defined content as '{md5}26e5d44aae258b3e9d821cbbbd3e2826'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[unixd]/File[unixd.load]/ensure: defined content as '{md5}0e8468ecc1265f8947b8725f4d1be9c0'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[authz_host]/File[authz_host.load]/ensure: defined content as '{md5}d1045f54d2798499ca0f030ca0eef920'\", \n \"Notice: /Stage[main]/Apache::Mod::Actions/Apache::Mod[actions]/File[actions.load]/ensure: defined content as '{md5}599866dfaf734f60f7e2d41ee8235515'\", \n \"Notice: /Stage[main]/Apache::Mod::Authn_core/Apache::Mod[authn_core]/File[authn_core.load]/ensure: defined content as '{md5}704d6e8b02b0eca0eba4083960d16c52'\", \n \"Notice: /Stage[main]/Apache::Mod::Cache/Apache::Mod[cache]/File[cache.load]/ensure: defined content as '{md5}01e4d392225b518a65b0f7d6c4e21d29'\", \n \"Notice: /Stage[main]/Apache::Mod::Ext_filter/Apache::Mod[ext_filter]/File[ext_filter.load]/ensure: defined content as '{md5}76d5e0ac3411a4be57ac33ebe2e52ac8'\", \n \"Notice: /Stage[main]/Apache::Mod::Mime/Apache::Mod[mime]/File[mime.load]/ensure: defined content as '{md5}e36257b9efab01459141d423cae57c7c'\", \n \"Notice: /Stage[main]/Apache::Mod::Mime_magic/Apache::Mod[mime_magic]/File[mime_magic.load]/ensure: defined content as '{md5}cb8670bb2fb352aac7ebf3a85d52094c'\", \n \"Notice: /Stage[main]/Apache::Mod::Rewrite/Apache::Mod[rewrite]/File[rewrite.load]/ensure: defined content as '{md5}26e2683352fc1599f29573ff0d934e79'\", \n \"Notice: /Stage[main]/Apache::Mod::Speling/Apache::Mod[speling]/File[speling.load]/ensure: defined content as '{md5}f82e9e6b871a276c324c9eeffcec8a61'\", \n \"Notice: /Stage[main]/Apache::Mod::Suexec/Apache::Mod[suexec]/File[suexec.load]/ensure: defined content as '{md5}c7d5c61c534ba423a79b0ae78ff9be35'\", \n \"Notice: /Stage[main]/Apache::Mod::Version/Apache::Mod[version]/File[version.load]/ensure: defined content as '{md5}1c9243de22ace4dc8266442c48ae0c92'\", \n \"Notice: /Stage[main]/Apache::Mod::Vhost_alias/Apache::Mod[vhost_alias]/File[vhost_alias.load]/ensure: defined content as '{md5}eca907865997d50d5130497665c3f82e'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[auth_digest]/File[auth_digest.load]/ensure: defined content as '{md5}df9e85f8da0b239fe8e698ae7ead4f60'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[authn_anon]/File[authn_anon.load]/ensure: defined content as '{md5}bf57b94b5aec35476fc2a2dc3861f132'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[authn_dbm]/File[authn_dbm.load]/ensure: defined content as '{md5}90ee8f8ef1a017cacadfda4225e10651'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[authz_dbm]/File[authz_dbm.load]/ensure: defined content as '{md5}c1363277984d22f99b70f7dce8753b60'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[authz_owner]/File[authz_owner.load]/ensure: defined content as '{md5}f30a9be1016df87f195449d9e02d1857'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[expires]/File[expires.load]/ensure: defined content as '{md5}f0825bad1e470de86ffabeb86dcc5d95'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[include]/File[include.load]/ensure: defined content as '{md5}88095a914eedc3c2c184dd5d74c3954c'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[logio]/File[logio.load]/ensure: defined content as '{md5}084533c7a44e9129d0e6df952e2472b6'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[substitute]/File[substitute.load]/ensure: defined content as '{md5}8077c34a71afcf41c8fc644830935915'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[usertrack]/File[usertrack.load]/ensure: defined content as '{md5}e95fbbf030fabec98b948f8dc217775c'\", \n \"Notice: /Stage[main]/Apache::Mod::Alias/Apache::Mod[alias]/File[alias.load]/ensure: defined content as '{md5}3cf2fa309ccae4c29a4b875d0894cd79'\", \n \"Notice: /Stage[main]/Apache::Mod::Authn_file/Apache::Mod[authn_file]/File[authn_file.load]/ensure: defined content as '{md5}d41656680003d7b890267bb73621c60b'\", \n \"Notice: /Stage[main]/Apache::Mod::Autoindex/Apache::Mod[autoindex]/File[autoindex.load]/ensure: defined content as '{md5}515cdf5b573e961a60d2931d39248648'\", \n \"Notice: /Stage[main]/Apache::Mod::Dav/Apache::Mod[dav]/File[dav.load]/ensure: defined content as '{md5}588e496251838c4840c14b28b5aa7881'\", \n \"Notice: /Stage[main]/Apache::Mod::Dav_fs/File[dav_fs.conf]/ensure: defined content as '{md5}899a57534f3d84efa81887ec93c90c9b'\", \n \"Notice: /Stage[main]/Apache::Mod::Dav_fs/Apache::Mod[dav_fs]/File[dav_fs.load]/ensure: defined content as '{md5}2996277c73b1cd684a9a3111c355e0d3'\", \n \"Notice: /Stage[main]/Apache::Mod::Deflate/Apache::Mod[deflate]/File[deflate.load]/ensure: defined content as '{md5}2d1a1afcae0c70557251829a8586eeaf'\", \n \"Notice: /Stage[main]/Apache::Mod::Dir/Apache::Mod[dir]/File[dir.load]/ensure: defined content as '{md5}1bfb1c2a46d7351fc9eb47c659dee068'\", \n \"Notice: /Stage[main]/Apache::Mod::Negotiation/Apache::Mod[negotiation]/File[negotiation.load]/ensure: defined content as '{md5}d262ee6a5f20d9dd7f87770638dc2ccd'\", \n \"Notice: /Stage[main]/Apache::Mod::Setenvif/Apache::Mod[setenvif]/File[setenvif.load]/ensure: defined content as '{md5}ec6c99f7cc8e35bdbcf8028f652c9f6d'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[auth_basic]/File[auth_basic.load]/ensure: defined content as '{md5}494bcf4b843f7908675d663d8dc1bdc8'\", \n \"Notice: /Stage[main]/Apache::Mod::Filter/Apache::Mod[filter]/File[filter.load]/ensure: defined content as '{md5}66a1e2064a140c3e7dca7ac33877700e'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[authz_core]/File[authz_core.load]/ensure: defined content as '{md5}39942569bff2abdb259f9a347c7246bc'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[access_compat]/File[access_compat.load]/ensure: defined content as '{md5}d5feb88bec4570e2dbc41cce7e0de003'\", \n \"Notice: /Stage[main]/Apache::Mod::Authz_user/Apache::Mod[authz_user]/File[authz_user.load]/ensure: defined content as '{md5}63594303ee808423679b1ea13dd5a784'\", \n \"Notice: /Stage[main]/Apache::Default_mods/Apache::Mod[authz_groupfile]/File[authz_groupfile.load]/ensure: defined content as '{md5}ae005a36b3ac8c20af36c434561c8a75'\", \n \"Notice: /Stage[main]/Apache::Mod::Env/Apache::Mod[env]/File[env.load]/ensure: defined content as '{md5}d74184d40d0ee24ba02626a188ee7e1a'\", \n \"Notice: /Stage[main]/Apache::Mod::Prefork/Apache::Mpm[prefork]/File[/etc/httpd/conf.modules.d/prefork.load]/ensure: defined content as '{md5}157529aafcf03fa491bc924103e4608e'\", \n \"Notice: /Stage[main]/Apache::Mod::Cgi/Apache::Mod[cgi]/File[cgi.load]/ensure: defined content as '{md5}ac20c5c5779b37ab06b480d6485a0881'\", \n \"Notice: /Stage[main]/Apache::Mod::Ssl/Apache::Mod[ssl]/File[ssl.load]/ensure: defined content as '{md5}e282ac9f82fe5538692a4de3616fb695'\", \n \"Notice: /Stage[main]/Apache::Mod::Socache_shmcb/Apache::Mod[socache_shmcb]/File[socache_shmcb.load]/ensure: defined content as '{md5}ab31a6ea611785f74851b578572e4157'\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/00-nova-placement-api.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/README]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/auth_mellon.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/autoindex.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/openstack-dashboard.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/userdir.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/welcome.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache::Mod::Ssl/File[ssl.conf]/content: content changed '{md5}1888b608773b45f4acea3604eccf3562' to '{md5}b6f6f2773db25c777f1db887e7a3f57d'\", \n \"Notice: /Stage[main]/Apache::Mod::Wsgi/File[wsgi.conf]/ensure: defined content as '{md5}8b3feb3fc2563de439920bb2c52cbd11'\", \n \"Notice: /Stage[main]/Apache::Mod::Wsgi/Apache::Mod[wsgi]/File[wsgi.load]/ensure: defined content as '{md5}e1795e051e7aae1f865fde0d3b86a507'\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-base.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-dav.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-lua.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-mpm.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-proxy.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-ssl.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-systemd.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/01-cgi.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/10-auth_mellon.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/10-wsgi.conf]/ensure: removed\", \n \"Notice: /Stage[main]/Cinder::Wsgi::Apache/Openstacklib::Wsgi::Apache[cinder_wsgi]/File[/var/www/cgi-bin/cinder]/ensure: created\", \n \"Notice: /Stage[main]/Cinder::Wsgi::Apache/Openstacklib::Wsgi::Apache[cinder_wsgi]/File[cinder_wsgi]/ensure: defined content as '{md5}870efbe437d63cd260287cd36472d7b1'\", \n \"Notice: /Stage[main]/Heat::Logging/Oslo::Log[heat_config]/Heat_config[DEFAULT/log_dir]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Db/Oslo::Db[heat_config]/Heat_config[database/connection]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Db/Oslo::Db[heat_config]/Heat_config[database/max_retries]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Db/Oslo::Db[heat_config]/Heat_config[database/db_max_retries]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Authtoken/Keystone::Resource::Authtoken[heat_config]/Heat_config[keystone_authtoken/auth_uri]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Authtoken/Keystone::Resource::Authtoken[heat_config]/Heat_config[keystone_authtoken/auth_type]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Authtoken/Keystone::Resource::Authtoken[heat_config]/Heat_config[keystone_authtoken/auth_url]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Authtoken/Keystone::Resource::Authtoken[heat_config]/Heat_config[keystone_authtoken/username]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Authtoken/Keystone::Resource::Authtoken[heat_config]/Heat_config[keystone_authtoken/password]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Authtoken/Keystone::Resource::Authtoken[heat_config]/Heat_config[keystone_authtoken/user_domain_name]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Authtoken/Keystone::Resource::Authtoken[heat_config]/Heat_config[keystone_authtoken/project_name]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Keystone::Authtoken/Keystone::Resource::Authtoken[heat_config]/Heat_config[keystone_authtoken/project_domain_name]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Messaging::Rabbit[heat_config]/Heat_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Messaging::Rabbit[heat_config]/Heat_config[oslo_messaging_rabbit/rabbit_password]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Messaging::Rabbit[heat_config]/Heat_config[oslo_messaging_rabbit/ssl]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Messaging::Rabbit[heat_config]/Heat_config[oslo_messaging_rabbit/rabbit_userid]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Messaging::Rabbit[heat_config]/Heat_config[oslo_messaging_rabbit/rabbit_port]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Messaging::Notifications[heat_config]/Heat_config[oslo_messaging_notifications/driver]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Messaging::Notifications[heat_config]/Heat_config[oslo_messaging_notifications/transport_url]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Messaging::Default[heat_config]/Heat_config[DEFAULT/rpc_response_timeout]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Messaging::Default[heat_config]/Heat_config[DEFAULT/transport_url]/ensure: created\", \n \"Notice: /Stage[main]/Heat/Oslo::Middleware[heat_config]/Heat_config[oslo_middleware/enable_proxy_headers_parsing]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Policy/Openstacklib::Policy::Base[context_is_admin]/Augeas[/etc/heat/policy.json-context_is_admin-role:admin]/returns: executed successfully\", \n \"Notice: /Stage[main]/Cinder::Deps/Anchor[cinder::config::end]: Triggered 'refresh' from 38 events\", \n \"Notice: /Stage[main]/Cinder::Db::Sync/Exec[cinder-manage db_sync]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Cinder::Deps/Anchor[cinder::dbsync::end]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Cinder::Deps/Anchor[cinder::service::begin]: Triggered 'refresh' from 2 events\", \n \"Notice: /Stage[main]/Cinder::Api/Service[cinder-api]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Cinder::Deps/Anchor[cinder::service::end]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::config::end]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::service::begin]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Heat::Policy/Oslo::Policy[heat_config]/Heat_config[oslo_policy/policy_file]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Deps/Anchor[heat::config::end]: Triggered 'refresh' from 39 events\", \n \"Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Triggered 'refresh' from 2 events\", \n \"Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[nf_conntrack_proto_sctp]/Exec[modprobe nf_conntrack_proto_sctp]/returns: executed successfully\", \n \"Notice: /Stage[main]/Neutron::Logging/Oslo::Log[neutron_config]/Neutron_config[DEFAULT/log_dir]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Messaging::Default[neutron_config]/Neutron_config[DEFAULT/transport_url]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Messaging::Default[neutron_config]/Neutron_config[DEFAULT/control_exchange]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Concurrency[neutron_config]/Neutron_config[oslo_concurrency/lock_path]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Messaging::Notifications[neutron_config]/Neutron_config[oslo_messaging_notifications/driver]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Messaging::Notifications[neutron_config]/Neutron_config[oslo_messaging_notifications/transport_url]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_password]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/ssl]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_userid]/ensure: created\", \n \"Notice: /Stage[main]/Neutron/Oslo::Messaging::Rabbit[neutron_config]/Neutron_config[oslo_messaging_rabbit/rabbit_port]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[vxlan]/Neutron_plugin_ml2[ml2_type_vxlan/vxlan_group]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[vxlan]/Neutron_plugin_ml2[ml2_type_vxlan/vni_ranges]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[vlan]/Neutron_plugin_ml2[ml2_type_vlan/network_vlan_ranges]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[flat]/Neutron_plugin_ml2[ml2_type_flat/flat_networks]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Type_driver[gre]/Neutron_plugin_ml2[ml2_type_gre/tunnel_id_ranges]/ensure: created\", \n \"Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::config::end]: Triggered 'refresh' from 36 events\", \n \"Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::begin]: Triggered 'refresh' from 1 events\", \n \"Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/File[/var/www/cgi-bin/heat]/ensure: created\", \n \"Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/File[heat_api_cloudwatch_wsgi]/ensure: defined content as '{md5}2eb19266988f424046d53acfbcf01c2c'\", \n \"Notice: /Stage[main]/Cinder::Wsgi::Apache/Openstacklib::Wsgi::Apache[cinder_wsgi]/Apache::Vhost[cinder_wsgi]/Concat[10-cinder_wsgi.conf]/File[/etc/httpd/conf.d/10-cinder_wsgi.conf]/ensure: defined content as '{md5}d195c906271aed26c1a218773864ac48'\", \n \"Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/Apache::Vhost[heat_api_cloudwatch_wsgi]/Concat[10-heat_api_cloudwatch_wsgi.conf]/File[/etc/httpd/conf.d/10-heat_api_cloudwatch_wsgi.conf]/ensure: defined content as '{md5}b68ebb86250f0bf151d756400dda5aa5'\", \n \"Notice: /Stage[main]/Apache::Service/Service[httpd]/ensure: ensure changed 'stopped' to 'running'\", \n \"Notice: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/iptables]/seltype: seltype changed 'etc_t' to 'system_conf_t'\", \n \"Notice: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/ip6tables]/seltype: seltype changed 'etc_t' to 'system_conf_t'\", \n \"Notice: Applied catalog in 27.24 seconds\"\n ], \n \"failed\": false, \n \"failed_when_result\": false\n}\n\nTASK [Run docker-puppet tasks (generate config)] *******************************\nskipping: [localhost]\n\nTASK [debug] *******************************************************************\nok: [localhost] => {\n \"(outputs.stderr|default('')).split('\\n')|union(outputs.stdout_lines|default([]))\": [\n \"\"\n ], \n \"failed\": false, \n \"failed_when_result\": false\n}\n\nTASK [Check if /var/lib/hashed-tripleo-config/docker-container-startup-config-step_3.json exists] ***\nok: [localhost]\n\nTASK [Start containers for step 3] *********************************************\nok: [localhost]\n\nTASK [debug] *******************************************************************\nfatal: [localhost]: FAILED! => {\n \"(outputs.stderr|default('')).split('\\n')|union(outputs.stdout_lines|default([]))\": [\n \"stdout: \", \n \"stderr: \", \n \"stdout: 2017-09-25 19:56:58.542 11 WARNING oslo_config.cfg [-] Option \\\"db_backend\\\" from group \\\"DEFAULT\\\" is deprecated. Use option \\\"backend\\\" from group \\\"database\\\".\", \n \"\", \n \"stdout: Upgraded database to: pike01, current revision(s): pike01\", \n \"stderr: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\", \n \"INFO:__main__:Validating config file\", \n \"INFO:__main__:Kolla config strategy set to: COPY_ALWAYS\", \n \"INFO:__main__:Copying service configuration files\", \n \"INFO:__main__:Deleting /etc/glance/glance-api.conf\", \n \"INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-api.conf to /etc/glance/glance-api.conf\", \n \"INFO:__main__:Deleting /etc/glance/glance-cache.conf\", \n \"INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-cache.conf to /etc/glance/glance-cache.conf\", \n \"INFO:__main__:Deleting /etc/glance/glance-swift.conf\", \n \"INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-swift.conf to /etc/glance/glance-swift.conf\", \n \"INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf\", \n \"INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/rbdmap to /etc/ceph/rbdmap\", \n \"INFO:__main__:Writing out command to execute\", \n \"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:1328: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade\", \n \" expire_on_commit=expire_on_commit, _conf=conf)\", \n \"INFO [alembic.runtime.migration] Context impl MySQLImpl.\", \n \"INFO [alembic.runtime.migration] Will assume non-transactional DDL.\", \n \"INFO [alembic.runtime.migration] Running upgrade -> liberty, liberty initial\", \n \"INFO [alembic.runtime.migration] Running upgrade liberty -> mitaka01, add index on created_at and updated_at columns of 'images' table\", \n \"INFO [alembic.runtime.migration] Running upgrade mitaka01 -> mitaka02, update metadef os_nova_server\", \n \"INFO [alembic.runtime.migration] Running upgrade mitaka02 -> ocata01, add visibility to and remove is_public from images\", \n \"INFO [alembic.runtime.migration] Running upgrade ocata01 -> pike01, drop glare artifacts tables\", \n \"stdout: '/swift_ringbuilder/etc/swift/account.ring.gz' -> '/etc/swift/account.ring.gz'\", \n \"'/swift_ringbuilder/etc/swift/container.ring.gz' -> '/etc/swift/container.ring.gz'\", \n \"'/swift_ringbuilder/etc/swift/object.ring.gz' -> '/etc/swift/object.ring.gz'\", \n \"'/swift_ringbuilder/etc/swift/account.builder' -> '/etc/swift/account.builder'\", \n \"'/swift_ringbuilder/etc/swift/container.builder' -> '/etc/swift/container.builder'\", \n \"'/swift_ringbuilder/etc/swift/object.builder' -> '/etc/swift/object.builder'\", \n \"'/swift_ringbuilder/etc/swift/backups' -> '/etc/swift/backups'\", \n \"'/swift_ringbuilder/etc/swift/backups/1506368899.object.builder' -> '/etc/swift/backups/1506368899.object.builder'\", \n \"'/swift_ringbuilder/etc/swift/backups/1506368900.account.builder' -> '/etc/swift/backups/1506368900.account.builder'\", \n \"'/swift_ringbuilder/etc/swift/backups/1506368900.container.builder' -> '/etc/swift/backups/1506368900.container.builder'\", \n \"'/swift_ringbuilder/etc/swift/backups/1506368905.account.builder' -> '/etc/swift/backups/1506368905.account.builder'\", \n \"'/swift_ringbuilder/etc/swift/backups/1506368905.account.ring.gz' -> '/etc/swift/backups/1506368905.account.ring.gz'\", \n \"'/swift_ringbuilder/etc/swift/backups/1506368905.object.builder' -> '/etc/swift/backups/1506368905.object.builder'\", \n \"'/swift_ringbuilder/etc/swift/backups/1506368905.object.ring.gz' -> '/etc/swift/backups/1506368905.object.ring.gz'\", \n \"'/swift_ringbuilder/etc/swift/backups/1506368906.container.builder' -> '/etc/swift/backups/1506368906.container.builder'\", \n \"'/swift_ringbuilder/etc/swift/backups/1506368906.container.ring.gz' -> '/etc/swift/backups/1506368906.container.ring.gz'\", \n \"Error running ['docker', 'run', '--name', 'keystone_db_sync', '--label', 'config_id=tripleo_step3', '--label', 'container_name=keystone_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\\\"image\\\": \\\"192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-09-22.5\\\", \\\"environment\\\": [\\\"KOLLA_BOOTSTRAP=True\\\", \\\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\\\"], \\\"command\\\": [\\\"/usr/bin/bootstrap_host_exec\\\", \\\"keystone\\\", \\\"/usr/local/bin/kolla_start\\\"], \\\"user\\\": \\\"root\\\", \\\"volumes\\\": [\\\"/etc/hosts:/etc/hosts:ro\\\", \\\"/etc/localtime:/etc/localtime:ro\\\", \\\"/etc/puppet:/etc/puppet:ro\\\", \\\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\\", \\\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\\", \\\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\\", \\\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\\", \\\"/dev/log:/dev/log\\\", \\\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\\", \\\"/sys/fs/selinux:/sys/fs/selinux\\\", \\\"/var/lib/kolla/config_files/keystone.json:/var/lib/kolla/config_files/config.json:ro\\\", \\\"/var/lib/config-data/puppet-generated/keystone/:/var/lib/kolla/config_files/src:ro\\\", \\\"/var/log/containers/keystone:/var/log/keystone\\\", \\\"/var/log/containers/httpd/keystone:/var/log/httpd\\\", \\\"\\\", \\\"\\\"], \\\"net\\\": \\\"host\\\", \\\"detach\\\": false, \\\"privileged\\\": false}', '--env=KOLLA_BOOTSTRAP=True', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/sys/fs/selinux:/sys/fs/selinux', '--volume=/var/lib/kolla/config_files/keystone.json:/var/lib/kolla/config_files/config.json:ro', '--volume=/var/lib/config-data/puppet-generated/keystone/:/var/lib/kolla/config_files/src:ro', '--volume=/var/log/containers/keystone:/var/log/keystone', '--volume=/var/log/containers/httpd/keystone:/var/log/httpd', '192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-09-22.5', '/usr/bin/bootstrap_host_exec', 'keystone', '/usr/local/bin/kolla_start']. [1]\", \n \"ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found\", \n \"stdout: 7bbd0bc896f434121159bb8a6b58122e9032ed94a864a67663236b0b3ad9394c\", \n \"stderr: Unable to find image '192.168.24.1:8787/rhosp12/openstack-ceilometer-notification-docker:2017-09-22.5' locally\", \n \"Trying to pull repository 192.168.24.1:8787/rhosp12/openstack-ceilometer-notification-docker ... \", \n \"2017-09-22.5: Pulling from 192.168.24.1:8787/rhosp12/openstack-ceilometer-notification-docker\", \n \"00406150827c: Already exists\", \n \"00c572151848: Already exists\", \n \"aed1081b5ee5: Already exists\", \n \"eb43a065e807: Already exists\", \n \"e7072b048a83: Already exists\", \n \"076e36bb2fff: Pulling fs layer\", \n \"076e36bb2fff: Verifying Checksum\", \n \"076e36bb2fff: Download complete\", \n \"076e36bb2fff: Pull complete\", \n \"Digest: sha256:955210ba5f2df155eb14d2fd9ddd9266f03e954be59aba196dfda939b60bbbd3\", \n \"stdout: a52697400c97f2105569f01d0d978a7a7fafdc3cfaa8fa1060602e1daec1912f\", \n \"stdout: 541a817d18bd575790824683c7451a1270e5600c385a4d13874a8400cddd1f21\", \n \"stderr: Unable to find image '192.168.24.1:8787/rhosp12/openstack-swift-account-docker:2017-09-22.5' locally\", \n \"Trying to pull repository 192.168.24.1:8787/rhosp12/openstack-swift-account-docker ... \", \n \"2017-09-22.5: Pulling from 192.168.24.1:8787/rhosp12/openstack-swift-account-docker\", \n \"f5834c3aa86e: Already exists\", \n \"4ef157ef439b: Pulling fs layer\", \n \"4ef157ef439b: Verifying Checksum\", \n \"4ef157ef439b: Download complete\", \n \"4ef157ef439b: Pull complete\", \n \"Digest: sha256:9edf748ab4a37c1e93246c5bad340b9e14eb2c2115c23ccf00efea11ae87e6e7\", \n \"stdout: cc00cba18b1a786e559d422369b51224d1d344fda28d562cc7cc549e32ddb70f\", \n \"stderr: INFO [alembic.runtime.migration] Context impl MySQLImpl.\", \n \"INFO [alembic.runtime.migration] Running upgrade -> 2581ebaf0cb2, initial migration\", \n \"INFO [alembic.runtime.migration] Running upgrade 2581ebaf0cb2 -> 21b331f883ef, Add provision_updated_at\", \n \"INFO [alembic.runtime.migration] Running upgrade 21b331f883ef -> 3cb628139ea4, Nodes add console enabled\", \n \"INFO [alembic.runtime.migration] Running upgrade 3cb628139ea4 -> 31baaf680d2b, Add Node instance info\", \n \"INFO [alembic.runtime.migration] Running upgrade 31baaf680d2b -> 3bea56f25597, add unique constraint to instance_uuid\", \n \"INFO [alembic.runtime.migration] Running upgrade 3bea56f25597 -> 487deb87cc9d, add conductor_affinity and online\", \n \"INFO [alembic.runtime.migration] Running upgrade 487deb87cc9d -> 242cc6a923b3, Add Node.maintenance_reason\", \n \"INFO [alembic.runtime.migration] Running upgrade 242cc6a923b3 -> 5674c57409b9, replace NOSTATE with AVAILABLE\", \n \"INFO [alembic.runtime.migration] Running upgrade 5674c57409b9 -> bb59b63f55a, add_node_driver_internal_info\", \n \"INFO [alembic.runtime.migration] Running upgrade bb59b63f55a -> 3ae36a5f5131, add_logical_name\", \n \"INFO [alembic.runtime.migration] Running upgrade 3ae36a5f5131 -> 1e1d5ace7dc6, add inspection_started_at and inspection_finished_at\", \n \"INFO [alembic.runtime.migration] Running upgrade 1e1d5ace7dc6 -> 4f399b21ae71, Add node.clean_step\", \n \"INFO [alembic.runtime.migration] Running upgrade 4f399b21ae71 -> 2fb93ffd2af1, increase-node-name-length\", \n \"INFO [alembic.runtime.migration] Running upgrade 2fb93ffd2af1 -> 789acc877671, Add node.raid_config and node.target_raid_config\", \n \"INFO [alembic.runtime.migration] Running upgrade 789acc877671 -> 516faf1bb9b1, Resizing column nodes.driver\", \n \"INFO [alembic.runtime.migration] Running upgrade 516faf1bb9b1 -> 48d6c242bb9b, add node tags\", \n \"INFO [alembic.runtime.migration] Running upgrade 48d6c242bb9b -> 5ea1b0d310e, Added portgroups table and altered ports\", \n \"INFO [alembic.runtime.migration] Running upgrade 5ea1b0d310e -> f6fdb920c182, Set Port.pxe_enabled to True if NULL\", \n \"INFO [alembic.runtime.migration] Running upgrade f6fdb920c182 -> e294876e8028, add-node-network-interface\", \n \"INFO [alembic.runtime.migration] Running upgrade e294876e8028 -> 10b163d4481e, add port portgroup internal info\", \n \"INFO [alembic.runtime.migration] Running upgrade 10b163d4481e -> dd34e1f1303b, add resource_class to node\", \n \"INFO [alembic.runtime.migration] Running upgrade dd34e1f1303b -> c14cef6dfedf, Populate node.network_interface\", \n \"INFO [alembic.runtime.migration] Running upgrade c14cef6dfedf -> 60cf717201bc, add_standalone_ports_supported_to_portgroup\", \n \"INFO [alembic.runtime.migration] Running upgrade 60cf717201bc -> bcdd431ba0bf, Add fields for all interfaces\", \n \"INFO [alembic.runtime.migration] Running upgrade bcdd431ba0bf -> daa1ba02d98, Add volume_connectors table\", \n \"INFO [alembic.runtime.migration] Running upgrade daa1ba02d98 -> 1a59178ebdf6, Add volume_targets table\", \n \"INFO [alembic.runtime.migration] Running upgrade 1a59178ebdf6 -> 493d8f27f235, add portgroup configuration fields\", \n \"INFO [alembic.runtime.migration] Running upgrade 493d8f27f235 -> 1d6951876d68, Add storage_interface DB field and object\", \n \"INFO [alembic.runtime.migration] Running upgrade 1d6951876d68 -> 2353895ecfae, Add conductor_hardware_interfaces table\", \n \"INFO [alembic.runtime.migration] Running upgrade 2353895ecfae -> dbefd6bdaa2c, Add default column to ConductorHardwareInterfaces\", \n \"INFO [alembic.runtime.migration] Running upgrade dbefd6bdaa2c -> 3d86a077a3f2, add port physical network\", \n \"INFO [alembic.runtime.migration] Running upgrade 3d86a077a3f2 -> 868cb606a74a, add version field in base class\", \n \"stdout: 6d6e46f6c714d26965539645256eef10308a6c585d2af528ca5440f79d1b1e1a\", \n \"stdout: 238858453ca83067ad75169c19aabe36b7b771d8daaf9e41b06ccb047258f041\", \n \"Error running ['docker', 'exec', '--user=root', u'keystone', '/usr/bin/bootstrap_host_exec', 'keystone', 'keystone-manage', 'bootstrap', '--bootstrap-password', 'AuPN2YF2FVEN8JkgpDXFgwDuW']. [1]\", \n \"stderr: Error response from daemon: Container 238858453ca83067ad75169c19aabe36b7b771d8daaf9e41b06ccb047258f041 is restarting, wait until the container is running\", \n \"stdout: 2fbc79b9c7c3d954b36342608348930dfe16a62aff83be7adcefa4edbe5bff69\"\n ], \n \"changed\": false, \n \"failed\": true, \n \"failed_when_result\": true\n}\n\tto retry, use: --limit @/var/lib/heat-config/heat-config-ansible/0181074a-08f5-4b6f-8280-16886194b53f_playbook.retry\n\nPLAY RECAP *********************************************************************\nlocalhost : ok=7 changed=1 unreachable=0 failed=1 \n\n", "deploy_stderr": "", "deploy_status_code": 2 }, "creation_time": "2017-09-25T19:57:06Z", "updated_time": "2017-09-25T19:59:09Z", "input_values": { "update_identifier": "1506367931", "bootstrap_server_id": "28d741cc-952c-4007-80d5-80883c3eef89", "docker_puppet_process_count": 3, "step": 3, "docker_puppet_debug": "", "role_name": "Controller" }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 2", "id": "d9aeb51d-9008-4097-b187-a7c883bdf50b" }
reproduced on my environment : Mike can you check if you got those AVCs on your controller : http://pastebin.test.redhat.com/519201
Hi Omri, here are the avc log for all (3) controllers. Please note you will see "received setenforce notice (enforcing=0)" which was done by me after the failure. You will see same errors earlier [root@overcloud-controller-0 ~]# grep avc /var/log/audit/audit.log type=AVC msg=audit(1506368858.594:172): avc: denied { read } for pid=21496 comm="grep" name="kvm.conf" dev="vda2" ino=12583138 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.595:173): avc: denied { read } for pid=21496 comm="grep" name="lockd.conf" dev="vda2" ino=12583135 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.595:174): avc: denied { read } for pid=21496 comm="grep" name="mlx4.conf" dev="vda2" ino=12583136 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.595:175): avc: denied { read } for pid=21496 comm="grep" name="truescale.conf" dev="vda2" ino=12583137 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.595:176): avc: denied { read } for pid=21496 comm="grep" name="tuned.conf" dev="vda2" ino=12583134 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.595:177): avc: denied { read } for pid=21496 comm="grep" name="vhost.conf" dev="vda2" ino=12583133 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=USER_AVC msg=audit(1506369416.026:1207): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { status } for auid=n/a uid=0 gid=0 path="/run/systemd/system/docker-88b27630aad020113ce4159e90a4a8525e2a69bfe3798b0d28d5bb76234a9f13.scope" cmdline="/usr/lib/systemd/systemd-machined" scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:object_r:container_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?' type=USER_AVC msg=audit(1506369416.027:1208): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { status } for auid=n/a uid=0 gid=0 path="/run/systemd/system/docker-88b27630aad020113ce4159e90a4a8525e2a69bfe3798b0d28d5bb76234a9f13.scope" cmdline="/usr/lib/systemd/systemd-machined" scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:object_r:container_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?' type=USER_AVC msg=audit(1506372719.864:4983): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: received setenforce notice (enforcing=0) exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?' type=USER_AVC msg=audit(1506372719.864:4984): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: received setenforce notice (enforcing=1) exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?' type=USER_AVC msg=audit(1506376716.097:9241): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: received setenforce notice (enforcing=0) exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?' type=AVC msg=audit(1506376911.038:9371): avc: denied { execute } for pid=612190 comm="iptables.init" name="plymouth" dev="vda2" ino=9271838 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file type=AVC msg=audit(1506376911.038:9373): avc: denied { read open } for pid=612222 comm="iptables.init" path="/usr/bin/plymouth" dev="vda2" ino=9271838 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file type=AVC msg=audit(1506376911.038:9373): avc: denied { execute_no_trans } for pid=612222 comm="iptables.init" path="/usr/bin/plymouth" dev="vda2" ino=9271838 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file type=AVC msg=audit(1506376911.039:9375): avc: denied { connectto } for pid=612222 comm="plymouth" path=002F6F72672F667265656465736B746F702F706C796D6F75746864 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:system_r:plymouthd_t:s0 tclass=unix_stream_socket type=AVC msg=audit(1506376911.039:9376): avc: denied { block_suspend } for pid=612222 comm="plymouth" capability=36 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:system_r:iptables_t:s0 tclass=capability2 type=AVC msg=audit(1506377014.619:12): avc: denied { read } for pid=770 comm="grep" name="kvm.conf" dev="vda2" ino=12583138 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506377014.620:13): avc: denied { read } for pid=770 comm="grep" name="lockd.conf" dev="vda2" ino=12583135 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506377014.620:14): avc: denied { read } for pid=770 comm="grep" name="mlx4.conf" dev="vda2" ino=12583136 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506377014.620:15): avc: denied { read } for pid=770 comm="grep" name="truescale.conf" dev="vda2" ino=12583137 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506377014.620:16): avc: denied { read } for pid=770 comm="grep" name="tuned.conf" dev="vda2" ino=12583134 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506377014.620:17): avc: denied { read } for pid=770 comm="grep" name="vhost.conf" dev="vda2" ino=12583133 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506377014.672:29): avc: denied { execute } for pid=783 comm="iptables.init" name="plymouth" dev="vda2" ino=9271838 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file type=USER_AVC msg=audit(1506377129.883:334): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: received setenforce notice (enforcing=0) exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?' [root@overcloud-controller-1 ~]# grep avc /var/log/audit/audit.log type=AVC msg=audit(1506368853.863:162): avc: denied { read } for pid=20991 comm="grep" name="kvm.conf" dev="vda2" ino=12583138 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368853.863:163): avc: denied { read } for pid=20991 comm="grep" name="lockd.conf" dev="vda2" ino=12583135 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368853.863:164): avc: denied { read } for pid=20991 comm="grep" name="mlx4.conf" dev="vda2" ino=12583136 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368853.863:165): avc: denied { read } for pid=20991 comm="grep" name="truescale.conf" dev="vda2" ino=12583137 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368853.863:166): avc: denied { read } for pid=20991 comm="grep" name="tuned.conf" dev="vda2" ino=12583134 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368853.863:167): avc: denied { read } for pid=20991 comm="grep" name="vhost.conf" dev="vda2" ino=12583133 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506387045.836:19903): avc: denied { name_bind } for pid=305554 comm="dhclient" src=61000 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:ephemeral_port_t:s0 tclass=udp_socket [root@overcloud-controller-2 ~]# grep avc /var/log/audit/audit.log type=AVC msg=audit(1506368858.241:163): avc: denied { read } for pid=21060 comm="grep" name="kvm.conf" dev="vda2" ino=12583138 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.242:164): avc: denied { read } for pid=21060 comm="grep" name="lockd.conf" dev="vda2" ino=12583135 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.242:165): avc: denied { read } for pid=21060 comm="grep" name="mlx4.conf" dev="vda2" ino=12583136 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.242:166): avc: denied { read } for pid=21060 comm="grep" name="truescale.conf" dev="vda2" ino=12583137 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.242:167): avc: denied { read } for pid=21060 comm="grep" name="tuned.conf" dev="vda2" ino=12583134 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506368858.242:168): avc: denied { read } for pid=21060 comm="grep" name="vhost.conf" dev="vda2" ino=12583133 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file type=AVC msg=audit(1506387045.364:19807): avc: denied { name_bind } for pid=296593 comm="dhclient" src=61000 scontext=system_u:system_r:dhcpc_t:s0 tcontext=system_u:object_r:ephemeral_port_t:s0 tclass=udp_socket
According to the logs, keystone_db_sync throws an error "ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found" and is in a restarting loop. I've logged in the environment and effectively /var/lib/config-data/puppet-generated/keystone is empty, /var/lib/config-data/keystone is not. Also, generating the config again with "sudo CONFIG_VOLUME_PREFIX=~/puppet-test/ python /var/lib/docker-puppet/docker-puppet.py" I can see the generated files in ~/puppet-test/puppet-generated/keystone/ which makes me think of a race somewhere that deletes the content of /var/lib/config-data/puppet-generated/keystone.
Hi Mike - since we saw this bug on your env , we've got couple of new image builds , have you reproduced the issue on using latest images ? It might be something that not always reproduces , but we would need to add more info if it does reproduce again .
Hi Omri - I have not seen the issue anymore. I have had a few successful deployments last couple days. Thanks