Description of problem: Set up stable mitaka repos on instack (VM) [stack@instack ~]$ sudo curl -o /etc/yum.repos.d/delorean-mitaka.repo \ http://trunk.rdoproject.org/centos7-mitaka/current/delorean.repo [stack@instack ~]$ sudo curl -o /etc/yum.repos.d/delorean-deps-mitaka.repo \ http://trunk.rdoproject.org/centos7-mitaka/delorean-deps.repo [stack@instack ~]$ sudo yum -y install yum-plugin-priorities export NODE_DIST=centos7 export USE_DELOREAN_TRUNK=1 export DELOREAN_TRUNK_REPO="http://trunk.rdoproject.org/centos7-mitaka/current/" export DELOREAN_REPO_FILE="delorean.repo" $ sudo yum install -y python-tripleoclient $ openstack undercloud install Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Traceback (most recent call last): File "/bin/openstack", line 6, in <module> from openstackclient.shell import main File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 32, in <module> from openstackclient.common import clientmanager File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 297, in <module> 'openstack.cli.extension', File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 264, in get_plugin_modules __import__(ep.module_name) File "/usr/lib/python2.7/site-packages/gnocchiclient/osc.py", line 17, in <module> from osc_lib import utils ImportError: No module named osc_lib (tried 44, for a total of 170 seconds) Error: Not managing Keystone_service[Image Service] due to earlier Keystone API failures. Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[Image Service::image]/ensure: change from absent to present failed: Not managing Keystone_service[Image Service] due to earlier Keystone API failures. Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Traceback (most recent call last): File "/bin/openstack", line 6, in <module> from openstackclient.shell import main File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 32, in <module> from openstackclient.common import clientmanager File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 297, in <module> 'openstack.cli.extension', File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 264, in get_plugin_modules __import__(ep.module_name) File "/usr/lib/python2.7/site-packages/gnocchiclient/osc.py", line 17, in <module> from osc_lib import utils ImportError: No module named osc_lib (tried 44, for a total of 170 seconds) Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Traceback (most recent call last): File "/bin/openstack", line 6, in <module> from openstackclient.shell import main File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 32, in <module> from openstackclient.common import clientmanager File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 297, in <module> 'openstack.cli.extension', File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 264, in get_plugin_modules __import__(ep.module_name) File "/usr/lib/python2.7/site-packages/gnocchiclient/osc.py", line 17, in <module> from osc_lib import utils ImportError: No module named osc_lib (tried 44, for a total of 170 seconds) Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Traceback (most recent call last): File "/bin/openstack", line 6, in <module> from openstackclient.shell import main File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 32, in <module> from openstackclient.common import clientmanager File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 297, in <module> 'openstack.cli.extension', File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 264, in get_plugin_modules __import__(ep.module_name) File "/usr/lib/python2.7/site-packages/gnocchiclient/osc.py", line 17, in <module> from osc_lib import utils ImportError: No module named osc_lib (tried 44, for a total of 170 seconds) Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures. Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures. Error: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_user[aodh]: Could not evaluate: Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Traceback (most recent call last): File "/bin/openstack", line 6, in <module> from openstackclient.shell import main File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 32, in <module> from openstackclient.common import clientmanager File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 297, in <module> 'openstack.cli.extension', File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 264, in get_plugin_modules __import__(ep.module_name) File "/usr/lib/python2.7/site-packages/gnocchiclient/osc.py", line 17, in <module> from osc_lib import utils ImportError: No module named osc_lib (tried 44, for a total of 170 seconds) Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Traceback (most recent call last): File "/bin/openstack", line 6, in <module> from openstackclient.shell import main File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 32, in <module> from openstackclient.common import clientmanager File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 297, in <module> 'openstack.cli.extension', File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 264, in get_plugin_modules __import__(ep.module_name) File "/usr/lib/python2.7/site-packages/gnocchiclient/osc.py", line 17, in <module> from osc_lib import utils ImportError: No module named osc_lib (tried 44, for a total of 170 seconds) Error: Not managing Keystone_service[aodh] due to earlier Keystone API failures. Error: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_service[aodh::alarming]/ensure: change from absent to present failed: Not managing Keystone_service[aodh] due to earlier Keystone API failures. Version-Release number of selected component (if applicable): [stack@instack yum.repos.d]$ cat delorean-mitaka.repo [delorean] name=delorean-openstack-aodh-8b46d8a703be68a9d365a2fb8e354c0a47f13757 baseurl=http://trunk.rdoproject.org/centos7-mitaka/8b/46/8b46d8a703be68a9d365a2fb8e354c0a47f13757_a3fe8655 enabled=1 gpgcheck=0 priority=1[stack@instack yum.repos.d]$ cat delorean-deps-mitaka.repo [delorean-mitaka-testing] name=dlrn-mitaka-testing baseurl=http://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-mitaka/ enabled=1 gpgcheck=0 priority=2 How reproducible: Follow http://bderzhavets.blogspot.com/2016/07/stable-mitaka-ha-instack-virt-setup.html ( actually, official guide slightly updated for Network Isolation overcloud deployment ) Steps to Reproduce: 1. Follow official TripleO guide (upstream ) for Mitaka/stable http://docs.openstack.org/developer/tripleo-docs/installation/installation.html 2. sudo yum install -y python-tripleoclient 3. openstack undercloud install Actual results: Stack trace above Expected results: undercloud gets deployed Additional info: It worked on just yesterday 08/06/2016
pip install osc-lib just before `openstack undercloud install` allows undercloud to be installed , however finally deploying overcloud ( ironic node-list was in good shape after successful `openstack baremetal introspection bulk start`) I got :- 2016-08-07 18:07:59 [Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" 2016-08-07 18:07:59 [Controller]: DELETE_IN_PROGRESS state changed 2016-08-07 18:08:03 [Controller]: DELETE_COMPLETE state changed 2016-08-07 18:08:07 [Controller]: CREATE_IN_PROGRESS state changed 2016-08-07 18:08:10 [Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" 2016-08-07 18:08:10 [Controller]: DELETE_IN_PROGRESS state changed 2016-08-07 18:08:12 [Controller]: DELETE_COMPLETE state changed 2016-08-07 18:08:18 [Controller]: CREATE_IN_PROGRESS state changed 2016-08-07 18:08:20 [Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
This bug blocks both the Liberty and the Mitaka promotion jobs. Mitaka undercloud deploy log: https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-mitaka-delorean-minimal-657/undercloud/home/stack/undercloud_install.log.gz Liberty (same): https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-liberty-delorean-minimal-488/undercloud/home/stack/undercloud_install.log.gz Fix can be tested with the promotion jobs.
Two new fresh attempts with install as follows :- . . . . . $ sudo yum install -y python-tripleoclient $ sudo yum install python-pip python-devel $ sudo pip install osc-lib $ openstack undercloud install Success Proceed as usual and attempt to deploy ( worked fine 5-6 days ago on same box):- #!/bin/bash -x source /home/stack/stackrc openstack overcloud deploy --templates --control-scale 3 \ --compute-scale 2 \ --libvirt-type qemu \ --ntp-server pool.ntp.org \ -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \ -e $HOME/network_env.yaml Now deployment in both cases goes until same point 2016-08-08 10:01:41 [0]: SIGNAL_IN_PROGRESS Signal: deployment failed (6) 2016-08-08 10:01:42 [0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 2016-08-08 10:01:43 [overcloud-ControllerNodesPostDeployment-twy4ozonamwj-ControllerOvercloudServicesDeployment_Step6-q2rcdzbsucn5]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 Stack overcloud CREATE_FAILED Deployment failed: Heat Stack create failed. [stack@instack ~]$ heat resource-list overcloud | grep ControllerNodesPost | ControllerNodesPostDeployment | 8a0346e5-568e-4ccc-a79e-ef10dda49552 | OS::TripleO::ControllerPostDeployment | CREATE_FAILED | 2016-08-08T09:15:32 | [stack@instack ~]$ heat stack-list -n | grep "^|8a0346e5-568e-4ccc-a79e-ef10dda49552|" [stack@instack ~]$ heat event-list overcloud-ControllerNodesPostDeployment-twy4ozonamwj | tac | head -n2 +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ | overcloud-ControllerNodesPostDeployment-twy4ozonamwj | 60768cf5-8358-4ae2-9ec7-7983e876e18f | Resource CREATE failed: Error: resources.ControllerOvercloudServicesDeployment_Step6.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 | CREATE_FAILED | 2016-08-08T10:01:45 | [stack@instack ~]$ heat event-list -m 60768cf5-8358-4ae2-9ec7-7983e876e18f overcloud-ControllerNodesPostDeployment-twy4ozonamwj +----+------------------------+-----------------+------------+ | id | resource_status_reason | resource_status | event_time | +----+------------------------+-----------------+------------+ +----+------------------------+-----------------+------------+
(In reply to Attila Darazs from comment #2) > This bug blocks both the Liberty and the Mitaka promotion jobs. > > Mitaka undercloud deploy log: > https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote- > mitaka-delorean-minimal-657/undercloud/home/stack/undercloud_install.log.gz > > Liberty (same): > https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote- > liberty-delorean-minimal-488/undercloud/home/stack/undercloud_install.log.gz > > Fix can be tested with the promotion jobs. Error with import osc_lib may be fixed via sequence of commands $ sudo yum install -y python-tripleoclient $ sudo yum install python-pip python-devel $ sudo pip install osc-lib $ openstack undercloud install I install osc-lib right before `openstack undercloud install` what results undercloud been built OK. In meantime attempt of deployment 3xNode HA-Controller+2xComputes two times crashed as described above
Another attempt:- [stack@instack ~]$ cat ./overcloud-ceph.sh #!/bin/bash -x source /home/stack/stackrc openstack overcloud deploy --templates --control-scale 1 \ --compute-scale 2 \ --ceph-storage-scale 2 \ --libvirt-type qemu \ --ntp-server pool.ntp.org \ -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \ -e $HOME/network_env.yaml Fails with message :- 2016-08-08 11:38:15 [0]: SIGNAL_IN_PROGRESS Signal: deployment failed (6) 2016-08-08 11:38:16 [0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 2016-08-08 11:38:17 [ControllerOvercloudServicesDeployment_Step4]: CREATE_FAILED Error: resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 2016-08-08 11:38:17 [overcloud-ControllerNodesPostDeployment-x3mjnkcb666o-ControllerOvercloudServicesDeployment_Step4-f4groimtpsk3]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 Stack overcloud CREATE_FAILED Deployment failed: Heat Stack create failed. Upgrade keystoneauth from 2.4 to 2.11 allows openstack to install undercloud, but requires some changes on tripleo-heat templates && puppet's side. I believe so.
Issue resolved. See :- https://www.redhat.com/archives/rdo-list/2016-August/msg00099.html
Also :- https://www.redhat.com/archives/rdo-list/2016-August/msg00105.html
Ok, so closing issue then...