Description of problem: Attempt for creating Sahara cluster - vanilla 2.7.1 fails with error message: May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: 2016-05-26 20:17:38.406 15628 ERROR sahara.utils.api [req-4fc7de56-0628-4809-83b3-dd2badc8cad7 ] [instance: none, cluster: d4f8b21b-27fa-4ea5-885e-77af40488484] Request aborted with status code 500 and message 'I May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: 2016-05-26 20:17:38.410 15628 ERROR sahara.utils.api [req-4fc7de56-0628-4809-83b3-dd2badc8cad7 ] [instance: none, cluster: d4f8b21b-27fa-4ea5-885e-77af40488484] Traceback (most recent call last): May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/utils/api.py", line 97, in handler May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return func(**kwargs) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/api/acl.py", line 44, in handler May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return func(*args, **kwargs) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/validation.py", line 47, in handler May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return func(*args, **kwargs) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/api/v10.py", line 51, in clusters_create May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return u.render(api.create_cluster(data).to_wrapped_dict()) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/api.py", line 103, in create_cluster May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return _cluster_create(values, plugin) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/api.py", line 142, in _cluster_create May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: cluster, c_u.CLUSTER_STATUS_ERROR, six.text_type(e)) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: six.reraise(self.type_, self.value, self.tb) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/api.py", line 137, in _cluster_create May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: quotas.check_cluster(cluster) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/quotas.py", line 53, in check_cluster May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: _check_limits(req_limits) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/quotas.py", line 74, in _check_limits May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: avail_limits = _get_avail_limits() May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/quotas.py", line 128, in _get_avail_limits May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: limits.update(_get_cinder_limits()) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/quotas.py", line 196, in _get_cinder_limits May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: for l in b.execute_with_retries(cinder.limits.get).absolute: May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/utils/openstack/base.py", line 100, in execute_with_retries May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: method=method.__name__, error_msg=e)) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/oslo_i18n/_message.py", line 223, in __str__ May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: raise UnicodeError(msg) May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: UnicodeError: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead. Version-Release number of selected component (if applicable): RH OSP 8 [root@overcloud-controller-0 sahara]# rpm -qa | grep sahara openstack-sahara-engine-3.0.1-4.el7ost.noarch python-sahara-3.0.1-4.el7ost.noarch python-saharaclient-0.11.1-1.el7ost.noarch openstack-sahara-api-3.0.1-4.el7ost.noarch openstack-sahara-common-3.0.1-4.el7ost.noarch How reproducible: Every time Steps to Reproduce: 1. follow steps to create cluster -> http://docs.openstack.org/developer/sahara/devref/quickstart.html 2. 3. Actual results: UnicodeError: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead. Expected results: Hadoop cluster created Additional info:
Few questions: - how did you deploy Sahara? This seems to be a TripleO environment - the vanilla plugin is not enabled by default in our packaging. Does it work with HDP or CDH images? - which vanilla image did you use? - from a quick look the errors may be due to the message coming from cinder; do you see other error messages around that timestamp? - could you please provide more complete logs (sahara in debug mode, and also the other services)?
Created attachment 1162518 [details] sosreport from one controller
Created attachment 1162519 [details] sahara.conf
Luigi: - with -> https://github.com/eggmaster/osp-director-helpers/tree/sahara-poc/overcloud_heat_examples/sahara - made few modifications to accommodate osp 8 - just multiple vanilla images including centos 6.6 and 7, raw and qcow2 - I have used both cinder and nova ephemeral .. I keep getting error on both - I enabled debug and restarted cluster - the sos report attached as well as sahara.conf After rebooting cluster I am not getting following error from sahara-engine: May 27 14:45:28 overcloud-controller-2.localdomain sahara-engine[6892]: 2016-05-27 14:45:28.181 6892 ERROR sahara.service.ops [instance: none, cluster: 9c314a17-1529-4c38-88e3-44a511dfed83] HeatStackException: Heat stack failed with status Resource CREATE failed: resources.master: resources[0]: Property error: resources.inst.properties.image: Error validating value '04174409-9903-4eb2-affe-60bcc8303b5a': The Image (04174409-9903-4eb2-affe-60bcc8303b5a) could not be found. - I was able to deploy uploaded sahara images from nova to both cinder and ephemeral storage. - the error above indicates a heat error, and I verified heat service being up and no errors with pcs status.
Can you please try HDP or CDH images? About the last error, what does glance and sahara says about that image for the project you are using?
I am getting similar error with CDH : Creating cluster failed for the following reason(s): Heat stack failed with status. Resource CREATE failed: resources.master-cdh: resources[0]: Property error: resources.inst.properties.image: Error validating value '4c671edf-3127-48cf-829b-dec00b29b775': The Image (4c671edf-3127-48cf-829b-dec00b29b775) could not be found. Error ID: 6338edd7-1ea9-468a-a332-7f5d4297af96 Here is the output from glance/sahara: [stack@undercloud nfsshare]$ glance image-list +--------------------------------------+-----------------------------------+-------------+------------------+-------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+-----------------------------------+-------------+------------------+-------------+--------+ | 8c184193-377a-4810-b9a8-16da6a4ab682 | cirros | qcow2 | bare | 13287936 | active | | 08619238-4884-4c95-a702-27968ee58a22 | rhel7 | qcow2 | bare | 492961792 | active | | b4be3b6b-d3fa-4278-a0f0-ba68c28a4ef5 | rhel7-raw | raw | bare | 10737418240 | active | | 4c671edf-3127-48cf-829b-dec00b29b775 | sahara-cdh | raw | bare | 8824291328 | active | | 1c2574d9-0687-4536-863c-d28cd430ea6c | sahara-hdp | raw | bare | 10737418240 | active | | 45c38380-ad56-4705-ba4f-9a0b5fe9c73e | sahara-vanilla-centos6-raw | raw | bare | 4396023808 | active | | d82ba957-43c2-4897-8017-6800dd7ee5a5 | sahara-vanilla-latest-centos7 | qcow2 | bare | 1362165760 | active | | 072d732c-1e58-4f06-a843-e4950003abc7 | sahara-vanilla-latest-centos7-raw | raw | bare | 4321837056 | active | +--------------------------------------+-----------------------------------+-------------+------------------+-------------+--------+ [stack@undercloud nfsshare]$ glance image-show sahara-cdh +------------------------------+--------------------------------------+ | Property | Value | +------------------------------+--------------------------------------+ | Property '_sahara_tag_5.4.0' | True | | Property '_sahara_tag_cdh' | True | | Property '_sahara_username' | cloud-user | | checksum | 3c3ec3db1d40dbbd133f7d934021edad | | container_format | bare | | created_at | 2016-06-01T19:08:48.000000 | | deleted | False | | disk_format | raw | | id | 4c671edf-3127-48cf-829b-dec00b29b775 | | is_public | False | | min_disk | 0 | | min_ram | 0 | | name | sahara-cdh | | owner | 6bc92dce49fd458ea4340b5b91c479ed | | protected | False | | size | 8824291328 | | status | active | | updated_at | 2016-06-01T19:16:18.000000 | +------------------------------+--------------------------------------+ [stack@undercloud nfsshare]$ sahara image-list +-----------------------------------+--------------------------------------+------------+----------------+-------------+ | name | id | username | tags | description | +-----------------------------------+--------------------------------------+------------+----------------+-------------+ | sahara-cdh | 4c671edf-3127-48cf-829b-dec00b29b775 | cloud-user | 5.4.0, cdh | None | | sahara-hdp | 1c2574d9-0687-4536-863c-d28cd430ea6c | cloud-user | 2.0.6, hdp | None | | sahara-vanilla-centos6-raw | 45c38380-ad56-4705-ba4f-9a0b5fe9c73e | cloud-user | vanilla, 2.7.1 | None | | sahara-vanilla-latest-centos7 | d82ba957-43c2-4897-8017-6800dd7ee5a5 | centos | vanilla, 2.7.1 | None | | sahara-vanilla-latest-centos7-raw | 072d732c-1e58-4f06-a843-e4950003abc7 | centos | vanilla, 2.7.1 | None | +-----------------------------------+--------------------------------------+------------+----------------+-------------+ [stack@undercloud nfsshare]$ sahara image-show --id 4c671edf-3127-48cf-829b-dec00b29b775 +----------------------+--------------------------------------+ | Property | Value | +----------------------+--------------------------------------+ | status | ACTIVE | | username | cloud-user | | updated | 2016-06-01T19:16:18Z | | OS-EXT-IMG-SIZE:size | 8824291328 | | created | 2016-06-01T19:08:48Z | | tags | cdh, 5.4.0 | | minDisk | 0 | | name | sahara-cdh | | progress | 100 | | minRam | 0 | | id | 4c671edf-3127-48cf-829b-dec00b29b775 | | description | None | +----------------------+--------------------------------------+
You tried different images, but which kind of images are those? RHEL-based? CentOS-based? From which source, prebuilt from upstream or generated using sahara-image-elements?
Centos. I tried 6 and 7. Here is the link to the repo: http://sahara-files.mirantis.com/images/upstream/liberty/
Please try to rebuild the images with the packaged (RPM) sahara-image-elements. I remember a recent discussion (that I can't find right now) on the upstream channels where an user had no luck with the current set of prebuilt images.
Hi Chris, Looks like this investigation thread has gone a little stale; were you able to try sahara-image-elements?
Unfortunately, I no longer have access to this deployment, so I won't be able to try sahara-image-elements.
We do not support Vanilla in the product release, and do not support upstream images regardless. Closing, sadly; if you see this again please do let the upstream Sahara team know.