Bug 1340253 - Creating Vanilla cluster fails with Error - UnicodeError: Message objects do not support str() because they may contain non-ascii characters
Summary: Creating Vanilla cluster fails with Error - UnicodeError: Message objects do ...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-sahara
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Elise Gafford
QA Contact: Luigi Toscano
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-26 20:46 UTC by Chris Janiszewski
Modified: 2016-07-19 18:10 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-19 18:10:13 UTC
Target Upstream Version:


Attachments (Terms of Use)
sosreport from one controller (14.79 MB, application/x-xz)
2016-05-27 16:23 UTC, Chris Janiszewski
no flags Details
sahara.conf (39.37 KB, text/plain)
2016-05-27 16:24 UTC, Chris Janiszewski
no flags Details

Description Chris Janiszewski 2016-05-26 20:46:30 UTC
Description of problem:
Attempt for creating Sahara cluster - vanilla 2.7.1 fails with error message:

May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: 2016-05-26 20:17:38.406 15628 ERROR sahara.utils.api [req-4fc7de56-0628-4809-83b3-dd2badc8cad7 ] [instance: none, cluster: d4f8b21b-27fa-4ea5-885e-77af40488484] Request aborted with status code 500 and message 'I
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: 2016-05-26 20:17:38.410 15628 ERROR sahara.utils.api [req-4fc7de56-0628-4809-83b3-dd2badc8cad7 ] [instance: none, cluster: d4f8b21b-27fa-4ea5-885e-77af40488484] Traceback (most recent call last):
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/utils/api.py", line 97, in handler
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return func(**kwargs)
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/api/acl.py", line 44, in handler
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return func(*args, **kwargs)
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/validation.py", line 47, in handler
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return func(*args, **kwargs)
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/api/v10.py", line 51, in clusters_create
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return u.render(api.create_cluster(data).to_wrapped_dict())
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/api.py", line 103, in create_cluster
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: return _cluster_create(values, plugin)
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/api.py", line 142, in _cluster_create
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: cluster, c_u.CLUSTER_STATUS_ERROR, six.text_type(e))
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: six.reraise(self.type_, self.value, self.tb)
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/api.py", line 137, in _cluster_create
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: quotas.check_cluster(cluster)
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/quotas.py", line 53, in check_cluster
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: _check_limits(req_limits)
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/quotas.py", line 74, in _check_limits
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: avail_limits = _get_avail_limits()
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/quotas.py", line 128, in _get_avail_limits
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: limits.update(_get_cinder_limits())
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/service/quotas.py", line 196, in _get_cinder_limits
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: for l in b.execute_with_retries(cinder.limits.get).absolute:
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/sahara/utils/openstack/base.py", line 100, in execute_with_retries
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: method=method.__name__, error_msg=e))
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: File "/usr/lib/python2.7/site-packages/oslo_i18n/_message.py", line 223, in __str__
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: raise UnicodeError(msg)
May 26 20:17:38 overcloud-controller-0.localdomain sahara-api[15628]: UnicodeError: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead.



Version-Release number of selected component (if applicable):
RH OSP 8
[root@overcloud-controller-0 sahara]# rpm -qa | grep sahara
openstack-sahara-engine-3.0.1-4.el7ost.noarch
python-sahara-3.0.1-4.el7ost.noarch
python-saharaclient-0.11.1-1.el7ost.noarch
openstack-sahara-api-3.0.1-4.el7ost.noarch
openstack-sahara-common-3.0.1-4.el7ost.noarch


How reproducible:
Every time


Steps to Reproduce:
1. follow steps to create cluster -> http://docs.openstack.org/developer/sahara/devref/quickstart.html
2.
3.

Actual results:
UnicodeError: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead.

Expected results:
Hadoop cluster created

Additional info:

Comment 2 Luigi Toscano 2016-05-27 09:15:40 UTC
Few questions:

- how did you deploy Sahara? This seems to be a TripleO environment
- the vanilla plugin is not enabled by default in our packaging. Does it work with HDP or CDH images?
- which vanilla image did you use?
- from a quick look the errors may be due to the message coming from cinder; do you see other error messages around that timestamp?
- could you please provide more complete logs (sahara in debug mode, and also the other services)?

Comment 3 Chris Janiszewski 2016-05-27 16:23:27 UTC
Created attachment 1162518 [details]
sosreport from one controller

Comment 4 Chris Janiszewski 2016-05-27 16:24:28 UTC
Created attachment 1162519 [details]
sahara.conf

Comment 5 Chris Janiszewski 2016-05-27 16:30:22 UTC
Luigi:

- with -> https://github.com/eggmaster/osp-director-helpers/tree/sahara-poc/overcloud_heat_examples/sahara - made few modifications to accommodate osp 8
- just multiple vanilla images including centos 6.6 and 7, raw and qcow2
- I have used both cinder and nova ephemeral .. I keep getting error on both
- I enabled debug and restarted cluster - the sos report attached as well as sahara.conf

After rebooting cluster I am not getting following error from sahara-engine:
May 27 14:45:28 overcloud-controller-2.localdomain sahara-engine[6892]: 2016-05-27 14:45:28.181 6892 ERROR sahara.service.ops [instance: none, cluster: 9c314a17-1529-4c38-88e3-44a511dfed83] HeatStackException: Heat stack failed with status Resource CREATE failed: resources.master: resources[0]: Property error: resources.inst.properties.image: Error validating value '04174409-9903-4eb2-affe-60bcc8303b5a': The Image (04174409-9903-4eb2-affe-60bcc8303b5a) could not be found.

- I was able to deploy uploaded sahara images from nova to both cinder and ephemeral storage.
- the error above indicates a heat error, and I verified heat service being up and no errors with pcs status.

Comment 6 Luigi Toscano 2016-05-30 07:47:24 UTC
Can you please try HDP or CDH images?

About the last error, what does glance and sahara says about that image for the project you are using?

Comment 7 Chris Janiszewski 2016-06-01 19:52:15 UTC
I am getting similar error with CDH :

Creating cluster failed for the following reason(s): Heat stack failed with status. Resource CREATE failed: resources.master-cdh: resources[0]: Property error: resources.inst.properties.image: Error validating value '4c671edf-3127-48cf-829b-dec00b29b775': The Image (4c671edf-3127-48cf-829b-dec00b29b775) could not be found. Error ID: 6338edd7-1ea9-468a-a332-7f5d4297af96

Here is the output from glance/sahara:

[stack@undercloud nfsshare]$ glance image-list
+--------------------------------------+-----------------------------------+-------------+------------------+-------------+--------+
| ID                                   | Name                              | Disk Format | Container Format | Size        | Status |
+--------------------------------------+-----------------------------------+-------------+------------------+-------------+--------+
| 8c184193-377a-4810-b9a8-16da6a4ab682 | cirros                            | qcow2       | bare             | 13287936    | active |
| 08619238-4884-4c95-a702-27968ee58a22 | rhel7                             | qcow2       | bare             | 492961792   | active |
| b4be3b6b-d3fa-4278-a0f0-ba68c28a4ef5 | rhel7-raw                         | raw         | bare             | 10737418240 | active |
| 4c671edf-3127-48cf-829b-dec00b29b775 | sahara-cdh                        | raw         | bare             | 8824291328  | active |
| 1c2574d9-0687-4536-863c-d28cd430ea6c | sahara-hdp                        | raw         | bare             | 10737418240 | active |
| 45c38380-ad56-4705-ba4f-9a0b5fe9c73e | sahara-vanilla-centos6-raw        | raw         | bare             | 4396023808  | active |
| d82ba957-43c2-4897-8017-6800dd7ee5a5 | sahara-vanilla-latest-centos7     | qcow2       | bare             | 1362165760  | active |
| 072d732c-1e58-4f06-a843-e4950003abc7 | sahara-vanilla-latest-centos7-raw | raw         | bare             | 4321837056  | active |
+--------------------------------------+-----------------------------------+-------------+------------------+-------------+--------+
[stack@undercloud nfsshare]$ glance image-show sahara-cdh
+------------------------------+--------------------------------------+
| Property                     | Value                                |
+------------------------------+--------------------------------------+
| Property '_sahara_tag_5.4.0' | True                                 |
| Property '_sahara_tag_cdh'   | True                                 |
| Property '_sahara_username'  | cloud-user                           |
| checksum                     | 3c3ec3db1d40dbbd133f7d934021edad     |
| container_format             | bare                                 |
| created_at                   | 2016-06-01T19:08:48.000000           |
| deleted                      | False                                |
| disk_format                  | raw                                  |
| id                           | 4c671edf-3127-48cf-829b-dec00b29b775 |
| is_public                    | False                                |
| min_disk                     | 0                                    |
| min_ram                      | 0                                    |
| name                         | sahara-cdh                           |
| owner                        | 6bc92dce49fd458ea4340b5b91c479ed     |
| protected                    | False                                |
| size                         | 8824291328                           |
| status                       | active                               |
| updated_at                   | 2016-06-01T19:16:18.000000           |
+------------------------------+--------------------------------------+
[stack@undercloud nfsshare]$ sahara image-list
+-----------------------------------+--------------------------------------+------------+----------------+-------------+
| name                              | id                                   | username   | tags           | description |
+-----------------------------------+--------------------------------------+------------+----------------+-------------+
| sahara-cdh                        | 4c671edf-3127-48cf-829b-dec00b29b775 | cloud-user | 5.4.0, cdh     | None        |
| sahara-hdp                        | 1c2574d9-0687-4536-863c-d28cd430ea6c | cloud-user | 2.0.6, hdp     | None        |
| sahara-vanilla-centos6-raw        | 45c38380-ad56-4705-ba4f-9a0b5fe9c73e | cloud-user | vanilla, 2.7.1 | None        |
| sahara-vanilla-latest-centos7     | d82ba957-43c2-4897-8017-6800dd7ee5a5 | centos     | vanilla, 2.7.1 | None        |
| sahara-vanilla-latest-centos7-raw | 072d732c-1e58-4f06-a843-e4950003abc7 | centos     | vanilla, 2.7.1 | None        |
+-----------------------------------+--------------------------------------+------------+----------------+-------------+
[stack@undercloud nfsshare]$ sahara image-show --id 4c671edf-3127-48cf-829b-dec00b29b775
+----------------------+--------------------------------------+
| Property             | Value                                |
+----------------------+--------------------------------------+
| status               | ACTIVE                               |
| username             | cloud-user                           |
| updated              | 2016-06-01T19:16:18Z                 |
| OS-EXT-IMG-SIZE:size | 8824291328                           |
| created              | 2016-06-01T19:08:48Z                 |
| tags                 | cdh, 5.4.0                           |
| minDisk              | 0                                    |
| name                 | sahara-cdh                           |
| progress             | 100                                  |
| minRam               | 0                                    |
| id                   | 4c671edf-3127-48cf-829b-dec00b29b775 |
| description          | None                                 |
+----------------------+--------------------------------------+

Comment 8 Luigi Toscano 2016-06-08 16:09:26 UTC
You tried different images, but which kind of images are those? RHEL-based? CentOS-based? From which source, prebuilt from upstream or generated using sahara-image-elements?

Comment 9 Chris Janiszewski 2016-06-08 19:31:46 UTC
Centos. I tried 6 and 7. Here is the link to the repo:
http://sahara-files.mirantis.com/images/upstream/liberty/

Comment 10 Luigi Toscano 2016-06-08 21:05:53 UTC
Please try to rebuild the images with the packaged (RPM) sahara-image-elements. I remember a recent discussion (that I can't find right now) on the upstream channels where an user had no luck with the current set of prebuilt images.

Comment 11 Elise Gafford 2016-07-13 16:48:40 UTC
Hi Chris,

Looks like this investigation thread has gone a little stale; were you able to try sahara-image-elements?

Comment 12 Chris Janiszewski 2016-07-13 16:53:36 UTC
Unfortunately, I no longer have access to this deployment, so I won't be able to try sahara-image-elements.

Comment 13 Elise Gafford 2016-07-19 18:10:13 UTC
We do not support Vanilla in the product release, and do not support upstream images regardless. Closing, sadly; if you see this again please do let the upstream Sahara team know.


Note You need to log in before you can comment on or make changes to this bug.