| Summary: | Remote error: UnicodeEncodeError 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Benjamin Schmaus <bschmaus> | ||||
| Component: | openstack-heat | Assignee: | Zane Bitter <zbitter> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Amit Ugol <augol> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.0 (Kilo) | CC: | felipe.alfaro, mburns, mcornea, mlopes, rhel-osp-director-maint, sbaker, shardy, srevivo, zbitter | ||||
| Target Milestone: | ga | Keywords: | TestOnly, ZStream | ||||
| Target Release: | 8.0 (Liberty) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-heat-5.0.1-1.el7ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
Previously, non-ASCII characters were not supported in parameter input values.
Consequently, showing a stack, or listing stacks, would fail with an error if any of the parameters values contained non-ASCII characters.
With this fix, all unicode characters are now supported. As a result, stacks can be listed or shown even if their input parameters contain non-ASCII characters.
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1332040 (view as bug list) | Environment: | |||||
| Last Closed: | 2016-04-28 13:51:41 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1332040 | ||||||
| Attachments: |
|
||||||
It makes sense to improve Heat's support for unicode, but it doesn't make sense to use a toolchain that inserts random garbage (in this case zero-width spaces) into your templates. Having Heat scrub the incoming yaml files doesn't make a lot of sense to me - there's no way we can maintain a blacklist of every non-printing unicode character.
My best guess is that this is occurring as a result of copying-and-pasting from a web page.
Incidentally, none of the attached files contain the offending character, or any other non-ASCII character ('cat -v *.yaml | grep M-' prints nothing).
Without a traceback from heat-engine itself, it's impossible to know where the problem is manifesting.
@Zane here is the traceback from the heat-engine.log: 2016-01-13 10:59:54.321 3852 INFO heat.common.urlfetch [-] Fetching data from file:///home/stack/templates/puppet/swift-devices-and-proxy-config.yaml 2016-01-13 10:59:54.322 3853 ERROR oslo_messaging.rpc.dispatcher [req-e5b19a3a-718c-4953-9e95-8b9ad2101270 30deef294e274d5aaa13dbcbc55b4dfa 9fbfdd7dfc5c48e88aadb47d59756257] Exception during message handling: 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher executor_callback)) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher executor_callback) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 300, in wrapped 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher return func(self, ctx, *args, **kwargs) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 502, in list_stacks 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher return [api.format_stack(stack) for stack in stacks] 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/engine/api.py", line 99, in format_stack 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher rpc_api.STACK_PARAMETERS: stack.parameters.map(str), 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/engine/parameters.py", line 526, in map 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher for n, p in six.iteritems(self.params) if filter_func(p)) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/engine/parameters.py", line 526, in <genexpr> 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher for n, p in six.iteritems(self.params) if filter_func(p)) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher UnicodeEncodeError: 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher 2016-01-13 10:59:54.322 3853 ERROR oslo_messaging._drivers.common [req-e5b19a3a-718c-4953-9e95-8b9ad2101270 30deef294e274d5aaa13dbcbc55b4dfa 9fbfdd7dfc5c48e88aadb47d59756257] Returning exception 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) to caller 2016-01-13 10:59:54.322 3853 ERROR oslo_messaging._drivers.common [req-e5b19a3a-718c-4953-9e95-8b9ad2101270 30deef294e274d5aaa13dbcbc55b4dfa 9fbfdd7dfc5c48e88aadb47d59756257] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 300, in wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 502, in list_stacks\n return [api.format_stack(stack) for stack in stacks]\n', ' File "/usr/lib/python2.7/site-packages/heat/engine/api.py", line 99, in format_stack\n rpc_api.STACK_PARAMETERS: stack.parameters.map(str),\n', ' File "/usr/lib/python2.7/site-packages/heat/engine/parameters.py", line 526, in map\n for n, p in six.iteritems(self.params) if filter_func(p))\n', ' File "/usr/lib/python2.7/site-packages/heat/engine/parameters.py", line 526, in <genexpr>\n for n, p in six.iteritems(self.params) if filter_func(p))\n', "UnicodeEncodeError: 'ascii' codec can't encode character u'\\u200b' in position 39: ordinal not in range(128)\n"] Thanks! Upstream bug created. OK, this is fixed in the latest RHOS 8 build; we can clone this bug to RHOS 7 if someone wants a backport. Please, do backport to OSP7. This is preventing our OpenStack cluster from working properly. (In reply to Felipe Alfaro Solana from comment #15) > Please, do backport to OSP7. This is preventing our OpenStack cluster from > working properly. Felipe, is there a reason you cannot do the recommended workaround of removing the non-ascii characters from your parameter values? These are likely to cause other problems anyway. Felipe - to be more specific, it depends on the reason that you're passing non-ASCII data. If it's non-breaking space characters specifically ('\u200b') as in the original bug report then you *really* want to not do that. If it's just that you have unicode data that you need to pass as parameters then a backport will help.
if it can handle Klingon... [stack@undercloud ~]$ heat stack-show a002 [trimmed] | | "output_value": "", |
Created attachment 1114839 [details] Example of group of YAML files that contain the unicode character Description of problem: When a customer uses custom heat template files in OSP Director, sometimes stray unicode characters get introduced and the customer will get the following error when trying to do an overcoud deploy: ERROR: openstack ERROR: Remote error: UnicodeEncodeError 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) [u' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 1223, in take_action self._deploy_tripleo_heat_templates(stack, parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 527, in _deploy_tripleo_heat_templates environments, parsed_args.timeout) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 446, in _heat_deploy orchestration_client, stack_name) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/utils.py", line 145, in wait_for_stack_ready stack = orchestration_client.stacks.get(stack_name) File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 202, in get resp, body = self.client.json_request('GET', '/stacks/%s' % stack_id) File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 265, in json_request resp = self._http_request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 227, in _http_request resp = self._http_request(path, method, **kwargs) File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 220, in _http_request raise exc.from_response(resp) HTTPInternalServerError: ERROR: Remote error: UnicodeEncodeError 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) [u Would it be possible to fix this by scrubbing in the incoming yaml files before processing? Attached are example yaml files that exhibit the issue. Version-Release number of selected component (if applicable):7.1, 7.2 How reproducible: 100% Steps to Reproduce: 1.Create YAML files with unicode character within them 2.Deploy overcloud 3.Error during overcloud deploy Actual results: Unicode error and deploy fails Expected results: Deployment should be successful Additional info: We can work around the issue if we run the following on the yaml files: perl -CD -i.bak -pe 's/\x{200b}//g' *.yaml Then delete the overcloud stack: heat stack-delete overcloud Then attempt a new deploy which will be successful.