Hide Forgot
Created attachment 1114839 [details] Example of group of YAML files that contain the unicode character Description of problem: When a customer uses custom heat template files in OSP Director, sometimes stray unicode characters get introduced and the customer will get the following error when trying to do an overcoud deploy: ERROR: openstack ERROR: Remote error: UnicodeEncodeError 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) [u' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 1223, in take_action self._deploy_tripleo_heat_templates(stack, parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 527, in _deploy_tripleo_heat_templates environments, parsed_args.timeout) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 446, in _heat_deploy orchestration_client, stack_name) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/utils.py", line 145, in wait_for_stack_ready stack = orchestration_client.stacks.get(stack_name) File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 202, in get resp, body = self.client.json_request('GET', '/stacks/%s' % stack_id) File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 265, in json_request resp = self._http_request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 227, in _http_request resp = self._http_request(path, method, **kwargs) File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 220, in _http_request raise exc.from_response(resp) HTTPInternalServerError: ERROR: Remote error: UnicodeEncodeError 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) [u Would it be possible to fix this by scrubbing in the incoming yaml files before processing? Attached are example yaml files that exhibit the issue. Version-Release number of selected component (if applicable):7.1, 7.2 How reproducible: 100% Steps to Reproduce: 1.Create YAML files with unicode character within them 2.Deploy overcloud 3.Error during overcloud deploy Actual results: Unicode error and deploy fails Expected results: Deployment should be successful Additional info: We can work around the issue if we run the following on the yaml files: perl -CD -i.bak -pe 's/\x{200b}//g' *.yaml Then delete the overcloud stack: heat stack-delete overcloud Then attempt a new deploy which will be successful.
It makes sense to improve Heat's support for unicode, but it doesn't make sense to use a toolchain that inserts random garbage (in this case zero-width spaces) into your templates. Having Heat scrub the incoming yaml files doesn't make a lot of sense to me - there's no way we can maintain a blacklist of every non-printing unicode character. My best guess is that this is occurring as a result of copying-and-pasting from a web page. Incidentally, none of the attached files contain the offending character, or any other non-ASCII character ('cat -v *.yaml | grep M-' prints nothing). Without a traceback from heat-engine itself, it's impossible to know where the problem is manifesting.
@Zane here is the traceback from the heat-engine.log: 2016-01-13 10:59:54.321 3852 INFO heat.common.urlfetch [-] Fetching data from file:///home/stack/templates/puppet/swift-devices-and-proxy-config.yaml 2016-01-13 10:59:54.322 3853 ERROR oslo_messaging.rpc.dispatcher [req-e5b19a3a-718c-4953-9e95-8b9ad2101270 30deef294e274d5aaa13dbcbc55b4dfa 9fbfdd7dfc5c48e88aadb47d59756257] Exception during message handling: 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher executor_callback)) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher executor_callback) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 300, in wrapped 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher return func(self, ctx, *args, **kwargs) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 502, in list_stacks 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher return [api.format_stack(stack) for stack in stacks] 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/engine/api.py", line 99, in format_stack 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher rpc_api.STACK_PARAMETERS: stack.parameters.map(str), 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/engine/parameters.py", line 526, in map 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher for n, p in six.iteritems(self.params) if filter_func(p)) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/heat/engine/parameters.py", line 526, in <genexpr> 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher for n, p in six.iteritems(self.params) if filter_func(p)) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher UnicodeEncodeError: 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) 2016-01-13 10:59:54.322 3853 TRACE oslo_messaging.rpc.dispatcher 2016-01-13 10:59:54.322 3853 ERROR oslo_messaging._drivers.common [req-e5b19a3a-718c-4953-9e95-8b9ad2101270 30deef294e274d5aaa13dbcbc55b4dfa 9fbfdd7dfc5c48e88aadb47d59756257] Returning exception 'ascii' codec can't encode character u'\u200b' in position 39: ordinal not in range(128) to caller 2016-01-13 10:59:54.322 3853 ERROR oslo_messaging._drivers.common [req-e5b19a3a-718c-4953-9e95-8b9ad2101270 30deef294e274d5aaa13dbcbc55b4dfa 9fbfdd7dfc5c48e88aadb47d59756257] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 300, in wrapped\n return func(self, ctx, *args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 502, in list_stacks\n return [api.format_stack(stack) for stack in stacks]\n', ' File "/usr/lib/python2.7/site-packages/heat/engine/api.py", line 99, in format_stack\n rpc_api.STACK_PARAMETERS: stack.parameters.map(str),\n', ' File "/usr/lib/python2.7/site-packages/heat/engine/parameters.py", line 526, in map\n for n, p in six.iteritems(self.params) if filter_func(p))\n', ' File "/usr/lib/python2.7/site-packages/heat/engine/parameters.py", line 526, in <genexpr>\n for n, p in six.iteritems(self.params) if filter_func(p))\n', "UnicodeEncodeError: 'ascii' codec can't encode character u'\\u200b' in position 39: ordinal not in range(128)\n"]
Thanks! Upstream bug created.
OK, this is fixed in the latest RHOS 8 build; we can clone this bug to RHOS 7 if someone wants a backport.
Please, do backport to OSP7. This is preventing our OpenStack cluster from working properly.
(In reply to Felipe Alfaro Solana from comment #15) > Please, do backport to OSP7. This is preventing our OpenStack cluster from > working properly. Felipe, is there a reason you cannot do the recommended workaround of removing the non-ascii characters from your parameter values? These are likely to cause other problems anyway.
Felipe - to be more specific, it depends on the reason that you're passing non-ASCII data. If it's non-breaking space characters specifically ('\u200b') as in the original bug report then you *really* want to not do that. If it's just that you have unicode data that you need to pass as parameters then a backport will help.
if it can handle Klingon... [stack@undercloud ~]$ heat stack-show a002 [trimmed] | | "output_value": "",