Bug 459562

Summary: [PATCH] charset configuration fix for luci
Product: Red Hat Enterprise Linux 5 Reporter: Satoru SATOH <ssato>
Component: congaAssignee: Ryan McCabe <rmccabe>
Status: CLOSED ERRATA QA Contact:
Severity: medium Docs Contact:
Priority: high    
Version: 5.2CC: bbrock, bstevens, cluster-maint, rmccabe
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-01-20 16:55:55 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Satoru SATOH 2008-08-20 01:28:16 EDT
Description of problem:
Any who set web browser's lang to any other than english might not
be able to access luci web interface at all because of the internal
error (the error message on web page is "'charmap' codec can't encode
characters ....").


Version-Release number of selected component (if applicable):
luci-0.12.0-7.el5

How reproducible: always


Steps to Reproduce:
1. set browser lang to any non english 
2. init luci
3. open luci web page with the browser
  
Actual results: internal error


Expected results: the web page is shown correctly


Additional info:
I think that this is because the default encoding specified in /var/lib/luci/etc/zope.conf is not appropriate at all especially for non-english languages, e.g. Japanese.

Here is a configuration change for this issue which works well as far as I tested and think better than current default even for English:


--- /var/lib/luci/etc/zope.conf.org     2008-08-13 01:20:36.000000000 +0900
+++ /var/lib/luci/etc/zope.conf 2008-08-13 00:40:28.000000000 +0900
@@ -871,6 +871,7 @@
 # Example:
 #
 #    default_zpublisher_encoding utf-8
+default-zpublisher-encoding utf-8

 # Directives: servers
 #
Comment 1 Satoru SATOH 2008-08-21 13:53:41 EDT
Here is the error message shown in the web.

-----------------------------------------------------------------------------
Site Error

An error was encountered while publishing this resource.

UnicodeEncodeError
'charmap' codec can't encode characters in position 93-100: character maps to (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: 'charmap' codec can't encode characters in position 69-75: character maps to <undefined>)Traceback (innermost last): Module ZPublisher.Publish, line 196, in publish_module_standard Module Products.PlacelessTranslationService.PatchStringIO, line 34, in new_publish Module ZPublisher.Publish, line 146, in publish Module Zope2.App.startup, line 222, in zpublisher_exception_hook Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Products.CMFCore.FSPythonScript, line 108, in __call__ Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.CMFCore.FSPythonScript, line 164, in _exec Module None, line 14, in require_login - - Line 14 Module Products.CMFFormController.ControllerPageTemplate, line 74, in __call__ Module Products.CMFFormController.BaseControllerPageTemplate, line 31, in _call Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.PageTemplates.ZopePageTemplate, line 255, in _exec Module Products.PageTemplates.PageTemplate, line 104, in pt_render - Module TAL.TALInterpreter, line 238, in __call__ Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 749, in do_useMacro Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 749, in do_useMacro Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 310, in do_startEndTag Module TAL.TALInterpreter, line 356, in do_startTag Module Products.PlacelessTranslationService.FasterStringIO, line 119, in write Module ZPublisher.HTTPResponse, line 454, in _encode_unicode Module encodings.iso8859_15, line 18, in encode UnicodeEncodeError: (see above)

Troubleshooting Suggestions

    * The URL may be incorrect.
    * The parameters passed to this resource may be incorrect.
    * A resource that this resource relies on may be encountering an error.

For more detailed information about the error, please refer to the error log.

If the error persists please contact the site maintainer. Thank you for your patience. 
-------------------------------------------------------------------------
Comment 2 Satoru SATOH 2008-08-21 14:14:10 EDT
Here is the Zope log (/var/lib/luci/log/event.log):

2008-08-22T03:03:25 ERROR root Exception while rendering an error message
Traceback (most recent call last):
  File "/usr/lib/luci/zope/lib/python/OFS/SimpleItem.py", line 223, in raise_standardErrorMessage
    v = s(**kwargs)
  File "/usr/lib/luci/zope/lib/python/Products/CMFCore/FSPythonScript.py", line 108, in __call__
    return Script.__call__(self, *args, **kw)
  File "/usr/lib/luci/zope/lib/python/Shared/DC/Scripts/Bindings.py", line 311, in __call__
    return self._bindAndExec(args, kw, None)
  File "/usr/lib/luci/zope/lib/python/Shared/DC/Scripts/Bindings.py", line 348, in _bindAndExec
    return self._exec(bound_data, args, kw)
  File "/usr/lib/luci/zope/lib/python/Products/CMFCore/FSPythonScript.py", line 164, in _exec
    result = f(*args, **kw)
  File "Script (Python)", line 25, in standard_error_message
  File "/usr/lib/luci/zope/lib/python/Shared/DC/Scripts/Bindings.py", line 311, in __call__
    return self._bindAndExec(args, kw, None)
  File "/usr/lib/luci/zope/lib/python/Shared/DC/Scripts/Bindings.py", line 348, in _bindAndExec
    return self._exec(bound_data, args, kw)
  File "/usr/lib/luci/zope/lib/python/Products/CMFCore/FSPageTemplate.py", line 195, in _exec
    result = self.pt_render(extra_context=bound_names)
  File "/usr/lib/luci/zope/lib/python/Products/CMFCore/FSPageTemplate.py", line 134, in pt_render
    result = FSPageTemplate.inheritedAttribute('pt_render')(
  File "/usr/lib/luci/zope/lib/python/Products/PageTemplates/PageTemplate.py", line 104, in pt_render
    tal=not source, strictinsert=0)()
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 238, in __call__
    self.interpret(self.program)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 281, in interpret
    handlers[opcode](self, args)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 749, in do_useMacro
    self.interpret(macro)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 281, in interpret
    handlers[opcode](self, args)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 457, in do_optTag_tal
    self.do_optTag(stuff)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 442, in do_optTag
    return self.no_tag(start, program)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 437, in no_tag
    self.interpret(program)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 281, in interpret
    handlers[opcode](self, args)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 749, in do_useMacro
    self.interpret(macro)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 281, in interpret
    handlers[opcode](self, args)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 310, in do_startEndTag
    self.do_startTag(stuff, self.endsep, self.endlen)
  File "/usr/lib/luci/zope/lib/python/TAL/TALInterpreter.py", line 356, in do_startTag
    self._stream_write(_nulljoin(L))
  File "/usr/lib/luci/zope/lib/python/Products/PlacelessTranslationService/FasterStringIO.py", line 119, in write
    s = response._encode_unicode(s)
  File "/usr/lib/luci/zope/lib/python/ZPublisher/HTTPResponse.py", line 454, in _encode_unicode
    return body.encode(encoding)
  File "/usr/lib/python2.4/encodings/iso8859_15.py", line 18, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 91-97: character maps to <undefined>
------
Comment 3 Satoru SATOH 2008-08-25 04:21:59 EDT
I found easier way to reproduce this problem.

 install w3m and run "LANG=ja_JP.UTF-8 w3m https://cluster-1:8084"
 ('cluster-1' must be replaced with a server name which runs luci).


I'm trying to reproduce with wget also but haven't succeeded yet.
Comment 4 Satoru SATOH 2008-08-26 02:05:15 EDT
Here is a response header log of another target server (rhel 5.2 ga) taken 
with web developer extension for firefox 3.


Response Headers - https://luci-server:8084/luci/acl_users/credentials_cookie_auth/require_login?came_from=https%3A//luci-server%3A8084/luci/homebase/index_html

Server: Zope/(Zope 2.9.8-final, python 2.4.3, linux2) ZServer/1.1 Plone/2.5.5
Date: Sun, 17 Aug 2008 08:14:48 GMT
Bobo-Exception-Line: 18
Content-Length: 3136
Bobo-Exception-Value: See the server error log for details
Bobo-Exception-File: iso8859_15.py
Bobo-Exception-Type: UnicodeEncodeError
Content-Type: text/html; charset=iso-8859-15

500 Internal Server Error
Comment 5 Ryan McCabe 2008-08-27 02:49:40 EDT
Thanks for all the info and the patch. I'll apply it to the source tree.
Comment 6 Satoru SATOH 2008-09-01 12:35:54 EDT
thanks!

Please let me know if you need further info.
Comment 7 Satoru SATOH 2008-09-08 23:11:06 EDT
No pressure, but it would be nice to have it soon because this bug may make luci
unusable for anyone use languages which characters cannot be encoded in iso8859-15.

Please let me know if you need further analysis or something.
Comment 11 Brian Brock 2008-12-16 18:34:16 EST
verified, didn't see error message listed above
Comment 13 errata-xmlrpc 2009-01-20 16:55:55 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0196.html