Bug 508686 - Squirrelmail could not use GB2312 for incorrect charset at i18n.php
Squirrelmail could not use GB2312 for incorrect charset at i18n.php
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: squirrelmail (Show other bugs)
5.3
All Linux
low Severity medium
: rc
: ---
Assigned To: Michal Hlavinka
qe-baseos-daemons
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-06-29 09:46 EDT by Joe Jin
Modified: 2013-01-07 23:58 EST (History)
4 users (show)

See Also:
Fixed In Version: squirrelmail-1.4.8-21.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-01-07 23:58:17 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Joe Jin 2009-06-29 09:46:52 EDT
Description of problem:

When try to deployed squirrelmail, configured display language to "Chinese Simp" or "Chinese Trad",
it could not display mail folder and reported "Reason Given: GB2312 character set is not supported."

Checked locale/zh_CN[zh_TW]/LC_MESSAGES/squirrelmail.po, file charset encode is utf-8,
also locale/zh_CN[zh_TW]/setup.php charset to utf-8, but at functions/i18n.php,
it used charset of zh_CN/zh_TW to gb2312/big5. after changed it to utf-8, it working fine.


How reproducible:

Always.
Comment 1 Michal Hlavinka 2009-06-30 03:44:01 EDT
thanks for reporting this

verified, squirrelmail.po is using utf8, but i18n.php is using big5

locales are regenerated using utf8 everywhere, but original translation is using different encoding.

Possible broken are: Korean, Japanese, Chinese Trad, Chinese Simp

we need to: 
a)regenerate locales using original encoding (preferred)
or
b)change charset information in i18n.php
Comment 2 Tomas 2009-09-12 02:32:56 EDT
Name your IMAP server. This error is not always reproducible. You need IMAP server which does not support GB2312 and mail folder must be sorted.
Comment 3 Joe Jin 2009-09-18 19:10:06 EDT
(In reply to comment #2)
> Name your IMAP server. This error is not always reproducible. You need IMAP
> server which does not support GB2312 and mail folder must be sorted.  

Now I have not the env for it, but here, I have to say when set language to 
zh_CN or zh_TW, right frame of login page display error, left frame works.

I guess IMAP server's name just related folder's name, and would not related
page, isn't it?

And, the root caused it inconsistently of configure file(i18n.php) and encode file.
Comment 4 Tomas 2009-09-19 02:59:04 EDT
No i(In reply to comment #3)
> (In reply to comment #2)
> > Name your IMAP server. This error is not always reproducible. You need IMAP
> > server which does not support GB2312 and mail folder must be sorted.  
> 
> Now I have not the env for it, but here, I have to say when set language to 
> zh_CN or zh_TW, right frame of login page display error, left frame works.
> 
> I guess IMAP server's name just related folder's name, and would not related
> page, isn't it?

No. Which IMAP server are you using? That's what I mean by "name your server" question.

IMAP SORT extension requires support of US-ASCII and UTF-8. Other character sets are optional. Courier sets list of available charsets when it is compiled and option which compiles all available charsets is not enabled by default. If I remember correctly, Cyrus also does not support some of charsets used in standard SquirrelMail package.

If you switch CJK translations to utf-8 without including CJK decoding functions, you'll get lots of bug reports from East Asia.

It does not matter that all locale files are in utf-8 and functions/i18n.php uses other charset for translation. Redhat does not use PHP 4.1 and PHP 4.2+ gettext functions convert translations to charsets defined in functions/i18n.php

It is possible to fix this issue in scripts. SquirrelMail can detect SORT extension error and switch to internal sorting. Fix is not related to translations. It is applied to mailbox display code and (if I remember code correctly) to some SquirrelMail IMAP functions.
Comment 5 RHEL Product and Program Management 2009-11-06 13:45:37 EST
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release. If you would like
this request to be reviewed for the next minor release, ask your
support representative to set the next rhel-x.y flag to "?".
Comment 6 Michal Hlavinka 2009-12-16 08:00:54 EST
Joe Jin, could you please provide requested info? Thanks
Comment 7 Joe Jin 2009-12-16 19:31:52 EST
IMAP server is cyrus-imap.
system default locale is zh_CN.UTF-8

$ locale
LANG=zh_CN.UTF-8
LC_CTYPE="zh_CN.UTF-8"
LC_NUMERIC="zh_CN.UTF-8"
LC_TIME="zh_CN.UTF-8"
LC_COLLATE="zh_CN.UTF-8"
LC_MONETARY="zh_CN.UTF-8"
LC_MESSAGES="zh_CN.UTF-8"
LC_PAPER="zh_CN.UTF-8"
LC_NAME="zh_CN.UTF-8"
LC_ADDRESS="zh_CN.UTF-8"
LC_TELEPHONE="zh_CN.UTF-8"
LC_MEASUREMENT="zh_CN.UTF-8"
LC_IDENTIFICATION="zh_CN.UTF-8"
LC_ALL=

Access the box from MS Windows for Simplified Chinese, failed
from Linux with zh_CN.UTF-8 also failed.

If set the language is en_US, all are OK and page's language is English.
Comment 8 Tomas 2009-12-17 00:37:10 EST
Are you sure about cyrus? If you have some proxy in front of real Cyrus IMAP server, try using webmail without that proxy.

----
tomas@miri:~$ telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK miri Cyrus IMAP4 v2.2.13-Debian-2.2.13-14+lenny3 server ready
A01 login ******** ********
A01 OK User logged in
A02 select inbox
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  
* 1 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1248285231]  
* OK [UIDNEXT 2]  
A02 OK [READ-WRITE] Completed
A02 UID SORT (SUBJECT) GB2312 ALL
* SORT 1
A02 OK Completed (1 msgs in 0.000 secs)
A03 UID THREAD REFERENCES GB2312 ALL
* THREAD (1)
A03 OK Completed (1 msgs in 0.000 secs)
A04 UID THREAD ORDEREDSUBJECT GB2312 ALL
* THREAD (1)
A04 OK Completed (1 msgs in 0.000 secs)
A03 UID SEARCH CHARSET GB2312 ALL
* SEARCH 1
A03 OK Completed (1 msgs in 0.000 secs)
A05 LOGOUT
* BYE LOGOUT received
A05 OK Completed
Connection closed by foreign host.
tomas@miri:~$ 
----

Cyrus supports GB2312 in search, sort and thread commands. "character set is not supported" comes not from Cyrus. They don't have this string in their sources. Cyrus IMAP would say "NO Unrecognized character set". Are you using avelsieve plugin?

-- 
Tomas
Comment 9 Tomas 2009-12-17 00:49:04 EST
"character set is not supported" might come from Courier IMAP compiled without --enable-unicode option. I repeat it is not related to locale. It does not matter what kind of locale you have on your server. This error is a glitch in the way SquirrelMail handles SORT/THREAD errors. Code can be changed to display mailbox after SORT/THREAD error by modifing sorting and threading functions. Error won't go away, but user will be able to see his or her mailbox, if mailbox does not exceed 2-3k messages.
Comment 10 Joe Jin 2009-12-17 00:56:27 EST
(In reply to comment #8)
> Are you sure about cyrus? If you have some proxy in front of real Cyrus IMAP
> server, try using webmail without that proxy.

I accessed webmail without proxy.

> 
> Cyrus supports GB2312 in search, sort and thread commands. "character set is
> not supported" comes not from Cyrus. They don't have this string in their
> sources. Cyrus IMAP would say "NO Unrecognized character set". Are you using
> avelsieve plugin?
> 
> -- 

I did not used the plugin what you mentioned.
"character set is not supported" caused by Squirremail's config file.

As I have mentioned:

Checked locale/zh_CN[zh_TW]/LC_MESSAGES/squirrelmail.po, file charset encode is
utf-8,
also locale/zh_CN[zh_TW]/setup.php charset to utf-8, but at functions/i18n.php,
it used charset of zh_CN/zh_TW to gb2312/big5. after changed it to utf-8, it
working fine.
Comment 11 Tomas 2009-12-17 02:34:16 EST
(In reply to comment #10)
> "character set is not supported" caused by Squirremail's config file.
> 
> As I have mentioned:
> 
> Checked locale/zh_CN[zh_TW]/LC_MESSAGES/squirrelmail.po, file charset encode is
> utf-8,
> also locale/zh_CN[zh_TW]/setup.php charset to utf-8, but at functions/i18n.php,
> it used charset of zh_CN/zh_TW to gb2312/big5. after changed it to utf-8, it
> working fine.  

Setting locale to certain value triggers it, but it does not mean that locale itself is buggy. It is not locale problem. By changing locale charset you remove condition which triggers that error.

It does not matter that charsets don't match in functions/i18n.php and squirrelmail.po/squirrelmail.mo files. I have already said that gettext will convert utf-8 squirrelmail strings to gb2312. If gettext does not support gb2312, error will pop in webmail.php or login.php and not in right_main.php

SquirrelMail does not have "character set is not supported" string either. If error is displayed in mailbox listing, string comes from IMAP server and this server is not Cyrus IMAP.

Turn off server side thread and sorting support and filtering plugins in SquirrelMail configuration and error should go away.

Please note that I know how to reproduce this error, what triggers it and how to fix it, but I also know that you can't reproduce it on standard unmodified Cyrus IMAP setup. You are providing misleading information about your setup or redhat/fedora deliberately removed support of GB2312 from Cyrus/PHP/Gettext or some other software included in your apache/cyrus/php/squirrelmail setup.

If you want to fix it, you should make sure that your IMAP server supports GB2312 character set. This fix changes only one component and allows your Chinese users to read their emails. If you change charset to utf-8, you must install SquirrelMail extra decoding library and PHP recode extension. If you don't do that, error will go away, but Chinese won't be able to read emails written in GB2312 and GB18030 and these are main charsets used in continental Chinese emails. The third fix is to fix showMessagesForMailbox() function in SquirrelMail functions/mailbox_display.php and get_thread_sort()/sqimap_get_sort_order() functions.
Comment 13 RHEL Product and Program Management 2010-08-09 14:21:01 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 14 RHEL Product and Program Management 2011-01-11 15:23:05 EST
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 15 RHEL Product and Program Management 2011-01-12 10:11:53 EST
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.
Comment 16 RHEL Product and Program Management 2011-05-31 09:21:14 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 17 RHEL Product and Program Management 2012-04-02 06:21:40 EDT
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux release.  Product Management has
requested further review of this request by Red Hat Engineering, for
potential inclusion in a Red Hat Enterprise Linux release for currently
deployed products.  This request is not yet committed for inclusion in
a release.
Comment 20 Michal Hlavinka 2012-06-26 14:20:22 EDT
Joe Jin:
Tomas is right that if we change charset to utf-8, we'd only got mail corruption. I found out that some people tried it and it did not work well:
"""after changed it to utf-8, it working fine.However, all messages received is corrupted characters in chinese Either the title is corrupted characters, or information is corrupted characters"""

Anyway, I tried cyrus-imapd (2.3.7 as found in rhel5) with squirrelmail and both zh_CN and zh_TW and did not see any problem. 

I also verified that there really is no "character set is not supported" string in whole cyrus-imapd nor dovecot. Cyrus-imapd would report "Unrecognized character set" as Tomas said. 

Also as you can see at http://squirrelmail.org/wiki/CourierUnsupportedCharset this error message is really known to originate from courier-imap. I can't forward this bug to courier-imap maintainer to fix it, simply because there is no courier-imap maintainer in red hat as we don't have this package in rhel.

One way you can get rid of that error is turning off server side Turn off server side thread and sorting support as mentioned in the link above.
Comment 21 Tomas 2012-06-28 14:27:52 EDT
> """after changed it to utf-8, it working fine.However, all messages 
> received is corrupted characters in chinese Either the title is 
> corrupted characters, or information is corrupted characters"""

You need SquirrelMail extra decoding library functions to use Chinese SquirrelMail interface in utf-8. Stock install does not have functions to support common Chinese character sets. You can run Chinese SquirrelMail in utf-8 without problems in correctly formated messages. You just need some help and guidance from SquirrelMail i18n experts. The only problem will be in messages that violate MIME specs.
Comment 22 Michal Hlavinka 2012-07-02 12:09:34 EDT
Tomas, could you tell me when exactly are those extra decoding functions needed? I checked svn repository and I can see there is utf-8 used for zh_TW (not for zh_CN) since 2008. So, will it work with iconv or recode is really needed here? Also, do you know why it was changed only for traditional Chinese and not for both? Thanks
Comment 23 Tomas 2012-07-02 12:46:11 EDT
See http://www.squirrelmail.org/download.php

SquirrelMail Webmail decoding functions are used to display and convert messages encoded in different character sets. This extra decoding library provides additional support for some complex Eastern and Apple x-mac character sets.

Version: 1.2

squirrelmail-decode-1.2.tar.gz (12552 d/l)
squirrelmail-decode-1.2.tar.bz2 (6143 d/l)
squirrelmail-decode-1.2.zip (8229 d/l)

http://squirrelmail.svn.sourceforge.net/viewvc/squirrelmail/trunk/decode/

Functions work without recode too. They use any recoding function available.
First recode, then iconv, then mbstring, then they go to raw PHP mapping tables. Iconv functions have limited error handling, but I no longer develop for SquirrelMail and I refuse to resign my copyrights on charset conversion updates to SquirrelMail. Code in question is available under GPL and they are free to use it as long they keep correct attributions and copyrights.

In current setup Chinese and Korean translations that are not in utf-8 have disadvantage in encoding functions. Application does not have functions for big5, euc-kr and gb2312 encoding (http://squirrelmail.svn.sourceforge.net/viewvc/squirrelmail/trunk/squirrelmail/functions/encode/).
Comment 27 errata-xmlrpc 2013-01-07 23:58:17 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0126.html

Note You need to log in before you can comment on or make changes to this bug.