Bug 162852 - Character handling in squirrelmail
Character handling in squirrelmail
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: squirrelmail (Show other bugs)
rawhide
All Linux
medium Severity medium
: ---
: ---
Assigned To: Warren Togami
https://sourceforge.net/tracker/?func...
:
: 167877 (view as bug list)
Depends On:
Blocks: 171491 FC5Blocker
  Show dependency treegraph
 
Reported: 2005-07-10 09:56 EDT by Nicolas Mailhot
Modified: 2007-11-30 17:11 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-03-04 08:44:56 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Problem screenshot (310.82 KB, image/png)
2005-09-14 02:13 EDT, Nicolas Mailhot
no flags Details
In case this one still applies... (12.74 KB, patch)
2006-02-28 17:51 EST, Nicolas Mailhot
no flags Details | Diff

  None (edit)
Description Nicolas Mailhot 2005-07-10 09:56:02 EDT
Description of problem:

It seems squirrelmail is still stuck in the bad-old-days of multiple charsets,
with the main developpers lacking the will to enforce UTF-8 accross translators.

As a result every translation forces a single encoding and the default charset
parameter is ignored for everything except en_US.

Of course that also means if you reply to a mail someone sent you and your UI
translation forces a charset different from the one in this mail your quotes
will be b0rked.

Since upstream seems decided to let the mess perdure, the Fedora package should
convert all translations to UTF-8 so FC squirrelmail interracts properly with
the world (and not just the language zone the UI translation uses)

(See squirrelmail bug #1235345 for upstream position)
Comment 1 Warren Togami 2005-09-04 05:56:42 EDT
Closing this as this is work required upstream and not Fedora's sole responsibility.
Comment 2 David Woodhouse 2005-09-04 06:06:00 EDT
Do you have a reference to the squirrelmail bug (apparently #1235345)? I don't
seem to be able to find it.
Comment 3 Warren Togami 2005-09-09 18:30:02 EDT
*** Bug 167877 has been marked as a duplicate of this bug. ***
Comment 4 Nicolas Mailhot 2005-09-11 16:20:23 EDT
Fedora is the party that cares about software being UTF-8 compatible.
Not handling UTF-8 well was sufficient ground for removal in the past.

Unless this Red Hat policy changed (which I doubt) if no one steps up to do the
UTF-8 work, suirrelmail should leave FC IMHO.

Which would be a pity as I use and like it.
Comment 6 Warren Togami 2005-09-11 17:51:21 EDT
> Unless this Red Hat policy changed (which I doubt) if no one steps up to do 
> the UTF-8 work, squirrelmail should leave FC IMHO.

Agreed.

> Which would be a pity as I use and like it.

I know. =(
Comment 7 jørgen nørgaard 2005-09-12 03:26:23 EDT
From http://www.squirrelmail.org/wiki/en_US/WishList, where it is stated:

"From 1.5.1cvs and 1.4.4cvs squirrelmail allows setting default charset when default language is en_US"

Sounds like there should be a solution, though I still have to figure out how it is actually done, just saw it 
and I dont have time right now.

I surely would miss squirrelmail if it went away from FC. It understandable if it is removed from a Redhat 
product on the other hand.

Comment 8 Warren Togami 2005-09-12 04:26:27 EDT
It is sure a pity because I used squirrelmail for years, but it would probably
move to Extras if it does become removed.  If you are aware of a standards
compliant replacement that is secure and proven mature then please chime in here.
Comment 9 David Woodhouse 2005-09-12 04:49:36 EDT
Shouldn't be too hard just to convert all the locales to UTF-8 while we're
building the package. That ensures that their charset is a superset of the
charset in any mail they'll read and reply to, and was probably the right thing
to do anyway. 

Now, if I could just work out what bloody charset the ko_KR help files are
actually in at the moment, since iconv claims they're not valid euc-kr...
Comment 10 Fredrik Tolf 2005-09-12 05:24:23 EDT
#8: May I point you to Dolda Webmail (<http://doldawebmail.sf.net/>)? It is not
quite as feature complete or mature as Squirrelmail, but on the other hand, it
is perfectly compliant with UTF-8, and has some other advantages over
Squirrelmail (the most significant of which are listed on the home page).

I know that it is kind of shameless for me to advertise my own project like this
(especially since it doesn't even have an RPM specfile), but I thought it could
still be a good thing since it has precisely that which Squirrelmail seems to be
lacking. Internationalization was something that I kept a high priority while
writing it.
Comment 11 David Woodhouse 2005-09-12 06:13:07 EDT
squirrelmail-1.4.6-0.cvs20050812.2.fc5 has everything converted to UTF-8 (apart
from ko_KR as discussed). Please test.
Comment 12 Nicolas Mailhot 2005-09-13 16:18:44 EDT
I've lost the French localisation with this version and accented letters are
borked in my mail folders (�l�ments envoy�s for example)

I can't test the rest - without locale it's reverted to en_US which is already
UTF-8 compliant
Comment 13 Rahul Sundaram 2005-09-13 16:33:52 EDT
(In reply to comment #10)
> #8: May I point you to Dolda Webmail (<http://doldawebmail.sf.net/>)? It is not
> quite as feature complete or mature as Squirrelmail, but on the other hand, it
> is perfectly compliant with UTF-8, and has some other advantages over
> Squirrelmail (the most significant of which are listed on the home page).
> 
> I know that it is kind of shameless for me to advertise my own project like this
> (especially since it doesn't even have an RPM specfile), but I thought it could
> still be a good thing since it has precisely that which Squirrelmail seems to be
> lacking. Internationalization was something that I kept a high priority while
> writing it.
> 

(Not relevant to the report)

Good to know. Nothing shameful about being proud of your own work ;-). Might
want to take a look at packaging this for Fedora Extras repository

http://fedoraproject.org/wiki/Extras

Then do more evangelisation and let the user choose
Comment 14 David Woodhouse 2005-09-13 17:27:16 EDT
Hm, with squirrelmail-1.4.6-0.cvs20050812.1.fc5.noarch.rpm if I hit shift-reload
on a mail page it alternates between French and English. With the new one, it's
always English. 
Comment 15 David Woodhouse 2005-09-13 17:42:40 EDT
Hm. Confused now. First the previous version started working every time instead
of every other time, then I updated parts of it individually so see when it
stopped working.... and then I updated en masse to the new version and it's
_still_ showing me French every time.
Comment 16 David Woodhouse 2005-09-13 18:08:29 EDT
Definitely seems to be working in the new package now, with utf-8

http://david.woodhou.se/squirrelmail-view.jpeg
http://david.woodhou.se/squirrelmail-reply.jpeg
Comment 17 Nicolas Mailhot 2005-09-14 02:11:58 EDT
Did you try the mock-built FE package or your own one ?

[root@rousalka nim]# rpm -Uvh --force
/var/cache/yum/development/packages/squirrelmail-1.4.6-0.cvs20050812.2.fc5.noarch.rpm
Préparation...              ########################################### [100%]
   1:squirrelmail           ########################################### [100%]
[root@rousalka nim]# exit
exit
[nim@rousalka ~]$ rpm -Vq squirrelmail
S.5....T. c /etc/httpd/conf.d/squirrelmail.conf
S.?....T. c /etc/squirrelmail/config.php
..?...... c /etc/squirrelmail/config_local.php
..?...... c /etc/squirrelmail/default_pref
missing     /var/lib/squirrelmail/prefs/default_pref
[nim@rousalka ~]$ su
Password:
$[root@rousalka nim]# /etc/init.d/httpd restart
Arrêt de httpd :                                           [  OK  ]
Démarrage de httpd :                                       [  OK  ]


It's still not working there -> see the attached screenshot
Comment 18 Nicolas Mailhot 2005-09-14 02:13:24 EDT
Created attachment 118788 [details]
Problem screenshot
Comment 19 David Woodhouse 2005-09-14 10:10:23 EDT
It's not built in mock -- it's a Fedora Core package. I was trying my local
version. However, even with the beehive-built
squirrelmail-1.4.6-0.cvs20050812.1.fc5 I have bizarre problems with fr_FR locale. 

I've changed both my own preferences and /etc/squirrelmail/config.php to
'French' and fr_FR respectively.

If I view a mailbox index and keep clicking 'Previous' (or 
'Précédent' to go to another page, precisely every eighth page is in French; the
other seven are in English. If I middle-click on a mail to show it in a new tab,
then keep hitting shift-reload, then one in every _four_ pages is French. 

Was this not happening for you in squirrelmail-1.4.6-0.cvs20050812.1.fc5? If
not, please could you try moving selected parts of the old package into the new
until it stops working, and let me know? The interesting bits are
/usr/share/squirrelmail/functions/i18n.php,
/usr/share/squirrelmail/locale/fr_FR, and /usr/share/squirrelmail/help/fr_FR
Comment 20 Nicolas Mailhot 2005-09-14 10:19:47 EDT
I never had local switching problems before. I'll try to test what you ask this
evening, if I find the time and the previous SM package.

If I can't do it this evening though it'll have to wait for monday
Comment 21 David Woodhouse 2005-09-14 10:21:01 EDT
It's probably gone from the rawhide mirrors by now.
http://david.woodhou.se/squirrelmail-1.4.6-0.cvs20050812.1.fc5.noarch.rpm
Comment 22 Nicolas Mailhot 2005-09-14 13:38:37 EDT
Well, as soon as I inject the new files I loose the locale.
There is probably some magic to be done there - don't anyone know a sympathetic
SM developper ?
Comment 23 David Woodhouse 2005-09-14 15:45:00 EDT
Which new files? functions/i18n.php, locale/fr_FR/setup.php, the .mo files, the
help files?
Comment 24 Nicolas Mailhot 2005-09-14 16:52:05 EDT
just injecting the squirrelmail-1.4.6-0.cvs20050812.2.fc5.noarch.rpm
functions/i18n.php and locale dir in
squirrelmail-1.4.6-0.cvs20050812.1.fc5.noarch.rpm seems to remove access to the
french locale
Comment 25 David Woodhouse 2005-09-14 16:54:04 EDT
What about _only_ i18n.php? 
What about _only_ locale/fr_FR/setup.php?
What about _only_ the contents of locale/fr_FR/LC_MESSAGES/ ?
Comment 26 Nicolas Mailhot 2005-09-14 17:04:16 EDT
I'll try the various permutations, but not this night and probably not till
monday. I have a guest that's eating much of my free time
Comment 27 Nicolas Mailhot 2005-10-27 06:16:34 EDT
Ok I've found some time to look at this problem.
I've found one thing: during the UTF-8 change this line was added to the spec:
find -name '*.mo' |xargs rm

Turns out SM is using mofiles not pofiles so this line kills all translations
(rebuilt a package with this line commendted out and translations were restored)

If someone is ready to patch SM to use pofiles this line can be removed (do not
forget to fix the local test in configtest) but in the meanwhile it must be kept
Comment 28 Nicolas Mailhot 2005-10-27 06:18:14 EDT
I still have string problems with this change BTW (conversion of folder names
with accented letters) but 80% of the problems are fixed with this simple revert
Comment 29 Nicolas Mailhot 2005-10-27 06:30:19 EDT
Folder handling seems badly b0rked WRT folders with non 7bit ascii names

Just create a folder in a sane client that uses characters in the rest of the
UTF-8 range and SM will go mad when trying to display it. Unfortunately many
languages have some default folders with non ascii-letters, SM got by so far by
assuming ISO 8bit encodings
Comment 31 Nicolas Mailhot 2005-10-27 09:24:22 EDT
Upstream says mbstrings is needed for UTF-8 folders and testing confirms this.

So a full package fix is :
1. do not remove mo files 
2. add a require on php-mbstring

With this SM is a happy camper on my box. (French UTF-8 locale)

-> adding to FC5 tracker
Comment 32 Nicolas Mailhot 2005-10-27 09:26:35 EDT
And upstream adds this warning :

« One more thing. Be careful  in switching Japanese to utf-8.
It is not enough to convert translation files to utf-8. Some
Japanese specific code uses euc-jp and iso-2022-jp charsets. »
Comment 33 Nicolas Mailhot 2005-11-12 13:56:44 EST
Escalading to FC5Blocker since the fix is known
Comment 34 Nicolas Mailhot 2006-01-17 17:16:01 EST
Ok, the problem is fixed now thanks

However i've quickly reverted to my own SM package since it seems the old FC5
snapshot does not like rawhide dovecot very much (I'm using
squirrelmail-1.4.6-0.cvs20051204 today). Some error about imap atoms

Will try to do a proper bug report this week -> bedtime
Comment 35 Nicolas Mailhot 2006-01-17 17:16:56 EST
Ok, the problem is fixed now thanks

However I've quickly reverted to my own SM package since it seems the old FC5
snapshot does not like rawhide dovecot very much (I'm using
squirrelmail-1.4.6-0.cvs20051204 today). Some error about imap atoms

Will try to do a proper bug report this week -> bedtime
Comment 36 Warren Togami 2006-02-28 16:43:22 EST
This is causing a bit of a mess as I attempt to upgrade it to 1.4.6 in order to
gain security fixes.  dwmw2 can you help redo this for 1.4.6 and the new locale
package?

*WHY* has upstream not done this themselves?  pain...

Comment 37 David Woodhouse 2006-02-28 16:49:30 EST
It was just a global search and replace and iconv -- for each locale file,
change it to claim to be UTF-8, and use iconv to make that true. Except for the
Korean ones, iirc, which didn't seem to be in any valid character set that iconv
liked.

I'll take a look -- what's the problem?
Comment 38 Nicolas Mailhot 2006-02-28 17:44:56 EST
The problem is upstream is slowly converting to utf-8 one at a time so you have
to re-do the locale patches every time

Warren, I have a more current SRPM if you want (cvs20060118)
I needed it since the last openssl changes broke imaps with older squirrelmail
versions (I know I should have reporter it way back)
Comment 39 Nicolas Mailhot 2006-02-28 17:51:29 EST
Created attachment 125428 [details]
In case this one still applies...
Comment 40 Nicolas Mailhot 2006-02-28 17:53:59 EST
(Upstream considers each locale maintainer is free to choose his preferred
encoding, that's why it's a big changing mess. Fortunately some sanity seems to
begin to prevail and locales are being converted to UTF-8)
Comment 41 David Woodhouse 2006-02-28 18:05:51 EST
(In reply to comment #38)
> The problem is upstream is slowly converting to utf-8 one at a time so you have
> to re-do the locale patches every time

Ah. In that case we should probably just script that change too instead of using
a patch for it?
Comment 42 Warren Togami 2006-02-28 23:06:47 EST
If you know exactly how to do it, then please help me on this.  I'm wanting to
just dump squirrelmail entirely.

And yes, scripting the change (excluding Korean) is ideal instead of a patch
that will repeatedly break with upstream changes.
Comment 43 David Woodhouse 2006-03-01 05:11:55 EST
OK, I just checked something in -- take a look and see if you like it.
Comment 44 Warren Togami 2006-03-02 17:58:49 EST
Seems good in English and Japanese.  It will be in FC5 tomorrow, and will
probably push FC4 update tomorrow too.
Comment 45 Nicolas Mailhot 2006-03-03 16:36:57 EST
I'm afraid this release (1.4.6-1.fc5) is very broken.
A good test procedure (the one I use) is the following :

1. install dejavu from FE to get good cyrillic & greek coverage
2. configure the gnome keyboard switching applet to have an alternate greek or
russian keyboard (additionnaly people are currently trying to nail down the last
boogs there - more testers will help)
3. go into sm, send yourself a mail with cyrillic or greek glyphs in the title
and text
4. check sm displays it all right (you don't need to understand the text, just
recognize what you typed)
5. use another MUA to check it agrees with SM (at least evo and firefox) <-
crucial step, sm often does double mistakes wich result in gibberish only the
broken version can read

6. get an IMAP account somewhere
7. try to create a folder with greek or cyrillic letters in it
8. check the result in SM (refresh)
9. check the result in SM and thunderbird

Your release fails 5. and 9. My old jan 18 2006 cvs snapshot build OTOH succeeds
- I'll try now to isolate the difference 
Comment 46 David Woodhouse 2006-03-03 16:54:26 EST
Thanks for the testing. Could you describe the failure? What was wrong with the
mail in #5? Was it actually valid UTF-8 with an invalid charset header, or was
something else wrong? Can you send me an example? Did your web browser actually
operate in UTF-8 mode while you were doing this? Replying to a known-good mail
containing non-ASCII characters, so that those characters get quoted, is also a
good test.

What was the name of the directory you tried to create, and what _actually_ got
created (the horrid IMAP UTF-7 version)?
Comment 47 Nicolas Mailhot 2006-03-03 17:09:53 EST
I've already identified one problem in the spec file: function/i18n.php is
munged *after* a copy has already been "installed", so the munging has no effect

This basic error is made possible by the fact the spec does not cleanly separate
file processing from file installation, and mixes all in %install. Moving
processing to a %build fwould certainly bring some sanity there

I'll now test a rebuilt rpm to see if it was the core problem
Comment 48 David Woodhouse 2006-03-03 17:28:18 EST
OK, I just checked in http://david.woodhou.se/squirrelmail.spec

Better?
Comment 49 Nicolas Mailhot 2006-03-03 17:48:21 EST
looks better, I'll test it

With the previous spec I can confirm the ordering of the i18n.php sed-ing was
the problem (on my box, with my test procedures)
Comment 50 Nicolas Mailhot 2006-03-03 18:01:45 EST
Ok, your spec works for me
Not that it proves a lot of things
But at least it's not broken the same way as the previous FC SM packages

Note You need to log in before you can comment on or make changes to this bug.