Bug 162852

Closing this as this is work required upstream and not Fedora's sole responsibility.

Do you have a reference to the squirrelmail bug (apparently #1235345)? I don't
seem to be able to find it.

*** Bug 167877 has been marked as a duplicate of this bug. ***

Fedora is the party that cares about software being UTF-8 compatible.
Not handling UTF-8 well was sufficient ground for removal in the past.

Unless this Red Hat policy changed (which I doubt) if no one steps up to do the
UTF-8 work, suirrelmail should leave FC IMHO.

Which would be a pity as I use and like it.

https://sourceforge.net/tracker/?func=detail&atid=423691&aid=1235345&group_id=311

> Unless this Red Hat policy changed (which I doubt) if no one steps up to do 
> the UTF-8 work, squirrelmail should leave FC IMHO.

Agreed.

> Which would be a pity as I use and like it.

I know. =(

From http://www.squirrelmail.org/wiki/en_US/WishList, where it is stated:

"From 1.5.1cvs and 1.4.4cvs squirrelmail allows setting default charset when default language is en_US"

Sounds like there should be a solution, though I still have to figure out how it is actually done, just saw it 
and I dont have time right now.

I surely would miss squirrelmail if it went away from FC. It understandable if it is removed from a Redhat 
product on the other hand.

It is sure a pity because I used squirrelmail for years, but it would probably
move to Extras if it does become removed.  If you are aware of a standards
compliant replacement that is secure and proven mature then please chime in here.

Shouldn't be too hard just to convert all the locales to UTF-8 while we're
building the package. That ensures that their charset is a superset of the
charset in any mail they'll read and reply to, and was probably the right thing
to do anyway. 

Now, if I could just work out what bloody charset the ko_KR help files are
actually in at the moment, since iconv claims they're not valid euc-kr...

#8: May I point you to Dolda Webmail (<http://doldawebmail.sf.net/>)? It is not
quite as feature complete or mature as Squirrelmail, but on the other hand, it
is perfectly compliant with UTF-8, and has some other advantages over
Squirrelmail (the most significant of which are listed on the home page).

I know that it is kind of shameless for me to advertise my own project like this
(especially since it doesn't even have an RPM specfile), but I thought it could
still be a good thing since it has precisely that which Squirrelmail seems to be
lacking. Internationalization was something that I kept a high priority while
writing it.

squirrelmail-1.4.6-0.cvs20050812.2.fc5 has everything converted to UTF-8 (apart
from ko_KR as discussed). Please test.

I've lost the French localisation with this version and accented letters are
borked in my mail folders (�l�ments envoy�s for example)

I can't test the rest - without locale it's reverted to en_US which is already
UTF-8 compliant

(In reply to comment #10)
> #8: May I point you to Dolda Webmail (<http://doldawebmail.sf.net/>)? It is not
> quite as feature complete or mature as Squirrelmail, but on the other hand, it
> is perfectly compliant with UTF-8, and has some other advantages over
> Squirrelmail (the most significant of which are listed on the home page).
> 
> I know that it is kind of shameless for me to advertise my own project like this
> (especially since it doesn't even have an RPM specfile), but I thought it could
> still be a good thing since it has precisely that which Squirrelmail seems to be
> lacking. Internationalization was something that I kept a high priority while
> writing it.
> 

(Not relevant to the report)

Good to know. Nothing shameful about being proud of your own work ;-). Might
want to take a look at packaging this for Fedora Extras repository

http://fedoraproject.org/wiki/Extras

Then do more evangelisation and let the user choose

Hm, with squirrelmail-1.4.6-0.cvs20050812.1.fc5.noarch.rpm if I hit shift-reload
on a mail page it alternates between French and English. With the new one, it's
always English. 

Hm. Confused now. First the previous version started working every time instead
of every other time, then I updated parts of it individually so see when it
stopped working.... and then I updated en masse to the new version and it's
_still_ showing me French every time.

Definitely seems to be working in the new package now, with utf-8

http://david.woodhou.se/squirrelmail-view.jpeg
http://david.woodhou.se/squirrelmail-reply.jpeg

Did you try the mock-built FE package or your own one ?

[root@rousalka nim]# rpm -Uvh --force
/var/cache/yum/development/packages/squirrelmail-1.4.6-0.cvs20050812.2.fc5.noarch.rpm
Préparation...              ########################################### [100%]
   1:squirrelmail           ########################################### [100%]
[root@rousalka nim]# exit
exit
[nim@rousalka ~]$ rpm -Vq squirrelmail
S.5....T. c /etc/httpd/conf.d/squirrelmail.conf
S.?....T. c /etc/squirrelmail/config.php
..?...... c /etc/squirrelmail/config_local.php
..?...... c /etc/squirrelmail/default_pref
missing     /var/lib/squirrelmail/prefs/default_pref
[nim@rousalka ~]$ su
Password:
$[root@rousalka nim]# /etc/init.d/httpd restart
Arrêt de httpd :                                           [  OK  ]
Démarrage de httpd :                                       [  OK  ]


It's still not working there -> see the attached screenshot

Created attachment 118788 [details]
Problem screenshot

It's not built in mock -- it's a Fedora Core package. I was trying my local
version. However, even with the beehive-built
squirrelmail-1.4.6-0.cvs20050812.1.fc5 I have bizarre problems with fr_FR locale. 

I've changed both my own preferences and /etc/squirrelmail/config.php to
'French' and fr_FR respectively.

If I view a mailbox index and keep clicking 'Previous' (or 
'Précédent' to go to another page, precisely every eighth page is in French; the
other seven are in English. If I middle-click on a mail to show it in a new tab,
then keep hitting shift-reload, then one in every _four_ pages is French. 

Was this not happening for you in squirrelmail-1.4.6-0.cvs20050812.1.fc5? If
not, please could you try moving selected parts of the old package into the new
until it stops working, and let me know? The interesting bits are
/usr/share/squirrelmail/functions/i18n.php,
/usr/share/squirrelmail/locale/fr_FR, and /usr/share/squirrelmail/help/fr_FR

I never had local switching problems before. I'll try to test what you ask this
evening, if I find the time and the previous SM package.

If I can't do it this evening though it'll have to wait for monday

It's probably gone from the rawhide mirrors by now.
http://david.woodhou.se/squirrelmail-1.4.6-0.cvs20050812.1.fc5.noarch.rpm

Well, as soon as I inject the new files I loose the locale.
There is probably some magic to be done there - don't anyone know a sympathetic
SM developper ?

Which new files? functions/i18n.php, locale/fr_FR/setup.php, the .mo files, the
help files?

just injecting the squirrelmail-1.4.6-0.cvs20050812.2.fc5.noarch.rpm
functions/i18n.php and locale dir in
squirrelmail-1.4.6-0.cvs20050812.1.fc5.noarch.rpm seems to remove access to the
french locale

What about _only_ i18n.php? 
What about _only_ locale/fr_FR/setup.php?
What about _only_ the contents of locale/fr_FR/LC_MESSAGES/ ?

I'll try the various permutations, but not this night and probably not till
monday. I have a guest that's eating much of my free time

Ok I've found some time to look at this problem.
I've found one thing: during the UTF-8 change this line was added to the spec:
find -name '*.mo' |xargs rm

Turns out SM is using mofiles not pofiles so this line kills all translations
(rebuilt a package with this line commendted out and translations were restored)

If someone is ready to patch SM to use pofiles this line can be removed (do not
forget to fix the local test in configtest) but in the meanwhile it must be kept

I still have string problems with this change BTW (conversion of folder names
with accented letters) but 80% of the problems are fixed with this simple revert

Folder handling seems badly b0rked WRT folders with non 7bit ascii names

Just create a folder in a sane client that uses characters in the rest of the
UTF-8 range and SM will go mad when trying to display it. Unfortunately many
languages have some default folders with non ascii-letters, SM got by so far by
assuming ISO 8bit encodings

Reported upstream as bug 1339393
https://sourceforge.net/tracker/index.php?func=detail&aid=1339393&group_id=311&atid=100311

Upstream says mbstrings is needed for UTF-8 folders and testing confirms this.

So a full package fix is :
1. do not remove mo files 
2. add a require on php-mbstring

With this SM is a happy camper on my box. (French UTF-8 locale)

-> adding to FC5 tracker

And upstream adds this warning :

« One more thing. Be careful  in switching Japanese to utf-8.
It is not enough to convert translation files to utf-8. Some
Japanese specific code uses euc-jp and iso-2022-jp charsets. »

Escalading to FC5Blocker since the fix is known

Ok, the problem is fixed now thanks

However i've quickly reverted to my own SM package since it seems the old FC5
snapshot does not like rawhide dovecot very much (I'm using
squirrelmail-1.4.6-0.cvs20051204 today). Some error about imap atoms

Will try to do a proper bug report this week -> bedtime

Ok, the problem is fixed now thanks

However I've quickly reverted to my own SM package since it seems the old FC5
snapshot does not like rawhide dovecot very much (I'm using
squirrelmail-1.4.6-0.cvs20051204 today). Some error about imap atoms

Will try to do a proper bug report this week -> bedtime

This is causing a bit of a mess as I attempt to upgrade it to 1.4.6 in order to
gain security fixes.  dwmw2 can you help redo this for 1.4.6 and the new locale
package?

*WHY* has upstream not done this themselves?  pain...

It was just a global search and replace and iconv -- for each locale file,
change it to claim to be UTF-8, and use iconv to make that true. Except for the
Korean ones, iirc, which didn't seem to be in any valid character set that iconv
liked.

I'll take a look -- what's the problem?

The problem is upstream is slowly converting to utf-8 one at a time so you have
to re-do the locale patches every time

Warren, I have a more current SRPM if you want (cvs20060118)
I needed it since the last openssl changes broke imaps with older squirrelmail
versions (I know I should have reporter it way back)

Created attachment 125428 [details]
In case this one still applies...

(Upstream considers each locale maintainer is free to choose his preferred
encoding, that's why it's a big changing mess. Fortunately some sanity seems to
begin to prevail and locales are being converted to UTF-8)

(In reply to comment #38)
> The problem is upstream is slowly converting to utf-8 one at a time so you have
> to re-do the locale patches every time

Ah. In that case we should probably just script that change too instead of using
a patch for it?

If you know exactly how to do it, then please help me on this.  I'm wanting to
just dump squirrelmail entirely.

And yes, scripting the change (excluding Korean) is ideal instead of a patch
that will repeatedly break with upstream changes.

OK, I just checked something in -- take a look and see if you like it.

Seems good in English and Japanese.  It will be in FC5 tomorrow, and will
probably push FC4 update tomorrow too.

I'm afraid this release (1.4.6-1.fc5) is very broken.
A good test procedure (the one I use) is the following :

1. install dejavu from FE to get good cyrillic & greek coverage
2. configure the gnome keyboard switching applet to have an alternate greek or
russian keyboard (additionnaly people are currently trying to nail down the last
boogs there - more testers will help)
3. go into sm, send yourself a mail with cyrillic or greek glyphs in the title
and text
4. check sm displays it all right (you don't need to understand the text, just
recognize what you typed)
5. use another MUA to check it agrees with SM (at least evo and firefox) <-
crucial step, sm often does double mistakes wich result in gibberish only the
broken version can read

6. get an IMAP account somewhere
7. try to create a folder with greek or cyrillic letters in it
8. check the result in SM (refresh)
9. check the result in SM and thunderbird

Your release fails 5. and 9. My old jan 18 2006 cvs snapshot build OTOH succeeds
- I'll try now to isolate the difference 

Thanks for the testing. Could you describe the failure? What was wrong with the
mail in #5? Was it actually valid UTF-8 with an invalid charset header, or was
something else wrong? Can you send me an example? Did your web browser actually
operate in UTF-8 mode while you were doing this? Replying to a known-good mail
containing non-ASCII characters, so that those characters get quoted, is also a
good test.

What was the name of the directory you tried to create, and what _actually_ got
created (the horrid IMAP UTF-7 version)?

I've already identified one problem in the spec file: function/i18n.php is
munged *after* a copy has already been "installed", so the munging has no effect

This basic error is made possible by the fact the spec does not cleanly separate
file processing from file installation, and mixes all in %install. Moving
processing to a %build fwould certainly bring some sanity there

I'll now test a rebuilt rpm to see if it was the core problem

OK, I just checked in http://david.woodhou.se/squirrelmail.spec

Better?

looks better, I'll test it

With the previous spec I can confirm the ordering of the i18n.php sed-ing was
the problem (on my box, with my test procedures)

Ok, your spec works for me
Not that it proves a lot of things
But at least it's not broken the same way as the previous FC SM packages