Bug 303991 - Transifex can't handle non-UTF or invalid UTF files
Summary: Transifex can't handle non-UTF or invalid UTF files
Alias: None
Product: Fedora Localization
Classification: Fedora
Component: Website
Version: unspecified
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: Dimitris Glezos
QA Contact: Diego Búrigo Zacarão
: 430998 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2007-09-24 20:39 UTC by Fabian Affolter
Modified: 2009-07-03 20:07 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2008-03-11 20:54:51 UTC

Attachments (Terms of Use)
Traceback (3.19 KB, application/octet-stream)
2007-09-24 20:39 UTC, Fabian Affolter
no flags Details

Description Fabian Affolter 2007-09-24 20:39:56 UTC
Description of problem:
Clicking 'preview' gets the following error.
"500 Internal error" (more details, see attachment)

Version-Release number of selected component (if applicable):

How reproducible:
I did a couple of time, got same error message.

Steps to Reproduce:
1. go to http://publictest5.fedora.redhat.com/submit/module/ and choose revisor
2a. At 'select local file', select ~/revisor/de.po
2b. At 'select local file', select ~/revisor/de.po
3. At 'select a destination file', select de.po from dropdown menu of
'Overwrite existing file'
4. At 'write a commit message', enter 'updated' (or something else)
5. Click 'preview'
6. Check the preview and click 'submit'
7. '500 Internal error'
Actual results:
At step 7, the submission to be successful

Expected results:
Successful submission

Comment 1 Fabian Affolter 2007-09-24 20:39:56 UTC
Created attachment 204551 [details]

Comment 2 Dimitris Glezos 2007-09-24 21:27:39 UTC
I think this is because de.po on the server is not a valid UTF-8 file. Can you
confirm this by testing it with something like `iconv -f utf-8 -t utf-8 de.po`?

Nevertheless, transifex shouldn't choke with this and should commit the file
without producing diff and patches.

Comment 3 Dimitris Glezos 2008-01-25 08:24:13 UTC
A minor update: I tried looking into how we could fix this, but it requires more
than just a small change in the code. We'll need to test thoroughly before
putting in production.

I'd suggest to open a bug report on smolt, so that one of the developers
converts the file to UTF, or delete it completely, so that you can upload it for

Comment 4 Fabian Affolter 2008-01-29 09:06:10 UTC
The submitting through transifex works now. It's no longer a bug...

Comment 5 Dimitris Glezos 2008-01-29 10:05:03 UTC
I think the submission was successful because the file was fixed to be UTF:


Still, Transifex shouldn't panic when the remote file isn't UTF-encoded. We
should be able to choose "overwrite" on the submission form, which will
completely replace the remote file.

Re-opening with a changed subject to remember fixing it.

Comment 6 Dimitris Glezos 2008-02-06 23:19:05 UTC
*** Bug 430998 has been marked as a duplicate of this bug. ***

Comment 7 Dimitris Glezos 2008-02-06 23:25:01 UTC
So, the thing is, we currently only support valid UTF-encoded files. Not even
ISO-8859-1. We should fix this. In bug #430998, I think the files on the server
were ISO-8859-1 or something non-UTF, so Tx panicked.

To fix this, we'll need to open up both files with the correct encoding, which
is *always* the one mentioned in the PO-file's header. If the uploaded file is
invalid (ie the file's encoding doesn't much with the header entry and fopen
fails), raise an error. If the server file is invalid, offer the choice to the
user to overwrite it.

Comment 8 Francesco Tombolini 2008-02-07 18:18:54 UTC
Workaround: If the upload fails with "500 Internal error", the file can be
uploaded successfully, if the user write the filename by hand in the field, like
if he want upload a new file, and not by selecting the filename from the drop
down box, and skipping the preview.
I have tested it on firstboot it.po.
My case was ISO8859 stored on server and UTF-8 to be uploaded.

Comment 9 Piotr Drąg 2008-02-07 18:36:10 UTC
(In reply to comment #8)

Doesn't work for me, still shows error. New file and the one stored on VCS are
UTF-8. Maybe it's not the same issue?

Comment 10 Francesco Tombolini 2008-02-07 19:24:27 UTC
(In reply to comment #9)
Yes, you are right.
My solution was a bug, the file uploaded was appended to the existing one and
should be fixed.

Comment 11 Dimitris Glezos 2008-02-11 12:34:32 UTC
An update was pushed yesterday (v0.2) that allows submission of non-UTF files,
and also overwriting of remote non-UTF files. It should go live sometime today,
I'll send an email to the list as well.

The only caveat is that preview isn't available for such submissions. To fix
this as well, we'll have to support encoding detection: open the file as UTF,
read up to the "Content-type" line, and open the file with the correct encoding.

Comment 12 Piotr Drąg 2008-03-11 18:53:24 UTC
So this bug is FIXED now?

Comment 13 Dimitris Glezos 2008-03-11 20:54:51 UTC
Yes, this is fixed. Closing bug report, thanks for the feedback.

Comment 14 Piotr Drąg 2009-07-03 20:07:17 UTC
Mass change of component from Transifex to Website, since all our websites are handled by Transifex now.

Note You need to log in before you can comment on or make changes to this bug.