Bug 744277

I assume this string comes from as.po in subscriptionmanager?

BTW, there seems to be a stray quote char on the second line of that diff
(after \n).

Splitting by line should be fine; if you run the original PO through Gettext's
msgcat it will do the same thing.  

What is the tool/library which consumes the PO file?  msgfmt?

We are using this command in the make file:

compile-po:
        for lang in $(basename $(notdir $(wildcard po/*.po))) ; do \
                echo $$lang ; \
                mkdir -p po/build/$$lang/LC_MESSAGES/ ; \
                msgfmt -c --statistics -o po/build/$$lang/LC_MESSAGES/rhsm.mo po/$$lang.po ; \
        done


We have fixd the bug in as.po

-- bk

I can't get msgfmt to complain about as.po, either with the version from Zanata as of yesterday, or today.

What exactly was the error from msgfmt?  Do you have a copy of the file from Zanata which triggered the error?

The issue we say was the the string split on multiple lines
msgid ""
"\n"
"Usage: %s [options] MODULENAME --help\n"
msgstr ""
"\n"
"\"ব্যৱহাৰ: %s [বিকল্পসমূহ] MODULENAME --help\n"

Was no picked up. We had to collpase the strings:

msgid "\nUsage: %s [options] MODULENAME --help\n"
msgstr "\n"ব্যৱহাৰ: %s [বিকল্পসমূহ] MODULENAME --help\n"

The strings are written as "\n"ব্যৱহাৰ: %s [বিকল্পসমূহ] MODULENAME --help\n" in the source file. It is the import/export from zanata which breaks it onto many lines.

Okay, I've run the original as.po and as.po as pulled from zanata through msgfmt then msgunfmt, and got this diff:

$ msgfmt -c --statistics as.po -o as.mo
420 translated messages.                                     
$ msgfmt -c --statistics as-pulled.po -o as-pulled.mo
420 translated messages.  
$ msgunfmt as.mo > as.mo.po
$ msgunfmt as-pulled.mo > as-pulled.mo.po
$ diff -U5 as.mo.po as-pulled.mo.po
--- as.mo.po    2011-10-14 11:34:14.411349895 +1000
+++ as-pulled.mo.po     2011-10-14 11:34:22.510224846 +1000
@@ -104,11 +104,11 @@
 msgid ""
 "\n"
 "Usage: %s [options] MODULENAME --help\n"
 msgstr ""
 "\n"
-"বযৱহাৰ: %s [বিকলপসমহ] MODULENAME --help\n"
+"\"বযৱহাৰ: %s [বিকলপসমহ] MODULENAME --help\n"

 msgid "    Entitled Repositories in %s"
 msgstr "%s -ত অনজঞা থকা ভৰালসমহ"

 msgid "    Installed Product Status"


We're investigating where the extra quote came from (a translator, or Zanata's import process).

Going by http://git.fedorahosted.org/git/?p=subscription-manager.git;a=blobdiff;f=po/as.po;h=7bb789bbe43a43a78f66bfbcde22cf9a8b9957b8;hp=34a810df2139bae091190483c18d0e906dd3d10d;hb=37a269921d46b74db0f2b32bc739647aa53263d1;hpb=c900ff31d099801fdb832801e85730c3a49a88c8

and by the database history, I think as.po was imported with the stray (unescaped) quote.  The python client *should* have rejected the entire file as illegal, but instead it treated the stray quote as a literal quote.  

I've created a test PO file so that we can fix the import bug, but in the meantime I've removed the extra quote from that string in Zanata.  You should be able to pull the document now.

NB: The fact that your runtime PO library treated it as an untranslated string may be a bug in that library, because it still had an entry for that msgid, even though it was a bad one.

Created attachment 528142 [details]
Test project containing illegal stray quote in as.po

The file as.po should be rejected as illegal, not imported.

Hi

I have create a patch for polib and submit it to upstream, please see the link below:
https://bitbucket.org/izi/polib/issue/27/polib-doesnt-check-unescaped-quote

I also suggest that use "msgfmt --check as.po" before pushing to zanata server with python-client, since python-client doesn't have any syntax validation or escape semantic check, the python-polib doesn't provide escape semantic check also. I have test 'msgfmt --check' with Sean's illegal po file, it shows following message:

$ msgfmt --check as.po
as.po:25:13: syntax error
as.po:25: keyword "s" unknown
as.po:26: end-of-line within string
msgfmt: found 3 fatal errors
   
So msgfmt could detect the syntax error, I think this could be a workaround for this issue.

Regarding the newlines:

From http://www.gnu.org/software/gettext/manual/html_node/PO-Files.html#PO-Files

"One should carefully distinguish between end of lines marked as ‘\n’ inside quotes, which are part of the represented string, and end of lines in the PO file itself, outside string quotes, which have no incidence on the represented string."

Thus if you run Gettext msgfmt on a PO file like this:
  msgid "\nUsage: %s [options] MODULENAME --help\n"
  msgstr "\nবযৱহাৰ: %s [বিকলপসমহ] MODULENAME --help\n"

or on a PO file like this:
  msgid ""
  "\n"
  "Usage: %s [options] MODULENAME --help\n"
  msgstr ""
  "\n"
  "বযৱহাৰ: %s [বিকলপসমহ] MODULENAME --help\n"

then the MO file produced by msgfmt is identical.  Zanata only adds insignificant newlines outside the quotes (just like Gettext msgcat).  


Naturally, we still need to fix the Zanata import (or polib) so that it rejects illegal PO files.  Running "msgfmt --check" _before_ zanata import should help as a workaround.

Hi

The author of polib have modified source code to fix unescaped quotes issue i reported, please look at commit 8ee09b305e8e at https://bitbucket.org/izi/polib/changesets. And he also accept my pull request for adding a check for quotes at the beginning of the strings, please look at commit dbafdc621bf4 at https://bitbucket.org/izi/polib/changesets

We will try to package 0.7.0 version polib on RHEL and Fedora, and include above commit as a patch.

Should I also include the commits between 0.7.0 and 8eee09, or just commit 8ee09b305e8e and dbafdc621bf4?

All the intervening commits look good, and there's another bugfix which might be good to have.  I'd say take them all.

python-polib-0.7.0-2 is push to stable for f16, f15, el6 and el5.

James, I an not quite sure whether python-polib provides the error information.
That is, it returns "Error: Stray quotes detected in <filename>" or "Error: invalid format: <filename>"

As in version 1.3.3-25-g7cff, the error message returns:
"error: Cannot process the po file" 
even if .pot file contains stray quotes.

Using the error message I provide should be more informative.

Hi Ding,

Thanks, i have modified source code to output the error message from python-polib, if unescaped quote is detected. It should give more informative info to users. Please check the latest code on git-hub for verify.

VERIFIED with 1.3.4-3-gfc46