Bug 870876 - PO download for non-PO projects cannot be uploaded
PO download for non-PO projects cannot be uploaded
Status: CLOSED CURRENTRELEASE
Product: Zanata
Classification: Community
Component: Component-Persistence (Show other bugs)
development
Unspecified Unspecified
medium Severity high
: ---
: 2.3
Assigned To: David Mason
Alex Eng
:
: 870873 873933 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-29 01:24 EDT by Ding-Yi Chen
Modified: 2013-11-26 22:36 EST (History)
6 users (show)

See Also:
Fixed In Version: 2.3-SNAPSHOT (20130418-1355)
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-26 22:25:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Ding-Yi Chen 2012-10-29 01:24:44 EDT
Description of problem:
Zanata provides "Download as po", however, if translators do download the translation this way and do the off-line translation, they won't be able to push it back if the project does not use po files.

A translator should be able push po files for non-po file projects.


Version-Release number of selected component (if applicable):
Zanata version 2.0.1-SNAPSHOT (20121029-1430)

How reproducible:
Always

Steps to Reproduce:
1. Push a non-po file project, such as a properties, utf8properties or xliff projects. 
2. Download a document as a po file.
3. Upload that po file
  
Actual results:
File Constants.po uploaded. There were some warnings, see below. 
 Could not find TextFlow for TextFlowTarget 2fd5df9452801bcc015367756752a895 with contents: [] 
  .... and so on ...

Expected results:
Push successfully.

Additional info:
Comment 1 sandeep shedmake 2012-10-29 02:25:54 EDT
Similar warning messages could be observed while pulling the non-po file project.

$ mvn zanata:pull -Dzanata.pullType=both -Dzanata.locales=XX
...
...
[WARNING] Missing POT entry for text-flow ID textFlow_getId
Comment 2 Sean Flanigan 2012-10-29 03:29:35 EDT
This happens because the PO upload (and REST clients) hash msgstr+msgctxt to a textflow resId in a way which does not correspond to the resIds we use for non-PO projects.  Non-PO projects typically use an identifier such as trans-unit id or property key directly as a textflow resId.  Thus when the PO file is uploaded to the server, its generated resId does not match the textflow resId.
Comment 5 Sean Flanigan 2012-11-07 00:59:32 EST
*** Bug 873933 has been marked as a duplicate of this bug. ***
Comment 6 Sean Flanigan 2012-11-07 01:01:51 EST
I have some implementation ideas recorded against bug 873933.
Comment 7 David Mason 2012-11-13 02:28:20 EST
*** Bug 870873 has been marked as a duplicate of this bug. ***
Comment 8 David Mason 2013-03-12 01:54:10 EDT
Updated Java clients to accept project type 'offlinepo'. This type is exclusively for offline translation working with po files generated from non-po project types and is not designed for normal project maintenance. This is a client-only project type, so it cannot be selected on the server.

Client behaviour when using project type 'offlinepo'
 - pull generates reversible po and pot files with Zanata id in msgctxt
 - push uses msgctxt for id in Zanata
 - as long as msgctxt is not modified, translations should always be correctly associated with source in Zanata when they are pushed.

Warning: offlinepo should not be used for gettext or podir projects as they may already have msgctxt, which would cause an error (this is intentional to prevent accidental overwriting of msgctxt)

see: https://github.com/zanata/zanata-client/commit/df16d22dd184f9db27d1ef3cbe9ed6eb183e3dc6
Comment 9 David Mason 2013-03-12 01:57:44 EDT
Updated server to use msgctxt<->Zanata-id mapping in downloaded and uploaded po/pot files at appropriate times:

 - uploads to gettext and podir projects behave exactly as before (do not use msgctxt mappings)
 - downloads from gettext and podir projects behave as before

 - uploads of pot files to 'file' projects (which may contain pot files):
     - if it is a new file, does not use mapping
     - if it is overwriting an existing pot file, does not use mapping
     - if it is overwriting a non-pot file, it will use the new mapping
 - download of pot files from 'file' projects
     - if the original is a pot file, does not use mapping
     - if the original is any other type, mapping is used

 - uploads of pot files to all other project types will use the mapping
     - since pot files are not allowed in these projects, it is assumed that pot files uploaded to these must have been generated from non-po files
 - downloads of po/pot files from all other project types will use the mapping

see: https://github.com/zanata/zanata/compare/1dbaf6e6234578fdfcc2316f945de51048656119...cc57236f73bb81e930aab584ba63ea1f76aa0a70
Comment 10 David Mason 2013-03-12 04:03:54 EDT
Updated server to allow zip download for non-po projects using msgctxt<->Zanata-id mapping, and bundled with a config file with project type set to 'offlinepo'.

There are 3 possible behaviours:

1. No project type is configured for the iteration: will show the normal zip download button, disabled with a tooltip about maintainer setting a project type.
2. Project type set to gettext or podir: normal zip download button is shown and enabled, downloaded zip has a normal config file and po files without msgctxt mapping.
3. Project type is set, and is not gettext or podir: "offline po" zip download button is shown and enabled, downloaded zip has config file with project-type set to 'offlinepo' and contains po files with msgctxt mapping.


At this stage, downloads of plain config files (not in zip) always have the configured project type (or a comment with valid types if no type is configured).

See: https://github.com/zanata/zanata/commit/b7cfba35139631a22afdede4f563bde8c71b64d1
Comment 11 David Mason 2013-03-13 22:34:47 EDT
Updated server to provide an additional config file with project type 'offlinepo' for non-po projects. Config file link now opens a modal dialog that has links for the default config file and special offline-translation config file.

See: https://github.com/zanata/zanata/commit/7a3ff69e5df5262d5d5bcbc89c1a2b4670b06bff


This and the above changes should cover all standard ways to retrieve and submit po files for offline translation.
Comment 12 Ding-Yi Chen 2013-03-14 03:19:58 EDT
For users that act as both project maintainers and translators (e.g. input methods) they can either put zanata.xml in different directories, or use different filenames and use --projectConfig to assign it.
Comment 13 David Mason 2013-03-14 03:31:44 EDT
(In reply to comment #12)
> For users that act as both project maintainers and translators (e.g. input
> methods) they can either put zanata.xml in different directories, or use
> different filenames and use --projectConfig to assign it.

I would strongly advise only the first option - using completely separate directories.

The default config file will pull and push files in their original format, whereas the offline translation config file will pull and push files as offlinepo. The files are incompatible, so there is no use case for using the different config files in the same zanata directory.

In other words, files for offline translation (config file and pulled files) should never be mixed with files for normal 'project maintainer' activities.
Comment 14 Ding-Yi Chen 2013-03-14 22:46:36 EDT
Tested with Zanata version 2.3-SNAPSHOT (20130314-1234),
Maven client 2.2.1-SNAPSHOTS

1. Download all files (zip) in document list does not work for non-po based projects.
   Click Projects->PROJECTNAME->VERSION->LOCALE 
   and on the upper right, the button (Donload all files (zip) does not work.

2, Project maintainers may accidently wipe out the project
   Step to reproduce:
   a. Download and extract the zip file that contains po
   b. push with maven client
      mvn zanata:push

   Actual:
   Project documents are wiped out

   Expected:
   1) Error messages states that no pot exists.
   or 2) Update po only, as if -Dzanata.pushType=trans are given.

In other words, the push should be fool proof.
Comment 15 David Mason 2013-03-14 22:54:37 EDT
(In reply to comment #14)
> Tested with Zanata version 2.3-SNAPSHOT (20130314-1234),
> Maven client 2.2.1-SNAPSHOTS
> 
> 1. Download all files (zip) in document list does not work for non-po based
> projects.
>    Click Projects->PROJECTNAME->VERSION->LOCALE 
>    and on the upper right, the button (Donload all files (zip) does not work.

Is the project type configured for that project or version?

(from comment #10)
[if] No project type is configured for the iteration: will show the normal zip download button, disabled with a tooltip about maintainer setting a project type.


> 2, Project maintainers may accidently wipe out the project
>    Step to reproduce:
>    a. Download and extract the zip file that contains po
>    b. push with maven client
>       mvn zanata:push
> 
>    Actual:
>    Project documents are wiped out
> 
>    Expected:
>    1) Error messages states that no pot exists.
>    or 2) Update po only, as if -Dzanata.pushType=trans are given.
> 
> In other words, the push should be fool proof.

Does this occur only with the "offline po" zip file, or can project maintainers wipe out a project with the zip file from a gettext or podir project as well?
Comment 16 Ding-Yi Chen 2013-03-15 04:09:49 EDT
Yes, it is, however, we have 2 ways to download po file(In reply to comment #15)
> (In reply to comment #14)
> > Tested with Zanata version 2.3-SNAPSHOT (20130314-1234),
> > Maven client 2.2.1-SNAPSHOTS
> > 
> > 1. Download all files (zip) in document list does not work for non-po based
> > projects.
> >    Click Projects->PROJECTNAME->VERSION->LOCALE 
> >    and on the upper right, the button (Donload all files (zip) does not work.
> 
> Is the project type configured for that project or version?
>
> (from comment #10)
> [if] No project type is configured for the iteration: will show the normal
> zip download button, disabled with a tooltip about maintainer setting a
> project type.
 
Yes, it it configured as properties.
However, we do have 2 ways to download ZIP.
One is Projects->PROJECTNAME->VERSION-> LOCALE Row Documents Col -> Action Donload All Files. That works.

The other one on the document list from workspace, on the upper right of the page. That one does not work.

 
> > 2, Project maintainers may accidently wipe out the project
> >    Step to reproduce:
> >    a. Download and extract the zip file that contains po
> >    b. push with maven client
> >       mvn zanata:push
> > 
> >    Actual:
> >    Project documents are wiped out
> > 
> >    Expected:
> >    1) Error messages states that no pot exists.
> >    or 2) Update po only, as if -Dzanata.pushType=trans are given.
> > 
> > In other words, the push should be fool proof.
> 
> Does this occur only with the "offline po" zip file, or can project
> maintainers wipe out a project with the zip file from a gettext or podir
> project as well?

Yes, they will wipe out the project as well.
Do you think that should be discuss in other bug?
Comment 17 David Mason 2013-03-17 19:01:45 EDT
(In reply to comment #16)
> The other one on the document list from workspace, on the upper right of the
> page. That one does not work.

That is an oversight on my part. Assigning to myself again and will set ON_QA again when this is fixed.

> > > 2, Project maintainers may accidently wipe out the project
<snip>
> > Does this occur only with the "offline po" zip file, or can project
> > maintainers wipe out a project with the zip file from a gettext or podir
> > project as well?
> 
> Yes, they will wipe out the project as well.
> Do you think that should be discuss in other bug?

Yes, since this issue was present before offlinepo was added it should be dealt with as a separate bug.
Comment 18 Ding-Yi Chen 2013-03-17 21:47:27 EDT
> > > Does this occur only with the "offline po" zip file, or can project
> > > maintainers wipe out a project with the zip file from a gettext or podir
> > > project as well?
> > 
> > Yes, they will wipe out the project as well.
> > Do you think that should be discuss in other bug?
> 
> Yes, since this issue was present before offlinepo was added it should be
> dealt with as a separate bug.

Bug 922584 is created. Please put the further concerns and comments regarding source documents wipe out in that bug.
Comment 19 David Mason 2013-03-18 01:15:43 EDT
(In reply to comment #14)
> 1. Download all files (zip) in document list does not work for non-po based
> projects.
>    Click Projects->PROJECTNAME->VERSION->LOCALE 
>    and on the upper right, the button (Donload all files (zip) does not work.

Updated document list to allow zip download for non-po projects using msgctxt<->Zanata-id mapping, and bundled with a config file with project type set to 'offlinepo'.

Behaviour should be identical to zip download from documents page (see comment #10).

See: https://github.com/zanata/zanata/commit/5d9ac53899582f6607250fc150089dc6baec9483
Comment 20 Ding-Yi Chen 2013-03-19 00:13:59 EDT
Tested with Zanata version 2.3-SNAPSHOT (20130319-0038)
and zanata-cli-2.2.1-20130318.141403-57

1. In Download zip in GWT, the url is 'null'.
   url should be the server you download from

2. Push as push-type "trans" does not work.
  e.g. mvn zanata:push -Dzanata.pushType=trans
  did nothing and return BUILD SUCCESS.

Reassigned.
Comment 21 David Mason 2013-03-24 22:59:12 EDT
(In reply to comment #20)
> Tested with Zanata version 2.3-SNAPSHOT (20130319-0038)
> and zanata-cli-2.2.1-20130318.141403-57
> 
> 1. In Download zip in GWT, the url is 'null'.
>    url should be the server you download from

Fixed. See https://github.com/zanata/zanata/commit/2748bdff3ef03916f25c84d791d5f2af3b2dbc21


> 2. Push as push-type "trans" does not work.
>   e.g. mvn zanata:push -Dzanata.pushType=trans
>   did nothing and return BUILD SUCCESS.

This works for me with both cli and maven clients. More information is required about the project type, documents and maven client version to reproduce.
Comment 22 David Mason 2013-03-24 23:04:34 EDT
Updated offline-translation config so that only a single locale is specified and moved download link for offline-translation config to locale-documents page. Original config download link has been returned to its previous behaviour (with no popup).

See: https://github.com/zanata/zanata/compare/62dcb0f6bf3d221d0ed1a0a1968418e920dcd1f4...417e908632606f72de656671875f6ed0d1212d1d


Updated gwt document list to allow single-file download as .offlinepo when appropriate, and updated buttons on zip download popup to be more user-friendly.

See:
 - https://github.com/zanata/zanata/commit/f20dc1f619ca1f29d8d386a0521e92f4e643e92d
 - https://github.com/zanata/zanata/commit/860a63b3a49c203c7ccf9fd87cb8757e79fece99
Comment 23 Alex Eng 2013-04-02 23:42:19 EDT
Verified in 2.3-SNAPSHOT (20130403-1306).
Comment 24 Sean Flanigan 2013-04-12 00:40:22 EDT
Could we please confirm that Zanata never uses offlinepo for gettext/podir projects (or for projects with projectType of null)?  

If the projectType is not set, we should refuse to generate zanata.xml (with an error message suggesting that project maintainer should set project type).
Comment 25 David Mason 2013-04-14 23:50:39 EDT
(In reply to comment #24)
> Could we please confirm that Zanata never uses offlinepo for gettext/podir
> projects (or for projects with projectType of null)?  
> 
> If the projectType is not set, we should refuse to generate zanata.xml (with
> an error message suggesting that project maintainer should set project type).

Not-set project type was being treated like any other non-po project type when determining config file and single-file download behaviour. This has been changed so that:-

 - the offline-translation config file button is disabled when project type is not set.
 - individual .po download links are disabled when project type is not set.

See: https://github.com/zanata/zanata/commit/e2291c367279952dcfa70b82edd2816c9b548700
Comment 26 Ding-Yi Chen 2013-04-18 03:56:31 EDT
VERIFIED with 2.3-SNAPSHOT (20130418-1355)

Note: Projects -> <PROJECT> -> <VERSION> -> Action Config file stills generates zanata.xml, however, this is acceptable at this stage, because:
1) zanata-util uses this to obtain supported languages for a given project/version.

2) There is no <project-type> in the geneated zanata.xml, that is, clients will show error message for not specifying project-type.

Perhaps I will open 2 new bugs, one is REST API for obtaining languages list for a given project/version, another is disable the Action "Config file" if project-type is not set.
Comment 27 Sean Flanigan 2013-04-18 19:41:17 EDT
(In reply to comment #26)
> Note: Projects -> <PROJECT> -> <VERSION> -> Action Config file stills
> generates zanata.xml, however, this is acceptable at this stage, because:
> 1) zanata-util uses this to obtain supported languages for a given
> project/version.
> 
> 2) There is no <project-type> in the geneated zanata.xml, that is, clients
> will show error message for not specifying project-type.
> 
> Perhaps I will open 2 new bugs, one is REST API for obtaining languages list
> for a given project/version, another is disable the Action "Config file" if
> project-type is not set.

I think it's fine for the project version page to generate zanata.xml.  The important thing is not to generate zanata.xml *with project-type offlinepo* unless it is safe to use offlinepo.  The project version page's zanata.xml has blank project-type when projectType is null, so that should be fine.

The REST API for language list is worth an RFE.
Comment 28 Sean Flanigan 2013-11-26 22:15:47 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 29 Sean Flanigan 2013-11-26 22:16:58 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 30 Sean Flanigan 2013-11-26 22:19:06 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 31 Sean Flanigan 2013-11-26 22:25:09 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 32 Sean Flanigan 2013-11-26 22:33:45 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.
Comment 33 Sean Flanigan 2013-11-26 22:36:01 EST
Closing VERIFIED bugs for Zanata versions <= 3.1.

Note You need to log in before you can comment on or make changes to this bug.