Bug 870876

Similar warning messages could be observed while pulling the non-po file project.

$ mvn zanata:pull -Dzanata.pullType=both -Dzanata.locales=XX
...
...
[WARNING] Missing POT entry for text-flow ID textFlow_getId

This happens because the PO upload (and REST clients) hash msgstr+msgctxt to a textflow resId in a way which does not correspond to the resIds we use for non-PO projects.  Non-PO projects typically use an identifier such as trans-unit id or property key directly as a textflow resId.  Thus when the PO file is uploaded to the server, its generated resId does not match the textflow resId.

*** Bug 873933 has been marked as a duplicate of this bug. ***

I have some implementation ideas recorded against bug 873933.

*** Bug 870873 has been marked as a duplicate of this bug. ***

Updated Java clients to accept project type 'offlinepo'. This type is exclusively for offline translation working with po files generated from non-po project types and is not designed for normal project maintenance. This is a client-only project type, so it cannot be selected on the server.

Client behaviour when using project type 'offlinepo'
 - pull generates reversible po and pot files with Zanata id in msgctxt
 - push uses msgctxt for id in Zanata
 - as long as msgctxt is not modified, translations should always be correctly associated with source in Zanata when they are pushed.

Warning: offlinepo should not be used for gettext or podir projects as they may already have msgctxt, which would cause an error (this is intentional to prevent accidental overwriting of msgctxt)

see: https://github.com/zanata/zanata-client/commit/df16d22dd184f9db27d1ef3cbe9ed6eb183e3dc6

Updated server to use msgctxt<->Zanata-id mapping in downloaded and uploaded po/pot files at appropriate times:

 - uploads to gettext and podir projects behave exactly as before (do not use msgctxt mappings)
 - downloads from gettext and podir projects behave as before

 - uploads of pot files to 'file' projects (which may contain pot files):
     - if it is a new file, does not use mapping
     - if it is overwriting an existing pot file, does not use mapping
     - if it is overwriting a non-pot file, it will use the new mapping
 - download of pot files from 'file' projects
     - if the original is a pot file, does not use mapping
     - if the original is any other type, mapping is used

 - uploads of pot files to all other project types will use the mapping
     - since pot files are not allowed in these projects, it is assumed that pot files uploaded to these must have been generated from non-po files
 - downloads of po/pot files from all other project types will use the mapping

see: https://github.com/zanata/zanata/compare/1dbaf6e6234578fdfcc2316f945de51048656119...cc57236f73bb81e930aab584ba63ea1f76aa0a70

Updated server to allow zip download for non-po projects using msgctxt<->Zanata-id mapping, and bundled with a config file with project type set to 'offlinepo'.

There are 3 possible behaviours:

1. No project type is configured for the iteration: will show the normal zip download button, disabled with a tooltip about maintainer setting a project type.
2. Project type set to gettext or podir: normal zip download button is shown and enabled, downloaded zip has a normal config file and po files without msgctxt mapping.
3. Project type is set, and is not gettext or podir: "offline po" zip download button is shown and enabled, downloaded zip has config file with project-type set to 'offlinepo' and contains po files with msgctxt mapping.


At this stage, downloads of plain config files (not in zip) always have the configured project type (or a comment with valid types if no type is configured).

See: https://github.com/zanata/zanata/commit/b7cfba35139631a22afdede4f563bde8c71b64d1

Updated server to provide an additional config file with project type 'offlinepo' for non-po projects. Config file link now opens a modal dialog that has links for the default config file and special offline-translation config file.

See: https://github.com/zanata/zanata/commit/7a3ff69e5df5262d5d5bcbc89c1a2b4670b06bff


This and the above changes should cover all standard ways to retrieve and submit po files for offline translation.

For users that act as both project maintainers and translators (e.g. input methods) they can either put zanata.xml in different directories, or use different filenames and use --projectConfig to assign it.

(In reply to comment #12)
> For users that act as both project maintainers and translators (e.g. input
> methods) they can either put zanata.xml in different directories, or use
> different filenames and use --projectConfig to assign it.

I would strongly advise only the first option - using completely separate directories.

The default config file will pull and push files in their original format, whereas the offline translation config file will pull and push files as offlinepo. The files are incompatible, so there is no use case for using the different config files in the same zanata directory.

In other words, files for offline translation (config file and pulled files) should never be mixed with files for normal 'project maintainer' activities.

Tested with Zanata version 2.3-SNAPSHOT (20130314-1234),
Maven client 2.2.1-SNAPSHOTS

1. Download all files (zip) in document list does not work for non-po based projects.
   Click Projects->PROJECTNAME->VERSION->LOCALE 
   and on the upper right, the button (Donload all files (zip) does not work.

2, Project maintainers may accidently wipe out the project
   Step to reproduce:
   a. Download and extract the zip file that contains po
   b. push with maven client
      mvn zanata:push

   Actual:
   Project documents are wiped out

   Expected:
   1) Error messages states that no pot exists.
   or 2) Update po only, as if -Dzanata.pushType=trans are given.

In other words, the push should be fool proof.

(In reply to comment #14)
> Tested with Zanata version 2.3-SNAPSHOT (20130314-1234),
> Maven client 2.2.1-SNAPSHOTS
> 
> 1. Download all files (zip) in document list does not work for non-po based
> projects.
>    Click Projects->PROJECTNAME->VERSION->LOCALE 
>    and on the upper right, the button (Donload all files (zip) does not work.

Is the project type configured for that project or version?

(from comment #10)
[if] No project type is configured for the iteration: will show the normal zip download button, disabled with a tooltip about maintainer setting a project type.


> 2, Project maintainers may accidently wipe out the project
>    Step to reproduce:
>    a. Download and extract the zip file that contains po
>    b. push with maven client
>       mvn zanata:push
> 
>    Actual:
>    Project documents are wiped out
> 
>    Expected:
>    1) Error messages states that no pot exists.
>    or 2) Update po only, as if -Dzanata.pushType=trans are given.
> 
> In other words, the push should be fool proof.

Does this occur only with the "offline po" zip file, or can project maintainers wipe out a project with the zip file from a gettext or podir project as well?

Yes, it is, however, we have 2 ways to download po file(In reply to comment #15)
> (In reply to comment #14)
> > Tested with Zanata version 2.3-SNAPSHOT (20130314-1234),
> > Maven client 2.2.1-SNAPSHOTS
> > 
> > 1. Download all files (zip) in document list does not work for non-po based
> > projects.
> >    Click Projects->PROJECTNAME->VERSION->LOCALE 
> >    and on the upper right, the button (Donload all files (zip) does not work.
> 
> Is the project type configured for that project or version?
>
> (from comment #10)
> [if] No project type is configured for the iteration: will show the normal
> zip download button, disabled with a tooltip about maintainer setting a
> project type.
 
Yes, it it configured as properties.
However, we do have 2 ways to download ZIP.
One is Projects->PROJECTNAME->VERSION-> LOCALE Row Documents Col -> Action Donload All Files. That works.

The other one on the document list from workspace, on the upper right of the page. That one does not work.

 
> > 2, Project maintainers may accidently wipe out the project
> >    Step to reproduce:
> >    a. Download and extract the zip file that contains po
> >    b. push with maven client
> >       mvn zanata:push
> > 
> >    Actual:
> >    Project documents are wiped out
> > 
> >    Expected:
> >    1) Error messages states that no pot exists.
> >    or 2) Update po only, as if -Dzanata.pushType=trans are given.
> > 
> > In other words, the push should be fool proof.
> 
> Does this occur only with the "offline po" zip file, or can project
> maintainers wipe out a project with the zip file from a gettext or podir
> project as well?

Yes, they will wipe out the project as well.
Do you think that should be discuss in other bug?

(In reply to comment #16)
> The other one on the document list from workspace, on the upper right of the
> page. That one does not work.

That is an oversight on my part. Assigning to myself again and will set ON_QA again when this is fixed.

> > > 2, Project maintainers may accidently wipe out the project
<snip>
> > Does this occur only with the "offline po" zip file, or can project
> > maintainers wipe out a project with the zip file from a gettext or podir
> > project as well?
> 
> Yes, they will wipe out the project as well.
> Do you think that should be discuss in other bug?

Yes, since this issue was present before offlinepo was added it should be dealt with as a separate bug.

> > > Does this occur only with the "offline po" zip file, or can project
> > > maintainers wipe out a project with the zip file from a gettext or podir
> > > project as well?
> > 
> > Yes, they will wipe out the project as well.
> > Do you think that should be discuss in other bug?
> 
> Yes, since this issue was present before offlinepo was added it should be
> dealt with as a separate bug.

Bug 922584 is created. Please put the further concerns and comments regarding source documents wipe out in that bug.

(In reply to comment #14)
> 1. Download all files (zip) in document list does not work for non-po based
> projects.
>    Click Projects->PROJECTNAME->VERSION->LOCALE 
>    and on the upper right, the button (Donload all files (zip) does not work.

Updated document list to allow zip download for non-po projects using msgctxt<->Zanata-id mapping, and bundled with a config file with project type set to 'offlinepo'.

Behaviour should be identical to zip download from documents page (see comment #10).

See: https://github.com/zanata/zanata/commit/5d9ac53899582f6607250fc150089dc6baec9483

Tested with Zanata version 2.3-SNAPSHOT (20130319-0038)
and zanata-cli-2.2.1-20130318.141403-57

1. In Download zip in GWT, the url is 'null'.
   url should be the server you download from

2. Push as push-type "trans" does not work.
  e.g. mvn zanata:push -Dzanata.pushType=trans
  did nothing and return BUILD SUCCESS.

Reassigned.

(In reply to comment #20)
> Tested with Zanata version 2.3-SNAPSHOT (20130319-0038)
> and zanata-cli-2.2.1-20130318.141403-57
> 
> 1. In Download zip in GWT, the url is 'null'.
>    url should be the server you download from

Fixed. See https://github.com/zanata/zanata/commit/2748bdff3ef03916f25c84d791d5f2af3b2dbc21


> 2. Push as push-type "trans" does not work.
>   e.g. mvn zanata:push -Dzanata.pushType=trans
>   did nothing and return BUILD SUCCESS.

This works for me with both cli and maven clients. More information is required about the project type, documents and maven client version to reproduce.

Updated offline-translation config so that only a single locale is specified and moved download link for offline-translation config to locale-documents page. Original config download link has been returned to its previous behaviour (with no popup).

See: https://github.com/zanata/zanata/compare/62dcb0f6bf3d221d0ed1a0a1968418e920dcd1f4...417e908632606f72de656671875f6ed0d1212d1d


Updated gwt document list to allow single-file download as .offlinepo when appropriate, and updated buttons on zip download popup to be more user-friendly.

See:
 - https://github.com/zanata/zanata/commit/f20dc1f619ca1f29d8d386a0521e92f4e643e92d
 - https://github.com/zanata/zanata/commit/860a63b3a49c203c7ccf9fd87cb8757e79fece99

Verified in 2.3-SNAPSHOT (20130403-1306).

Could we please confirm that Zanata never uses offlinepo for gettext/podir projects (or for projects with projectType of null)?  

If the projectType is not set, we should refuse to generate zanata.xml (with an error message suggesting that project maintainer should set project type).

(In reply to comment #24)
> Could we please confirm that Zanata never uses offlinepo for gettext/podir
> projects (or for projects with projectType of null)?  
> 
> If the projectType is not set, we should refuse to generate zanata.xml (with
> an error message suggesting that project maintainer should set project type).

Not-set project type was being treated like any other non-po project type when determining config file and single-file download behaviour. This has been changed so that:-

 - the offline-translation config file button is disabled when project type is not set.
 - individual .po download links are disabled when project type is not set.

See: https://github.com/zanata/zanata/commit/e2291c367279952dcfa70b82edd2816c9b548700

VERIFIED with 2.3-SNAPSHOT (20130418-1355)

Note: Projects -> <PROJECT> -> <VERSION> -> Action Config file stills generates zanata.xml, however, this is acceptable at this stage, because:
1) zanata-util uses this to obtain supported languages for a given project/version.

2) There is no <project-type> in the geneated zanata.xml, that is, clients will show error message for not specifying project-type.

Perhaps I will open 2 new bugs, one is REST API for obtaining languages list for a given project/version, another is disable the Action "Config file" if project-type is not set.

(In reply to comment #26)
> Note: Projects -> <PROJECT> -> <VERSION> -> Action Config file stills
> generates zanata.xml, however, this is acceptable at this stage, because:
> 1) zanata-util uses this to obtain supported languages for a given
> project/version.
> 
> 2) There is no <project-type> in the geneated zanata.xml, that is, clients
> will show error message for not specifying project-type.
> 
> Perhaps I will open 2 new bugs, one is REST API for obtaining languages list
> for a given project/version, another is disable the Action "Config file" if
> project-type is not set.

I think it's fine for the project version page to generate zanata.xml.  The important thing is not to generate zanata.xml *with project-type offlinepo* unless it is safe to use offlinepo.  The project version page's zanata.xml has blank project-type when projectType is null, so that should be fine.

The REST API for language list is worth an RFE.

Closing VERIFIED bugs for Zanata versions <= 3.1.

Closing VERIFIED bugs for Zanata versions <= 3.1.

Closing VERIFIED bugs for Zanata versions <= 3.1.

Closing VERIFIED bugs for Zanata versions <= 3.1.

Closing VERIFIED bugs for Zanata versions <= 3.1.

Closing VERIFIED bugs for Zanata versions <= 3.1.