Bug 1785028 - please migrate to the new Fedora translation platform
Summary: please migrate to the new Fedora translation platform
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 32
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jiri Konecny
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-18 22:34 UTC by jibecfed
Modified: 2020-02-21 22:17 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)

Description jibecfed 2019-12-18 22:34:59 UTC
Hello, the Fedora project migrates its translation platform to Weblate [1].

This tool directly interact with your git repository, and requires us to know:
* [mandatory] which branch is your development branch?
* [mandatory] have you merged latest translation from Zanata and locked the project?
* [info] Weblate will handle updates when pot file changes, don't edit po files for this [2]
* [optional] what is the license of translation? (please use a code from https://spdx.org/licenses/)
* [optional] do you have any announcement/warning you would like to display to the translators? (it will be displayed in Weblate)
* [optional] do you need us to activate any specific checks? (this is a setting per component [3])
* [optional] do you need us to automatically detect new translation files? (typical usecase: website translation with one translation file per page)

Please note:
* For github hosted projects, Weblate open pull request. For other project, you'll have to add a key to allow commits.
* In Weblate's vocable, one project is a group of component. Each component is a translation file. You can have many projects or many components or both.
* You can change your mind over time, just reach trans@lists.fedoraproject.org

[1] https://communityblog.fedoraproject.org/fedora-localization-platform-migrates-to-weblate/ 
[2] https://docs.weblate.org/en/latest/admin/continuous.html#avoiding-merge-conflicts
[3] https://docs.weblate.org/en/latest/user/checks.html#translation-checks

Comment 1 Martin Kolman 2019-12-19 14:36:42 UTC
(In reply to jibecfed from comment #0)
> Hello, the Fedora project migrates its translation platform to Weblate [1].
Thanks for the heads up!
> 
> This tool directly interact with your git repository, and requires us to
> know:
> * [mandatory] which branch is your development branch?
Rawhide development happens in the "master" branch, development for a specific
Fedora release happens in "f<number>-branch" once a given Fedora release has been branched.

RHEL releases follow a broadly similar outline:

"rhel7-branch" for RHEL 7
"rhel-8" for RHEL 8

So far, the Zanata branches with similar names have been created, to track the given release stream.

Also CCink Ludek Janda due to the impact the Weblate migration might have on RHEL localization & 
at the moment the RHEL Anaconda translations go via the rhel* branches in Zanata.

> * [mandatory] have you merged latest translation from Zanata and locked the
> project?
We do not have translations as part of our repository, instead we pull them in
only at Anaconda release time, when an official release tarball is created.

So I guess we don't need to do anything (maybe other than locking the Zanata project (how ?))
and I assume the content will be migrated to Weblate for us automatically ?

> * [info] Weblate will handle updates when pot file changes, don't edit po
> files for this [2]
This is what we do now at release time, we call the python Zanata client to push the pot file
to Zanata. So I guess Weblate will be doing this automatically for us ? The pot
file will still be regenerated only at release time with our current workflow
though, so externally not much would change.

> * [optional] what is the license of translation? (please use a code from
> https://spdx.org/licenses/)
> * [optional] do you have any announcement/warning you would like to display
> to the translators? (it will be displayed in Weblate)
> * [optional] do you need us to activate any specific checks? (this is a
> setting per component [3])
> * [optional] do you need us to automatically detect new translation files?
> (typical usecase: website translation with one translation file per page)
I think we have just a single pot file for the whole of Anaconda, so I guess
we don't need this ? Or do you mean something else ?

> 
> Please note:
> * For github hosted projects, Weblate open pull request. For other project,
> you'll have to add a key to allow commits.
I don't think we need this for Anaconda. As mentioned earlier, we don't have
the translations as part of the Anaconda project source code, so there should
be no need to open PRs and/or to trigger some automated commits. I assume Weblate
also has a CLI client, that we can call from our makefile (like we do for Zanata
Python client currently) to pull down updated translations at tarball creation time ?

BTW, I've checked a couple projects using Weblate that have it open PRs and/or automatically
commit translation updates and it really seems that it introduces quite a substantial unnecessary
churn in the git commit history. The commits seem to be happening basically at random,
possibly with just a single string translation being changed.

Given that new Anaconda releases happen roughly every 10-14 days and possibly even more often for
a branched Fedora release, I think batching translation updates with releases makes much more sense
for us. Translations will still be up to date & commit history will not be cluttered.


> * In Weblate's vocable, one project is a group of component. Each component
> is a translation file. You can have many projects or many components or both.
> * You can change your mind over time, just reach
> trans@lists.fedoraproject.org
> 
> [1]
> https://communityblog.fedoraproject.org/fedora-localization-platform-
> migrates-to-weblate/ 
> [2]
> https://docs.weblate.org/en/latest/admin/continuous.html#avoiding-merge-
> conflicts
> [3] https://docs.weblate.org/en/latest/user/checks.html#translation-checks

Comment 2 jibecfed 2019-12-31 10:46:20 UTC
(In reply to Martin Kolman from comment #1)
> (In reply to jibecfed from comment #0)
> > Hello, the Fedora project migrates its translation platform to Weblate [1].
> Thanks for the heads up!
> > 
> > This tool directly interact with your git repository, and requires us to
> > know:
> > * [mandatory] which branch is your development branch?
> Rawhide development happens in the "master" branch, development for a
> specific
> [..]
> So far, the Zanata branches with similar names have been created, to track
> the given release stream.


We can do the same using Weblate.

> > * [mandatory] have you merged latest translation from Zanata and locked the
> > project?
> We do not have translations as part of our repository, instead we pull them
> in
> only at Anaconda release time, when an official release tarball is created.
> 
> So I guess we don't need to do anything (maybe other than locking the Zanata
> project (how ?))
> and I assume the content will be migrated to Weblate for us automatically ?

Nope, you need to provide a git repository, and once done, I'll hand the migration.
What I suggest you to do is to have a localization specific repository with all pot and po files to translate.
You may wish to have only one branch for this repository, and to handle source branches as folders.

Note: I will be able to automate the detection of new branches (see the regex example to have an idea of how to organize files and directories):
https://docs.weblate.org/en/weblate-3.10/admin/addons.html#component-discovery


> * [info] Weblate will handle updates when pot file changes, don't edit po
> > files for this [2]
> This is what we do now at release time, we call the python Zanata client to
> push the pot file to Zanata. So I guess Weblate will be doing this automatically for us ? The
> pot > file will still be regenerated only at release time with our current workflow
> though, so externally not much would change.

The idea is to have translators constantly up to date with your translation.
If you follow the previous suggestion (dedicated repo for localization), I encourage you to update pot file at every commit.
But if your workflow already includes an update every 10 to 14 days, it may indeed be sufficient.

> > * [optional] do you need us to automatically detect new translation files?
> > (typical usecase: website translation with one translation file per page)
> I think we have just a single pot file for the whole of Anaconda, so I guess
> we don't need this ? Or do you mean something else ?

Answered ^^

> > Please note:
> > * For github hosted projects, Weblate open pull request. For other project,
> > you'll have to add a key to allow commits.
> I don't think we need this for Anaconda. As mentioned earlier, we don't have
> the translations as part of the Anaconda project source code, so there should
> be no need to open PRs and/or to trigger some automated commits. I assume
> Weblate also has a CLI client, that we can call from our makefile (like we do for Zanata
> Python client currently) to pull down updated translations at tarball
> creation time ?

Answered^^

> BTW, I've checked a couple projects using Weblate that have it open PRs
> and/or automatically
> commit translation updates and it really seems that it introduces quite a
> substantial unnecessary
> churn in the git commit history. The commits seem to be happening basically
> at random,
> possibly with just a single string translation being changed.
> 
> Given that new Anaconda releases happen roughly every 10-14 days and
> possibly even more often for
> a branched Fedora release, I think batching translation updates with
> releases makes much more sense
> for us. Translations will still be up to date & commit history will not be
> cluttered.

Well, for a big project such as anaconda, you may wish to have a git squash activated:
https://docs.weblate.org/en/weblate-3.10/admin/addons.html#squash-git-commits
I usually activate it by default per author for easier attribution and git stats, but if this proves to be too much, this can be changed. You'll be admin of your project anyway.

Comment 3 Jiri Konecny 2020-01-02 12:14:00 UTC
If we will go to the suggested separate localization-only repository (which I would personally prefer) then I think we don't mind bigger amount of the commits because they won't be part of the main development repository.

Comment 4 Jiri Konecny 2020-01-06 13:36:13 UTC
I have a few more questions about the separate localization repository.

* Do we need to update pot file in the new localization repository or will that be done somehow auto-magically?
* Is there a recommended way? Should we use git-submodule/git-subtree, push pot files there and pull po files when creating tarball?
* Could you please give us link to a project with similar use-case?

Comment 5 jibecfed 2020-01-06 21:26:49 UTC
(In reply to Jiri Konecny from comment #4)
> * Do we need to update pot file in the new localization repository or will
> that be done somehow auto-magically?

You'll have to manually or automatically update the pot files.

> * Is there a recommended way? Should we use git-submodule/git-subtree, push
> pot files there and pull po files when creating tarball?

With my git knowledge, I would rather not link the repositories.
Keep it simple. But you are the dev experts here, and it's your project.

> * Could you please give us link to a project with similar use-case?

I have complex scenario: https://pagure.io/fedora-l10n/fedora-release-notes/blob/master/f/pot
and a simple one: https://github.com/cockpit-project/cockpit-podman-weblate

Please note the file organization depends on you. It just have to be consistent.
For the complex scenario, it made more sense for us to split pot and po, and inside po directory, split per language.

In you scenario, I suggest to use the same Workflow I suggested for DNF: https://bugzilla.redhat.com/show_bug.cgi?id=1787204#c2

Comment 6 Jiri Konecny 2020-01-07 09:28:27 UTC
Hi Ludek,

I want to ask you if you are ok to migrate Anaconda and related projects from Zanata to Weblate even for RHEL translations? It would be great for us to have all the translations on one place. Even thought I'm not that sure about older RHELs (6 and olders).

When I'm looking into code I suggest this solution:

- rhel5-branch
  - translations are part of the repository
  - suggestion: leave it be as it is

- rhel6-branch
  - only pot is part of the repository, Zanata has po files
  - have to adjust Makefiles to use new Weblate translations
  - almost no development, chance that new translations will be required is really small
  - suggestion: move translations from Zanata to Weblate but do not change Makefiles until(if) needed

- rhel7-branch & rhel-8
  - only pot is part of the repository, Zanata has po files
  - have to adjust Makefiles to use new Weblate translations
  - change that new translations will be required is high
  - suggestion: move translations and create bugs to change Makefiles to be able to work with Weblate


Ludek do you agree or do you need to stay with Zanata for some reason?

Comment 7 Ludek Janda 2020-01-07 09:32:26 UTC
Agreed. No reason to stay with Zanata. Just please make sure I have write access to the new platform. Otherwise we'll need to figure out how to export PO files for us during RHEL 7 and 8 localization phases and send them over to me in ZIP file.

Comment 8 jibecfed 2020-01-07 20:47:58 UTC
You'll have admin rights in Weblate for this projects, no worries.

Anyone will have read access exporting it from the translation repository shouldn't provide any specific rights.
Everyone with a FAS account will have ability to edit existing translations in Weblate (either by uploading a po file or changing string by string).

Slight modification in what I said earlier: please create one repository per project so that I can properly configure "Repository browser" setting of a component, see https://docs.weblate.org/en/weblate-3.10/admin/projects.html#component

Comment 9 Ludek Janda 2020-01-14 12:46:02 UTC
Please can you delay migration and start after February 18? We are in the middle of RHEL 8.2 localization cycle, our team is working in Zanata and I want to minimize potential risk. Thanks.

Comment 10 jibecfed 2020-01-17 18:26:28 UTC
I asked the community for opinion on this, but you were included in the conversation, please answer: https://lists.fedoraproject.org/archives/list/trans@lists.fedoraproject.org/thread/I4JWI7KS2IO2SER3QHFOOTOFTEC6KGEW/

I see no reasons to delay and ask you to not delay this migration.

Comment 11 Ben Cotton 2020-02-11 17:33:40 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Comment 12 Jiri Konecny 2020-02-12 14:57:07 UTC
We have repository ready for translations:

https://github.com/rhinstaller/anaconda-l10n

There are translations from Fedora Rawhide now. It's the same translations for Fedora 32 because we did not branched yet. It also has write access for the weblate bot.

Could you please set-up the weblate project for us. Thanks!

Comment 13 Jiri Konecny 2020-02-12 14:57:40 UTC
Also Zanata is set to read-only for Fedora Rawhide now.

Comment 14 jibecfed 2020-02-13 04:46:11 UTC
configuration is done: https://translate.fedoraproject.org/projects/anaconda/master/

please connect to translation platform and give me your username so I can make you an admin

Weblate will update po files when pot file is updated

Comment 15 Jiri Konecny 2020-02-13 08:35:31 UTC
My username is:

jkonecny

Comment 16 jibecfed 2020-02-17 12:42:08 UTC
You are now admin.

Please note I see you still have active Branches in Zanata, which is weird.
Please check with RHEL localization manager on how to handle RHEL branches in Weblate.

Comment 17 Jiri Konecny 2020-02-17 12:58:55 UTC
Thank you.


Yes I know, I wanted to migrate Rawhide completely (including makefile and workflow) before I'll start working on other branches migration.

Comment 18 Jiri Konecny 2020-02-21 13:43:34 UTC
Hello jibecfed,

I wanted to ask about autodiscover of branches. We've created new f32 branch yesterday but it wasn't added to the weblate project.

Could you please look on this issue?

Comment 19 Jiri Konecny 2020-02-21 13:57:44 UTC
What can be possible problem is that we decided to name Fedora branches as

f32 (no -release or -devel suffixes)

Comment 20 jibecfed 2020-02-21 22:16:33 UTC
You may think about using this addon: https://docs.weblate.org/en/weblate-3.11.1/admin/addons.html#component-discovery

If you wish to use it, you'll have to have a file hierarchy allowing it.
Don't use git branches, and create one folder per branch.


Note You need to log in before you can comment on or make changes to this bug.