Bug 879030 - pungi strips langpacks metadata from comps, breaks langpacks installation for DVD
pungi strips langpacks metadata from comps, breaks langpacks installation for...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
18
All All
unspecified Severity high
: ---
: ---
Assigned To: Fedora Packaging Toolset Team
Fedora Extras Quality Assurance
https://fedoraproject.org/wiki/Common...
: CommonBugs, Reopened
Depends On: 881756
Blocks: F18Blocker/F18FinalBlocker
  Show dependency treegraph
 
Reported: 2012-11-21 16:09 EST by Adam Williamson
Modified: 2012-12-13 18:46 EST (History)
12 users (show)

See Also:
Fixed In Version: yum-3.4.3-47.fc18
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-13 18:46:17 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Adam Williamson 2012-11-21 16:09:32 EST
As figured out by notting: pungi is stripping the <langpacks> section from comps, which is breaking install of langpacks from DVD (and if you do a network install using a repo created by pungi).

* comps from Beta RC1 DVD image:

[adamw@adam repodata]$ grep langpacks b662714c38fb63326868be3d82fc6229a5abd40d11f49e39a9e484bcf07d1502-Fedora-18-Beta-comps.xml
      <packagereq type="mandatory">yum-langpacks</packagereq>

* comps from https://dl.fedoraproject.org/pub/fedora/linux/development/18/x86_64/os/repodata/ :

[adamw@adam images]$ grep langpacks 7231ecf2aa3d3a16c8b8c2483992f72b0d71d09b4ec06348b4c3c840171fdf84-comps-f18.xml 
      <packagereq>yum-langpacks</packagereq>
  <langpacks>
  </langpacks>

Spot the difference. As notting put it:

<notting> pungi assembles the comps files from all the repos it uses for the compose, and uses yum.comps.xml() to write out a merged version. unfortunately, that means every section that yum.comps doesn't specifically know about falls out

Proposing NTH, to match the install 'langpacks doesn't work' bug - 868869 - which was NTH.
Comment 1 Adam Williamson 2012-11-21 21:49:18 EST
So the bit that needs fixing is actually yum. Specifically yum.comps.xml() . It needs to know about langpacks, and I guess anything else we've put in comps lately...
Comment 2 Kamil Páral 2012-11-23 10:42:53 EST
Do I understand correctly this could be a Final blocker?
 All critical path actions on release-blocking desktop environments should correctly display all sufficiently complete translations available for use 
https://fedoraproject.org/wiki/Fedora_18_Final_Release_Criteria

If DVD installation fails to install language packs, then we can't correctly display translations.
Comment 3 Adam Williamson 2012-11-24 03:17:54 EST
yes, it very likely is.
Comment 4 Kamil Páral 2012-11-26 10:19:59 EST
Adam, I have done F18 Beta DVD/netinst/Live installation with Czech language, and yum-langpacks is installed every time.

$ grep -E '(-lang|-cs|-czech)' dvd_packages
yum-langpacks-0.3.0-3.fc18.noarch
$ grep -E '(-lang|-cs|-czech)' netinst_packages
yum-langpacks-0.3.0-4.fc18.noarch
$ grep -E '(-lang|-cs|-czech)' live_packages
yum-langpacks-0.3.0-3.fc18.noarch

No other l10n packages are installed, but I can't even found any in the repositories. There used to be language specific yum groups, now there are none.

I wanted to document this on CommonBugs page, but I fail to understand this problem properly. yum-langpacks are installed every time and no other language packs are available. What am I missing?
Comment 5 Adam Williamson 2012-11-27 05:43:24 EST
kamil: AIUI, this, from comps-f18.xml.in:

  <langpacks>
    <match install="LabPlot-doc-%s" name="LabPlot-doc"/>
    <match install="aspell-%s" name="aspell"/>
    <match install="autocorr-%s" name="autocorr-en"/>
    <match install="calligra-l10n-%s" name="calligra-core"/>
    <match install="childsplay-alphabet_sounds_%s" name="childsplay"/>
    <match install="eclipse-nls-%s" name="eclipse-platform"/>
    <match install="firefox-langpack-%s" name="firefox"/>
    <match install="gcompris-sound-%s" name="gcompris"/>
    <match install="gimp-help-%s" name="gimp-help"/>
    <match install="hunspell-%s" name="hunspell"/>
    <match install="hyphen-%s" name="hyphen"/>
    <match install="kde-l10n-%s" name="kdelibs"/>
    <match install="kde-i18n-%s" name="kdelibs3"/>
    <match install="libreoffice-langpack-%s" name="libreoffice-core"/>
    <match install="man-pages-%s" name="man-pages"/>
    <match install="moodle-%s" name="moodle"/>
    <match install="mythes-%s" name="mythes"/>
    <match install="nqc-doc-%s" name="nqc-doc"/>
    <match install="openoffice.org-langpack-%s" name="openoffice.org-core"/>
    <match install="tesseract-langpack-%s" name="tesseract"/>
    <match install="tkgate-%s" name="tkgate"/>
  </langpacks>

If I'm understanding correctly, what this means is that, say, if libreoffice-core is being installed, you should get libreoffice-langpack-cs . If I'm correct about this bug, if you do a stock install via netinst you'll get probably at least aspell-cs , firefox-langpack-cs , libreoffice-langpack-cs , man-pages-cs , and maybe some others; if you do a stock install via DVD you'll get none of these.

it seems pretty obvious how this works from the above snippet - comps defines what packages have their translations split out into locale-named subpackages, and the exact format for the names, and then yum does a simple 'if this then that' operation when installing.
Comment 6 Adam Williamson 2012-11-27 05:44:19 EST
though, hum, now I see your post again, it looks like it failed even in the netinst case. can you check if you remove libreoffice-core and install it again, does libreoffice-langpack-cs get picked up?
Comment 7 Adam Williamson 2012-11-27 05:54:39 EST
I just did a netinst from Beta RC1 netinst iso using French as the language, picked minimal + Standard, and got man-pages-fr and hunspell-fr. Will compare DVD. If it's not working for Czech, maybe there's another bug, to do with the Czech country code or smth?
Comment 8 Adam Williamson 2012-11-27 06:09:18 EST
Yup, on my French DVD install, man-pages-fr and hunspell-fr were missing. So in French the bug is exactly as I thought it was.
Comment 9 Kamil Páral 2012-11-27 08:42:06 EST
Adam, I can confirm your findings. With DVD Czech, hunspell-cs is not installed, but it is installed when I do "yum reinstall hunspell". When I do "yum install libreoffice", libreoffice-langpack-cs is correctly installed.

With netinst Czech, hunspell-cs is installed out of the box. I have no idea why it was not the case in comment 4. Maybe I did some mistake.
Comment 10 Adam Williamson 2012-12-03 13:38:12 EST
Discussed at 2012-12-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-12-03/f18final-blocker-review-1.2.2012-12-03-17.25.log.txt . This is accepted as a blocker per criterion cited in comment #2 (note that web use is in the critpath, and Firefox translations are affected). This acceptance is based on our belief that a simple 'yum update firefox' (for e.g.) after install will not cause the translation to be added - I'm currently testing this. If it does cause the translation to be added, we may re-evaluate the blocker status for this bug.
Comment 11 Adam Williamson 2012-12-03 14:23:34 EST
I confirmed that a simple 'yum update firefox' after installing with this bug does not appear to add the firefox translation to the package set. So the above comment stands, this is a blocker.
Comment 12 Daniel Mach 2012-12-06 07:11:34 EST
This needs pungi and yum to be patched.
I think I've got patches for both.

Yum patch is attached to bug#881756.
I'll provide pungi patches today or tomorrow at latest (need to review my git first).
Comment 13 Jaroslav Reznik 2012-12-06 07:27:54 EST
Dan,
thanks - I'm going to ping yum guys to include the patch. And this bug should be split to two bugs for Fedora - yum and pungi one.
Comment 14 Tim Flink 2012-12-06 11:36:18 EST
I just tried a test compose with a pungi built using Daniel's patches but I'm getting a TB pretty early on in the process:

Start: chroot pungi --force --nosource --nodebuginfo -G -C -B -I --force --flavor Fedora --name Fedora --ver 18 -c /tmp/fedora-install-fedora.ks
Pungi:INFO: Adding repo fedora
Pungi:INFO: URL for repo fedora is ['http://infrastructure.phx2.fedoraproject.org/pub/fedora/linux/development/18/x86_64/os/']
Pungi:INFO: Adding repo siderepo
Pungi:INFO: URL for repo siderepo is ['file:///tmp/siderepo/']
Pungi:INFO: Getting sacks for arches ['x86_64', 'noarch', 'src']
fedora                                                   | 4.2 kB     00:00     
siderepo                                                 | 2.9 kB     00:00 !!! 
Pungi:WARNING: The --nobase kickstart option is no longer supported; Please add @base to the kickstart if you want it in.
Warning: Reusing existing destination directory.
Traceback (most recent call last):
  File "/usr/bin/pungi", line 264, in <module>
     main()
  File "/usr/bin/pungi", line 102, in main
     mypungi.getPackageObjects()
  File "/usr/lib/python2.7/site-packages/pypungi/__init__.py", line 608, in getPackageObjects
     if "core" not in self.ksparser.handler.packages.groupList:
  File "/usr/lib/python2.7/site-packages/pykickstart/parser.py", line 252, in __cmp__
     if self.name < other.name:
AttributeError :  'str' object has no attribute 'name' 

The warning about @base is a little annoying. I'm not using --nobase and I did add @base and @core to the ks.

I'm using pykickstart-1.99.22-1.fc18 on the builder and did upgrade yum to yum-3.4.3-47.fc18
Comment 15 Daniel Mach 2012-12-06 13:09:47 EST
Thanks for cathing that; here's a new build:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4764307
Comment 16 Tim Flink 2012-12-06 16:22:31 EST
(In reply to comment #15)
> Thanks for cathing that; here's a new build:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=4764307

Updated to this build and got a different TB:

Start: chroot pungi --force --nosource --nodebuginfo -G -C -B -I --force --flavor Fedora --name Fedora --ver 18 -c /tmp/fedora-install-fedora.ks
Pungi:INFO: Adding repo fedora
Pungi:INFO: URL for repo fedora is ['http://infrastructure.phx2.fedoraproject.org/pub/fedora/linux/development/18/x86_64/os/']
Pungi:INFO: Adding repo siderepo
Pungi:INFO: URL for repo siderepo is ['file:///tmp/siderepo/']
Pungi:INFO: Getting sacks for arches ['x86_64', 'noarch', 'src']
fedora                                                   | 4.2 kB     00:00     
siderepo                                                 | 2.9 kB     00:00 !!! 
Warning: Reusing existing destination directory.
Traceback (most recent call last):
  File "/usr/bin/pungi", line 264, in <module>
     main()
  File "/usr/bin/pungi", line 102, in main
     mypungi.getPackageObjects()
  File "/usr/lib/python2.7/site-packages/pypungi/__init__.py", line 621, in getPackageObjects
     searchlist.extend(self.getPackagesFromGroup(group))
  File "/usr/lib/python2.7/site-packages/pypungi/__init__.py", line 550, in getPackagesFromGroup
     pkgs = self.ayum.bestPackagesFromList(pkgs, arch=self.ayum.compatarch)
AttributeError :  'PungiYum' object has no attribute 'compatarch' 
Finish: chroot pungi --force --nosource --nodebuginfo -G -C -B -I --force --flavor Fedora --name Fedora --ver 18 -c /tmp/fedora-install-fedora.ks
Comment 17 Fedora Update System 2012-12-07 05:10:11 EST
yum-3.4.3-47.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/FEDORA-2012-19846/yum-3.4.3-47.fc18
Comment 18 Daniel Mach 2012-12-07 09:16:38 EST
I've fixed it but found a single dependency incorrectly resolved when I tested the pungi build on RHEL. I'll need to write new tests to cover this corner-case.

I also realized there might be a workaround.
You could achieve all this by using the original pungi (pungi-2.12-1.fc18) and patched yum (yum-3.4.3-47.fc18) and updating the langpack patterns in f18 kickstart (see the # Langpacks section in /usr/share/spin-kickstarts/fedora-install-fedora.ks).
Comment 19 Tim Flink 2012-12-10 17:08:46 EST
(In reply to comment #18)
> I also realized there might be a workaround.
> You could achieve all this by using the original pungi (pungi-2.12-1.fc18)
> and patched yum (yum-3.4.3-47.fc18) and updating the langpack patterns in
> f18 kickstart (see the # Langpacks section in
> /usr/share/spin-kickstarts/fedora-install-fedora.ks).

OK, but I don't quite understand what you mean about updating the langpack patters in the kickstart. Can you be a bit more specific?
Comment 20 Fedora Update System 2012-12-11 00:54:03 EST
yum-3.4.3-47.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 21 Daniel Mach 2012-12-11 02:50:15 EST
(In reply to comment #19)
> (In reply to comment #18)
> > I also realized there might be a workaround.
> > You could achieve all this by using the original pungi (pungi-2.12-1.fc18)
> > and patched yum (yum-3.4.3-47.fc18) and updating the langpack patterns in
> > f18 kickstart (see the # Langpacks section in
> > /usr/share/spin-kickstarts/fedora-install-fedora.ks).
> 
> OK, but I don't quite understand what you mean about updating the langpack
> patters in the kickstart. Can you be a bit more specific?


/usr/share/spin-kickstarts/fedora-install-fedora.ks

# Langpacks
aspell-*
autocorr-*
eclipse-nls-*
hunspell-*
hyphen-*
calligra-l10n-*
kde-l10n-*
kde-i18n-*
libreoffice-langpack-*
man-pages-*
mythes-*


If there are some other langpacks to be pulled into composes, then they have to be added here (AFAIK there are more langpacks in comps than these).
Comment 22 Kamil Páral 2012-12-11 04:56:59 EST
We still need to verify, setting ON_QA.
Comment 23 Bill Nottingham 2012-12-11 14:08:15 EST
Those langpacks should be all the ones *for things that are pulled onto the DVD*.
Comment 24 Bill Nottingham 2012-12-11 14:11:22 EST
I lied. I'll add gimp-help-*.
Comment 25 Tim Flink 2012-12-12 18:15:30 EST
I built a smoke image with yum-3.4.3-47 and the changes to fedora-install-fedora.ks (and a few other changes that shouldn't affect this bug)

http://dl.fedoraproject.org/pub/alt/qa/20121212_f18-smoke6/
Comment 26 Adam Williamson 2012-12-12 18:41:45 EST
Small problem with adding gimp-help-*: it don't fit.

Each gimp-help package is approx. 30MB:

Name        : gimp-help-de
Arch        : noarch
Version     : 2.8.0
Release     : 3.fc18
Size        : 31 M

and there are 20 languages. So smoke6 DVD, built with this change, is 600MB larger than smoke5 DVD, and is now oversize.
Comment 27 Jaroslav Reznik 2012-12-13 08:36:06 EST
Maybe we should be more specific on supported languages (do we actually have a list of "the must" languages supported on install media???) - for gimp - 600 MB just for langpacks is really a lot.

So there are two options:
a) cut down supported languages (could be a little bit messy in ks as all languages has to be listed but with limited set...)
b) be more selective in what langpacks we want on DVD (but it's probably cleaner to install only specific language than selectivly not install langpack for a few packages)
c) but it's long term one - do the review of what we already have now and cut it down...
Comment 28 Bill Nottingham 2012-12-13 12:16:31 EST
So, yeah, gimp-help is way too big for the ~150MB of space we have available. I've removed it and we will just have a langpack gap with gimp on the DVD. If it wasn't the last minute, we could discuss more nuanced cutting of other groups or content.
Comment 29 Adam Williamson 2012-12-13 18:46:17 EST
So, with a smoke6 French install, I get hunspell-fr and man-pages-fr . So the bug seems fixed.

There are a few nitty issues in the langpacks themselves - the openoffice one looks like it needs updating to libreoffice, and the Firefox langpack packages no longer seem to exist, they seem to be back in the main firefox package. But I believe this bug is fixed.

Note You need to log in before you can comment on or make changes to this bug.