Bug 2031215 (F36Workstationraw-xzaarch64Oversize)

(Assigning to Allan as he was the last person to send a Workstation WG agenda. Feel free to pass the hot potato around).

Workstation raw-xz aarch64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20211229.n.1/compose/Workstation/aarch64/images/Fedora-Workstation-Rawhide-20211229.n.1.aarch64.raw.xz from compose Fedora-Rawhide-20211229.n.1 is 4082575488 bytes, exceeding the maximum size 4000000000.

Workstation raw-xz aarch64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20220114.n.0/compose/Workstation/aarch64/images/Fedora-Workstation-Rawhide-20220114.n.0.aarch64.raw.xz from compose Fedora-Rawhide-20220114.n.0 is 4031120192 bytes, exceeding the maximum size 4000000000.

Felipe noticed that Fedora-Workstation-Rawhide-20220118.n.0.aarch64.raw.xz is only 3.7 GB large, so it seems this issue resolved itself.

Yes seems to have dipped below 4GB again:

$ curl -s -I https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Workstation/aarch64/images/Fedora-Workstation-Rawhide-20220118.n.0.aarch64.raw.xz | grep Content-Length
Content-Length: 3920812232

Here's a diff of added / removed packages between the 2021-12-03 compose (the last one that was definitely under-size) and today's compose, with sizes:

Added:

libgweather4: 23084499
gtksourceview5: 5239714
libadwaita: 1805189
zxing-cpp: 1180479
zstd: 947818
gnome-text-editor: 830663
antiword: 637208
liblouis-utils: 597558
gnome-desktop4: 524765
liblouisutdml: 413147
plocate: 375583
avahi-tools: 281328
rit-meera-new-fonts: 234967
cups-filters-braille: 220271
libportal: 166483
lilv-libs: 149385
liblouisutdml-utils: 105659
libffi3.1: 85616
google-noto-sans-gurmukhi-vf-fonts: 65698

Removed:

gedit: 14774858
gtksourceview4: 4753637
openssl1.1: 3775659
google-noto-sans-gurmukhi-fonts: 1894101
initscripts: 1365696
chkconfig: 805926
lilv: 499072
mlocate: 438333
lv2: 408333
smc-meera-fonts: 349446
radvd: 263706
iptables-legacy-libs: 158700
libsss_autofs: 112265

Note that doesn't account for size changes of packages that were present both times. Still, looks like there's an obvious main suspect: we're now including both libgweather and libgweather4, and both packages are quite large. Previously we were including only libgweather. Here's the list of what packages depend on each:

libgweather
-----------

evolution-0:3.43.1-1.fc36.x86_64
evolution-data-server-0:3.43.1.1-1.fc36.i686
evolution-data-server-0:3.43.1.1-1.fc36.x86_64
gnome-applets-0:3.42.0-1.fc36.x86_64
gnome-calendar-0:41.2-1.fc36.x86_64
gnome-panel-0:3.42.0-1.fc36.x86_64
gnome-settings-daemon-0:42~alpha-2.fc36.i686
gnome-settings-daemon-0:42~alpha-2.fc36.x86_64
gnome-weather-0:42~alpha-1.fc36.x86_64
libgweather-devel-0:40.0-2.fc35.i686
libgweather-devel-0:40.0-2.fc35.x86_64

libgweather4
------------

gnome-clocks-0:42~alpha-1.fc36.x86_64
gnome-initial-setup-0:42~alpha.1-2.fc36.x86_64
gnome-shell-0:42~alpha-2.fc36.x86_64
libgweather4-devel-0:3.91.0-1.fc36.i686
libgweather4-devel-0:3.91.0-1.fc36.x86_64

So, I guess we need to rebuild the things in list #1 against libgweather4.

Methodology, in case we ever need to do this again: go to the Koji tasks for the relevant image builds. Click on the 'show all results' link and you get a big 'rpmlist', which is a bunch of Python dicts representing a single package each. Copy/paste the list for each compose into `/tmp/pkglistold` and `/tmp/pkglistnew` (don't use gedit, it hates big lines), add a [ and the start and a ] at the end. To parse them as json, do:

sed -i -e 's,None,null,g' /tmp/pkglist*
sed -i -e "s,',\",g" /tmp/pkglist*

Then run this script on them:

    #!/usr/bin/python

    import json

    with open("/tmp/pkglistold", "r") as oldfh:
        oldpkgs = json.load(oldfh)
    with open("/tmp/pkglistnew", "r") as newfh:
        newpkgs = json.load(newfh)

    oldpkgnames = set([pkg["name"] for pkg in oldpkgs])
    newpkgnames = set([pkg["name"] for pkg in newpkgs])

    print("Added:\n")
    added = []
    for addedname in newpkgnames.difference(oldpkgnames):
        size = [pkg["size"] for pkg in newpkgs if pkg["name"] == addedname][0]
        added.append((addedname, size))
    added.sort(key=lambda x:x[1], reverse=True)
    for addedpkg in added:
        print(f"{addedpkg[0]}: {addedpkg[1]}")

    print("\nRemoved:\n")
    removed = []
    for removedname in oldpkgnames.difference(newpkgnames):
        size = [pkg["size"] for pkg in oldpkgs if pkg["name"] == removedname][0]
        removed.append((removedname, size))
    removed.sort(key=lambda x:x[1], reverse=True)
    for removedpkg in removed:
        print(f"{removedpkg[0]}: {removedpkg[1]}")

(In reply to Michael Catanzaro from comment #4)
> Felipe noticed that Fedora-Workstation-Rawhide-20220118.n.0.aarch64.raw.xz
> is only 3.7 GB large, so it seems this issue resolved itself.

Interesting. This is a Firefox bug! I was in a meeting using Firefox for WebRTC, so tried to download this file there. Firefox reports 3.7 GB. But that is wrong. Epiphany correctly reports 3.9 GB. In reality, 3920812232 bytes is 3.65 GiB or 3.9 GB. Firefox got the units wrong. :O

yeah, this is why the test case says to use curl -I , units are always a minefield, it's best to just go with the number of bytes :) Sorry for the mid-air collision, it seems we've now outweighed the addition of libgweather4, but it's still making things bigger than necessary.

Looks like the switch from gedit to gnome-text-editor landed since the last nominated compose (20220114.n.0) and that cuts about 14M, so that was just enough to push it under the limit. Getting rid of the libgweather duplication would provide more breathing space, though.

per mclasen, it seems that GTK+3 things will still be built against libgweather while GTK+4 things will be built against libgweather4, so we may be stuck carrying both for a while. We'll just have to be careful not to make anything else bigger, I guess.

Something's not right with the raw.xz image being this much larger than the ISO. I put my 2 cents in this issue: https://pagure.io/releng/issue/10513#comment-775491

Oh, yeah, I forgot we were tracking that too. Thanks.

Workstation raw-xz aarch64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20220202.n.1/compose/Workstation/aarch64/images/Fedora-Workstation-Rawhide-20220202.n.1.aarch64.raw.xz from compose Fedora-Rawhide-20220202.n.1 is 4037262020 bytes, exceeding the maximum size 4000000000.

Just a rehash...
3942640 -rw-r--r--. 1 chris chris  4037262020 Feb  3 14:49 Fedora-Workstation-Rawhide-20220202.n.1.aarch64.raw.xz
5625396 -rw-r--r--. 1 chris chris 13958643712 Feb  3 14:49 Fedora-Workstation-Rawhide-20220202.n.1.aarch64.raw

Put the raw on loop, kpartx, mount loop0p3 (the large btrfs partition), fstrim it...

3309432 -rw-r--r--. 1 chris chris 13958643712 Feb  3 15:33 Fedora-Workstation-Rawhide-20220202.n.1.aarch64.raw

Before xz compression, we're well below the max allowed. The gist is, there's a lot of garbage in the raw file that hasn't been trimmed away. What I'm not sure about is whether the image building tool is (a) issuing fstrim before umount, and (b) the qemu-kvm doing the writes to this raw file has the virtioblk drive to do discard="unmap". Lacking either of this is enough to result in the garbage in the raw file not being discarded.

Aside from that issue, though, something did happen between 0127.n.0 and 0202.n.1 that made a lot of images bigger, this isn't the only one that went oversize. I'm trying to track down the issue now.

So the issue for the netinst images turned out to be the RPM database move: lorax is set to strip /var/lib/rpm/* from installer images, but now the RPM DB has moved so it's not being stripped, that caused the growth. I sent a Lorax PR to fix it: https://github.com/weldr/lorax/pull/1213 .

I'm not sure that would be the cause for this, though. It might be something else?

Workstation raw-xz aarch64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20220205.n.1/compose/Workstation/aarch64/images/Fedora-Workstation-Rawhide-20220205.n.1.aarch64.raw.xz from compose Fedora-Rawhide-20220205.n.1 is 4055381676 bytes, exceeding the maximum size 4000000000.

This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

Since F36 is stable, this is now useless.