Bug 2151495 (F38Everythingbootx86_64Oversize) - Fedora 38: Everything boot x86_64 image exceeds maximum size
Summary: Fedora 38: Everything boot x86_64 image exceeds maximum size
Alias: F38Everythingbootx86_64Oversize
Product: Fedora
Classification: Fedora
Component: distribution
Version: 38
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Aoife Moloney
QA Contact: Fedora Extras Quality Assurance
Whiteboard: AcceptedBlocker
Depends On:
Blocks: FedoraOversizeTracker F38BetaBlocker F38FinalBlocker
TreeView+ depends on / blocked
Reported: 2022-12-07 11:17 UTC by Fedora QA Tools SIG
Modified: 2023-05-09 13:56 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2023-02-14 07:09:56 UTC
Type: ---

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github weldr lorax pull 1291 0 None open Firmware cleanups 2022-12 2022-12-09 20:31:03 UTC

Description Fedora QA Tools SIG 2022-12-07 11:17:15 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221207.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20221207.n.0.iso from compose Fedora-Rawhide-20221207.n.0 is 736929792 bytes, exceeding the maximum size 734003200. Canonical maximum sizes can be found at https://fedoraproject.org/wiki/Releases/38/Spins and https://fedoraproject.org/wiki/Releases/38/ReleaseBlocking . This check is run by the 'relval' tool, which has its own list of maximum sizes derived from those pages. If the maximum size used for this comparison is wrong, please add a comment and file a bug against relval at https://pagure.io/fedora-qa/relval/issues and it will be corrected. If you believe the canonical maximum size for an image should be changed, please follow the appropriate process before filing a relval bug.

Comment 1 Adam Williamson 2022-12-07 17:27:39 UTC
Size grew 18M between 20221206.n.0 and 20221207.n.0 . This should not be the kexec-tools thing discussed in https://bugzilla.redhat.com/show_bug.cgi?id=2149246 , it must be something else.

Comment 2 Adam Williamson 2022-12-07 17:58:53 UTC
Here's a quick guide to debugging this kinda problem:

1. Go to the compose logs for each compose and find the relevant log for the image - in this case, https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221207.n.0/logs/x86_64/buildinstall-Everything.x86_64.log and https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221206.n.0/logs/x86_64/buildinstall-Everything.x86_64.log
2. Depending on the image there'll either be a full log in the compose logs, or you'll only get a pointer to a Koji task. In this case it's the latter, so go to the Koji tasks: 20221206.n.0 is https://koji.fedoraproject.org/koji/taskinfo?taskID=94998622 , 20221207.n.0 is https://koji.fedoraproject.org/koji/taskinfo?taskID=95039040
3. Find the full log with information on what was actually pulled into the image. In this case it's in root.log from the Koji tasks: https://kojipkgs.fedoraproject.org//work/tasks/8622/94998622/root.log and https://kojipkgs.fedoraproject.org//work/tasks/9040/95039040/root.log
4. Poke through the log and find where there's info about what packages were pulled into the image build. In this case it's just under "running runtime-install.tmpl". For 20221206.n.0 we see: "794 packages selected". For 20221207.n.0, we see: "811 packages selected". So seems pretty clear that's the problem - we're now including more packages in the installer environment. Let's find out why.
5. Copy the file lists into a text editor and do a bit of processing on them to boil them down to just NVRs, however you like to do this. First I stripped out all lines with a timestamp (to get rid of the duplication, so only lines like 'DEBUG util.py:445:  (21/794) linux-firmware-20221109-144.fc38.noarch' remained. Then I used a regex to strip the leading "DEBUG util.py:445:  (NN/NNN) " bit. Then I sorted them.
6. Diff the cleaned-up lists so you can see what packages changed, including what was added. That gets us this diff:

@@ -7,7 +7,7 @@
@@ -15 +15 @@
@@ -24,2 +24,2 @@
@@ -190,10 +190,10 @@
@@ -202,0 +203 @@
@@ -204,0 +206 @@
@@ -208,0 +211,2 @@
@@ -217,0 +222 @@
@@ -224 +229 @@
@@ -260 +265 @@
@@ -272,4 +277,4 @@
@@ -291,0 +297 @@
@@ -298,0 +305 @@
@@ -334,0 +342 @@
@@ -337,0 +346,2 @@
@@ -361 +370,0 @@
@@ -376,0 +386 @@
@@ -389,0 +400 @@
@@ -427,0 +439 @@
@@ -443 +455 @@
@@ -450,0 +463 @@
@@ -453,0 +467 @@
@@ -501 +515 @@
@@ -521,7 +535,7 @@
@@ -671,0 +686 @@
@@ -709,0 +725 @@
@@ -716,0 +733 @@
@@ -747 +764 @@
@@ -764,2 +781,2 @@

There are version updates (cases where older version of a package is removed and newer is added), and pure additions (+ lines with no older version - line). The additions are all media-related:

gssdsp gstreamer1-plugins-bad-free gupnp gupnp-igd highway libaom libavif libdav1d libdvdnav libdvdread libjxl libnice librstp libva libvmaf rav1e-libs soundtouch svt-av1-libs

Ultimately the list of things installed in this environment comes from that runtime-install.tmpl file - https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-install.tmpl . That list hasn't changed lately, so we didn't explicitly add something. So that means the new packages are being pulled in as deps of something else in that set. So, we need to figure out which of the changed packages started pulling in a bunch of media files. There's various ways you could go about this; I do it by spidey sense, and my spidey sense first suspected libtracker-sparql, but that package's deps haven't changed between 3.4.1-1 and 3.4.2-1, so that wasn't it. Next suspect was webkit2gtk4.1, and that time my spidey sense was bang on: webkit2gtk4.1 2.39.2-6 adds two gstreamer-y deps compared to 2.38.2-1, libgsttranscoder-1.0.so.0()(64bit) and libgstallocators-1.0.so.0()(64bit). allocators is OK because that's provided by gstreamer1-plugins-base, which was already in 1206, but transcoder is bad:

[adamw@toolbox os-autoinst-distri-fedora (lowlevel %)]$ sudo dnf repoquery --whatprovides "libgsttranscoder-1.0.so.0()(64bit)"

so, the new version of webkit2gtk4.1 adds a hard dependency on gstreamer1-plugins-bad-free which it didn't have before (it had a Recommends: for it, but not a hard require), and that pulls gstreamer1-plugins-bad-free and all its dependencies into the installer environment, which is the change here (all those other things are dependencies of gstreamer1-plugins-bad-free).

Comment 3 Adam Williamson 2022-12-07 18:28:40 UTC
mcatanzaro says "Yes indeed, if you don't like it then we have to split up gstreamer1-plugins-bad-free...if you need to split things up, then wtay is the person to talk to", so CCing him.

To answer some obvious questions:

Q: Why is webkit2gtk4.1 in the installer environment at all?
A: Because yelp requires it, and the installer uses yelp to display its Help pages. We could look at using something else or somehow having a 'special' yelp build or dep chain, but the new anaconda web UI - https://bugzilla.redhat.com/show_bug.cgi?id=2114325 etc. - also requires webkitgtk directly, so it's probably not worth wasting effort on that unless we can *also* do it for the new web UI.

Q: Isn't the 700M limit for network install images kinda obsolete?
A: In a strict sense yeah it probably is - I doubt many people are literally burning the images to CD-Rs any more. But there is a genuine reason to care beyond physical media size limits: the size of the installer environment is one of the two key factors determining the minimum RAM you need to run an install (the other being how much RAM dnf uses during the package install process). The entire installer environment is loaded into RAM; it has to be, we have nowhere else we can depend on loading it to. So the bigger it is, the more RAM you need to install. So I don't really want to just bump this limit to 1G or, even worse, 2G or DVD size, because without this "oversize" mechanism kicking in, nobody will notice bloat; if we bump the limit to 1G, this stuff will pile up until the image size hits 1G, and then we'll have to try and fix a bunch of it at once. I already think 700M is too big, if anything - we should be able to fit an installer environment in less. F10's boot.iso was 130M. F16's boot.iso was 270M (I think we already had yelp by then). F25's was 477M. Even F30's was 598M, 105M smaller than now with this bug.

Comment 4 Adam Williamson 2022-12-09 20:30:53 UTC
https://github.com/weldr/lorax/pull/1291 would get the netinsts back under size, so marking as POST. That saves space by trimming out some more firmwares, though, it does nothing about the webkitgtk dep. I will file a new bug for that.

Comment 5 Adam Williamson 2022-12-09 20:35:10 UTC
Filed https://bugzilla.redhat.com/show_bug.cgi?id=2152229 .

Comment 6 Fedora QA Tools SIG 2022-12-10 11:02:02 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221210.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20221210.n.0.iso from compose Fedora-Rawhide-20221210.n.0 is 738643968 bytes, exceeding the maximum size 734003200.

Comment 7 Adam Williamson 2022-12-14 17:46:44 UTC
This is resolved by the recent dracut changes, 1214.n.0 image is undersize.

Comment 8 Fedora QA Tools SIG 2022-12-23 10:16:44 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221223.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20221223.n.0.iso from compose Fedora-Rawhide-20221223.n.0 is 743749632 bytes, exceeding the maximum size 734003200.

Comment 9 Fedora QA Tools SIG 2023-01-07 10:28:53 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230107.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230107.n.0.iso from compose Fedora-Rawhide-20230107.n.0 is 746444800 bytes, exceeding the maximum size 734003200.

Comment 10 Fedora QA Tools SIG 2023-01-11 11:01:07 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230111.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230111.n.0.iso from compose Fedora-Rawhide-20230111.n.0 is 746344448 bytes, exceeding the maximum size 734003200.

Comment 11 Ben Cotton 2023-01-12 17:06:29 UTC
Setting AcceptedBlocker as oversize bugs are considered automatic blockers.

Comment 12 Adam Williamson 2023-01-17 23:44:57 UTC
So, I forgot I actually did look into this last month, on the Server netinst ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2151497#c4
the findings there are definitely the same for Everything netinst and probably also the problem for Workstation live as well.

Comment 13 Fedora QA Tools SIG 2023-01-22 00:59:42 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230121.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230121.n.0.iso from compose Fedora-Rawhide-20230121.n.0 is 758843392 bytes, exceeding the maximum size 734003200.

Comment 14 Fedora QA Tools SIG 2023-01-26 10:08:51 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230126.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230126.n.0.iso from compose Fedora-Rawhide-20230126.n.0 is 754311168 bytes, exceeding the maximum size 734003200.

Comment 15 Fedora QA Tools SIG 2023-02-02 10:18:26 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230202.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230202.n.0.iso from compose Fedora-Rawhide-20230202.n.0 is 755763200 bytes, exceeding the maximum size 734003200.

Comment 16 Fedora QA Tools SIG 2023-02-05 10:03:43 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230205.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230205.n.0.iso from compose Fedora-Rawhide-20230205.n.0 is 757598208 bytes, exceeding the maximum size 734003200.

Comment 17 Ben Cotton 2023-02-07 15:11:34 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

Comment 18 Fedora QA Tools SIG 2023-02-08 09:51:00 UTC
Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230208.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230208.n.0.iso from compose Fedora-Rawhide-20230208.n.0 is 757626880 bytes, exceeding the maximum size 734003200.

Comment 19 Adam Williamson 2023-02-14 07:09:56 UTC
Today's is 718012416 , back under size again for now, thanks to the linux-firmware changes.

Note You need to log in before you can comment on or make changes to this bug.