Size grew 18M between 20221206.n.0 and 20221207.n.0 . This should not be the kexec-tools thing discussed in https://bugzilla.redhat.com/show_bug.cgi?id=2149246 , it must be something else.

Here's a quick guide to debugging this kinda problem:

1. Go to the compose logs for each compose and find the relevant log for the image - in this case, https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221207.n.0/logs/x86_64/buildinstall-Everything.x86_64.log and https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221206.n.0/logs/x86_64/buildinstall-Everything.x86_64.log
2. Depending on the image there'll either be a full log in the compose logs, or you'll only get a pointer to a Koji task. In this case it's the latter, so go to the Koji tasks: 20221206.n.0 is https://koji.fedoraproject.org/koji/taskinfo?taskID=94998622 , 20221207.n.0 is https://koji.fedoraproject.org/koji/taskinfo?taskID=95039040
3. Find the full log with information on what was actually pulled into the image. In this case it's in root.log from the Koji tasks: https://kojipkgs.fedoraproject.org//work/tasks/8622/94998622/root.log and https://kojipkgs.fedoraproject.org//work/tasks/9040/95039040/root.log
4. Poke through the log and find where there's info about what packages were pulled into the image build. In this case it's just under "running runtime-install.tmpl". For 20221206.n.0 we see: "794 packages selected". For 20221207.n.0, we see: "811 packages selected". So seems pretty clear that's the problem - we're now including more packages in the installer environment. Let's find out why.
5. Copy the file lists into a text editor and do a bit of processing on them to boil them down to just NVRs, however you like to do this. First I stripped out all lines with a timestamp (to get rid of the duplication, so only lines like 'DEBUG util.py:445:  (21/794) linux-firmware-20221109-144.fc38.noarch' remained. Then I used a regex to strip the leading "DEBUG util.py:445:  (NN/NNN) " bit. Then I sorted them.
6. Diff the cleaned-up lists so you can see what packages changed, including what was added. That gets us this diff:

@@ -7,7 +7,7 @@
-anaconda-38.10-1.fc38.x86_64
-anaconda-core-38.10-1.fc38.x86_64
-anaconda-dracut-38.10-1.fc38.x86_64
-anaconda-gui-38.10-1.fc38.x86_64
-anaconda-install-env-deps-38.10-1.fc38.x86_64
-anaconda-install-img-deps-38.10-1.fc38.x86_64
-anaconda-tui-38.10-1.fc38.x86_64
+anaconda-38.12-1.fc38.x86_64
+anaconda-core-38.12-1.fc38.x86_64
+anaconda-dracut-38.12-1.fc38.x86_64
+anaconda-gui-38.12-1.fc38.x86_64
+anaconda-install-env-deps-38.12-1.fc38.x86_64
+anaconda-install-img-deps-38.12-1.fc38.x86_64
+anaconda-tui-38.12-1.fc38.x86_64
@@ -15 +15 @@
-anaconda-widgets-38.10-1.fc38.x86_64
+anaconda-widgets-38.12-1.fc38.x86_64
@@ -24,2 +24,2 @@
-authselect-1.4.0-3.fc37.x86_64
-authselect-libs-1.4.0-3.fc37.x86_64
+authselect-1.4.2-1.fc38.x86_64
+authselect-libs-1.4.2-1.fc38.x86_64
@@ -190,10 +190,10 @@
-grub2-common-1:2.06-68.fc38.noarch
-grub2-efi-ia32-1:2.06-68.fc38.x86_64
-grub2-efi-ia32-cdboot-1:2.06-68.fc38.x86_64
-grub2-efi-x64-1:2.06-68.fc38.x86_64
-grub2-efi-x64-cdboot-1:2.06-68.fc38.x86_64
-grub2-pc-modules-1:2.06-68.fc38.noarch
-grub2-tools-1:2.06-68.fc38.x86_64
-grub2-tools-efi-1:2.06-68.fc38.x86_64
-grub2-tools-extra-1:2.06-68.fc38.x86_64
-grub2-tools-minimal-1:2.06-68.fc38.x86_64
+grub2-common-1:2.06-69.fc38.noarch
+grub2-efi-ia32-1:2.06-69.fc38.x86_64
+grub2-efi-ia32-cdboot-1:2.06-69.fc38.x86_64
+grub2-efi-x64-1:2.06-69.fc38.x86_64
+grub2-efi-x64-cdboot-1:2.06-69.fc38.x86_64
+grub2-pc-modules-1:2.06-69.fc38.noarch
+grub2-tools-1:2.06-69.fc38.x86_64
+grub2-tools-efi-1:2.06-69.fc38.x86_64
+grub2-tools-extra-1:2.06-69.fc38.x86_64
+grub2-tools-minimal-1:2.06-69.fc38.x86_64
@@ -202,0 +203 @@
+gssdp-1.4.0.1-3.fc37.x86_64
@@ -204,0 +206 @@
+gstreamer1-plugins-bad-free-1.20.4-2.fc38.x86_64
@@ -208,0 +211,2 @@
+gupnp-1.4.3-3.fc37.x86_64
+gupnp-igd-1.2.0-6.fc37.x86_64
@@ -217,0 +222 @@
+highway-1.0.1-2.fc38.x86_64
@@ -224 +229 @@
-ibus-libs-1.5.27-7.fc38.x86_64
+ibus-libs-1.5.27-8.fc38.x86_64
@@ -260 +265 @@
-javascriptcoregtk4.1-2.38.2-1.fc38.x86_64
+javascriptcoregtk4.1-2.39.2-6.fc38.x86_64
@@ -272,4 +277,4 @@
-kernel-6.1.0-0.rc8.58.fc38.x86_64
-kernel-core-6.1.0-0.rc8.58.fc38.x86_64
-kernel-modules-6.1.0-0.rc8.58.fc38.x86_64
-kernel-modules-extra-6.1.0-0.rc8.58.fc38.x86_64
+kernel-6.1.0-0.rc8.20221206gitbce9332220bd.59.fc38.x86_64
+kernel-core-6.1.0-0.rc8.20221206gitbce9332220bd.59.fc38.x86_64
+kernel-modules-6.1.0-0.rc8.20221206gitbce9332220bd.59.fc38.x86_64
+kernel-modules-extra-6.1.0-0.rc8.20221206gitbce9332220bd.59.fc38.x86_64
@@ -291,0 +297 @@
+libaom-3.5.0-2.fc38.x86_64
@@ -298,0 +305 @@
+libavif-0.11.1-4.fc38.x86_64
@@ -334,0 +342 @@
+libdav1d-1.0.0-2.fc37.x86_64
@@ -337,0 +346,2 @@
+libdvdnav-6.1.1-4.fc37.x86_64
+libdvdread-6.1.3-2.fc37.x86_64
@@ -361 +370,0 @@
-libglvnd-opengl-1:1.6.0-1.fc38.x86_64
@@ -376,0 +386 @@
+libjxl-1:0.7.0-3.fc38.x86_64
@@ -389,0 +400 @@
+libnice-0.1.19-2.fc37.x86_64
@@ -427,0 +439 @@
+libsrtp-2.3.0-9.fc37.x86_64
@@ -443 +455 @@
-libtracker-sparql-3.4.1-1.fc38.x86_64
+libtracker-sparql-3.4.2-1.fc38.x86_64
@@ -450,0 +463 @@
+libva-2.16.0-1.fc38.x86_64
@@ -453,0 +467 @@
+libvmaf-2.3.0-4.fc37.x86_64
@@ -501 +515 @@
-llvm-libs-15.0.4-2.fc38.x86_64
+llvm-libs-15.0.6-1.fc38.x86_64
@@ -521,7 +535,7 @@
-mesa-dri-drivers-22.3.0-1.fc38.x86_64
-mesa-filesystem-22.3.0-1.fc38.x86_64
-mesa-libEGL-22.3.0-1.fc38.x86_64
-mesa-libgbm-22.3.0-1.fc38.x86_64
-mesa-libGL-22.3.0-1.fc38.x86_64
-mesa-libglapi-22.3.0-1.fc38.x86_64
-mesa-libxatracker-22.3.0-1.fc38.x86_64
+mesa-dri-drivers-22.3.0-2.fc38.x86_64
+mesa-filesystem-22.3.0-2.fc38.x86_64
+mesa-libEGL-22.3.0-2.fc38.x86_64
+mesa-libgbm-22.3.0-2.fc38.x86_64
+mesa-libGL-22.3.0-2.fc38.x86_64
+mesa-libglapi-22.3.0-2.fc38.x86_64
+mesa-libxatracker-22.3.0-2.fc38.x86_64
@@ -671,0 +686 @@
+rav1e-libs-0.5.1-5.fc37.x86_64
@@ -709,0 +725 @@
+soundtouch-2.3.1-3.fc37.x86_64
@@ -716,0 +733 @@
+svt-av1-libs-1.3.0-1.fc38.x86_64
@@ -747 +764 @@
-webkit2gtk4.1-2.38.2-1.fc38.x86_64
+webkit2gtk4.1-2.39.2-6.fc38.x86_64
@@ -764,2 +781,2 @@
-xmlrpc-c-1.51.0-14.fc36.x86_64
-xmlrpc-c-client-1.51.0-14.fc36.x86_64
+xmlrpc-c-1.51.0-17.fc38.x86_64
+xmlrpc-c-client-1.51.0-17.fc38.x86_64

There are version updates (cases where older version of a package is removed and newer is added), and pure additions (+ lines with no older version - line). The additions are all media-related:

gssdsp gstreamer1-plugins-bad-free gupnp gupnp-igd highway libaom libavif libdav1d libdvdnav libdvdread libjxl libnice librstp libva libvmaf rav1e-libs soundtouch svt-av1-libs

Ultimately the list of things installed in this environment comes from that runtime-install.tmpl file - https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-install.tmpl . That list hasn't changed lately, so we didn't explicitly add something. So that means the new packages are being pulled in as deps of something else in that set. So, we need to figure out which of the changed packages started pulling in a bunch of media files. There's various ways you could go about this; I do it by spidey sense, and my spidey sense first suspected libtracker-sparql, but that package's deps haven't changed between 3.4.1-1 and 3.4.2-1, so that wasn't it. Next suspect was webkit2gtk4.1, and that time my spidey sense was bang on: webkit2gtk4.1 2.39.2-6 adds two gstreamer-y deps compared to 2.38.2-1, libgsttranscoder-1.0.so.0()(64bit) and libgstallocators-1.0.so.0()(64bit). allocators is OK because that's provided by gstreamer1-plugins-base, which was already in 1206, but transcoder is bad:

[adamw@toolbox os-autoinst-distri-fedora (lowlevel %)]$ sudo dnf repoquery --whatprovides "libgsttranscoder-1.0.so.0()(64bit)"
gstreamer1-plugins-bad-free-0:1.20.4-2.fc38.x86_64

so, the new version of webkit2gtk4.1 adds a hard dependency on gstreamer1-plugins-bad-free which it didn't have before (it had a Recommends: for it, but not a hard require), and that pulls gstreamer1-plugins-bad-free and all its dependencies into the installer environment, which is the change here (all those other things are dependencies of gstreamer1-plugins-bad-free).

mcatanzaro says "Yes indeed, if you don't like it then we have to split up gstreamer1-plugins-bad-free...if you need to split things up, then wtay is the person to talk to", so CCing him.

To answer some obvious questions:

Q: Why is webkit2gtk4.1 in the installer environment at all?
A: Because yelp requires it, and the installer uses yelp to display its Help pages. We could look at using something else or somehow having a 'special' yelp build or dep chain, but the new anaconda web UI - https://bugzilla.redhat.com/show_bug.cgi?id=2114325 etc. - also requires webkitgtk directly, so it's probably not worth wasting effort on that unless we can *also* do it for the new web UI.

Q: Isn't the 700M limit for network install images kinda obsolete?
A: In a strict sense yeah it probably is - I doubt many people are literally burning the images to CD-Rs any more. But there is a genuine reason to care beyond physical media size limits: the size of the installer environment is one of the two key factors determining the minimum RAM you need to run an install (the other being how much RAM dnf uses during the package install process). The entire installer environment is loaded into RAM; it has to be, we have nowhere else we can depend on loading it to. So the bigger it is, the more RAM you need to install. So I don't really want to just bump this limit to 1G or, even worse, 2G or DVD size, because without this "oversize" mechanism kicking in, nobody will notice bloat; if we bump the limit to 1G, this stuff will pile up until the image size hits 1G, and then we'll have to try and fix a bunch of it at once. I already think 700M is too big, if anything - we should be able to fit an installer environment in less. F10's boot.iso was 130M. F16's boot.iso was 270M (I think we already had yelp by then). F25's was 477M. Even F30's was 598M, 105M smaller than now with this bug.

https://github.com/weldr/lorax/pull/1291 would get the netinsts back under size, so marking as POST. That saves space by trimming out some more firmwares, though, it does nothing about the webkitgtk dep. I will file a new bug for that.

Filed https://bugzilla.redhat.com/show_bug.cgi?id=2152229 .

Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221210.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20221210.n.0.iso from compose Fedora-Rawhide-20221210.n.0 is 738643968 bytes, exceeding the maximum size 734003200.

This is resolved by the recent dracut changes, 1214.n.0 image is undersize.

Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20221223.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20221223.n.0.iso from compose Fedora-Rawhide-20221223.n.0 is 743749632 bytes, exceeding the maximum size 734003200.

Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230107.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230107.n.0.iso from compose Fedora-Rawhide-20230107.n.0 is 746444800 bytes, exceeding the maximum size 734003200.

Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230111.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230111.n.0.iso from compose Fedora-Rawhide-20230111.n.0 is 746344448 bytes, exceeding the maximum size 734003200.

Setting AcceptedBlocker as oversize bugs are considered automatic blockers.

So, I forgot I actually did look into this last month, on the Server netinst ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2151497#c4
the findings there are definitely the same for Everything netinst and probably also the problem for Workstation live as well.

Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230121.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230121.n.0.iso from compose Fedora-Rawhide-20230121.n.0 is 758843392 bytes, exceeding the maximum size 734003200.

Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230126.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230126.n.0.iso from compose Fedora-Rawhide-20230126.n.0 is 754311168 bytes, exceeding the maximum size 734003200.

Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230202.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230202.n.0.iso from compose Fedora-Rawhide-20230202.n.0 is 755763200 bytes, exceeding the maximum size 734003200.

Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230205.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230205.n.0.iso from compose Fedora-Rawhide-20230205.n.0 is 757598208 bytes, exceeding the maximum size 734003200.

This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle.
Changing version to 38.

Everything boot x86_64 image https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230208.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20230208.n.0.iso from compose Fedora-Rawhide-20230208.n.0 is 757626880 bytes, exceeding the maximum size 734003200.

Today's is 718012416 , back under size again for now, thanks to the linux-firmware changes.