Bug 2352679 (F42Serverbootaarch64Oversize)

*** Bug 2351459 has been marked as a duplicate of this bug. ***

The dupe was already accepted as a blocker and waived to Final.

Server boot aarch64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-42-20250319.n.0/compose/Server/aarch64/iso/Fedora-Server-netinst-aarch64-42-20250319.n.0.iso from compose Fedora-42-20250319.n.0 is 1029435392 bytes, exceeding the maximum size 1000000000.

Server boot aarch64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-42-20250323.n.0/compose/Server/aarch64/iso/Fedora-Server-netinst-aarch64-42-20250323.n.0.iso from compose Fedora-42-20250323.n.0 is 1029795840 bytes, exceeding the maximum size 1000000000.

Notified the Server SIG here:
https://pagure.io/fedora-server/issue/158

Huh, so this is interesting.

F41 aarch64 netinst install.img: 787386368 bytes
F42 aarch64 netinst install.img: 831614976 bytes

but, if you mount each of those images and do a du on them:

2.0G	f41
1.9G	temp4

uncompressed, the contents of the F42 image appear to be *smaller* than the F41 one. But compressed, F42 is 50MB larger. I'm not sure why.

Regardless, as expected, the F42 image has some linux-firmware bloat:

316M	f41/usr/lib/firmware/
357M	f42/usr/lib/firmware/

so we've got 41M more of firmware to deal with. We also have an extra 12MB of kernel modules in /lib/modules:

136M	f41/lib/modules
148M	f42/lib/modules

There isn't a lot we can do about either of those, I don't think.

Possibly the difference in compressed size is just because firmware and kernel modules compress less well than whatever we lost to save some uncompressed space. But it's also possible we changed the compression approach somehow. Neal, do you know anything about this?

guh, that '1.9G   temp4' should say '1.9G    f42', obviously.

(In reply to Adam Williamson from comment #6)
>
> 
> Possibly the difference in compressed size is just because firmware and
> kernel modules compress less well than whatever we lost to save some
> uncompressed space. But it's also possible we changed the compression
> approach somehow. Neal, do you know anything about this?

Since install media is produced by Lorax, I'd guess we'd want to look at the runroot task logs to see whether we've changed the options passed in for what type of compression is used.

Lorax defaults don't seem to have changed yet in upstream, and unfortunately since we don't have a dedicated lorax task for these, it's hard for me to find out how they were built since runroot tasks are not captured in a package space like kiwi build tasks are.

oh, it's easy enough to just find them from compose logs, that's how I always do it. https://kojipkgs.fedoraproject.org/compose/branched/Fedora-42-20250324.n.0/logs/aarch64/buildinstall-Server-logs/

Looks like there have been no changes. It's squashfs+xz still.

> There isn't a lot we can do about either of those, I don't think.

Do you have a list of the firmware packages that are being pulled in? I know lorax does some level of trimming but I don't remember where/how that's done.

> Possibly the difference in compressed size is just because firmware and
> kernel modules compress less well than whatever we lost to save some
> But it's also possible we changed the compression approach somehow.

Neither kernel nor linux-firmware has changed the compression details at least.

> Do you have a list of the firmware packages that are being pulled in? I know lorax does some level of trimming but I don't remember where/how that's done.

you can find it in the same logs I pointed Neal at, specifically https://kojipkgs.fedoraproject.org/compose/branched/Fedora-42-20250324.n.0/logs/aarch64/buildinstall-Server-logs/pylorax.log . This line shows the instruction:

template line 7: installpkg --optional *-firmware --except alsa* --except midisport-firmware --except crystalhd-firmware --except ivtv-firmware --except cx18-firmware --except iscan-firmware --except uhd-firmware --except lulzbot-marlin-firmware --except gnome-firmware --except sigrok-firmware --except liquidio-firmware --except netronome-firmware --except mrvlprestera-firmware --except mlxsw_spectrum-firmware --except hackrf-firmware --except python-virt-firmware --except python3-virt-firmware --except iwl3945-firmware --except iwl4965-firmware --except iwl100-firmware --except iwl105-firmware --except iwl135-firmware --except iwl1000-firmware --except iwl2000-firmware --except iwl2030-firmware --except iwl5000-firmware --except iwl5150-firmware --except iwl6000-firmware --except iwl6000g2a-firmware --except iwl6000g2b-firmware --except iwl6050-firmware --except iwl3160-firmware --except iwl7260-firmware --except iwlax2xx-firmware --except libertas-sd8686-firmware --except libertas-sd8787-firmware --except libertas-usb8388-firmware --except libertas-usb8388-olpc-firmware --except qcom-firmware

that is, "install all packages that match *-firmware except (long list of specific exceptions). The next line shows the immediate interpretation of that:

installpkg: *-firmware expands to amd-gpu-firmware-20250311-1.fc42.noarch,amd-ucode-firmware-20250311-1.fc42.noarch,atheros-firmware-20250311-1.fc42.noarch,atmel-firmware-1.3-34.fc42.noarch,bcm2711-firmware-20250224-2.4cb13e0.fc42.aarch64,bcm2835-firmware-20250224-2.4cb13e0.fc42.aarch64,bcm283x-firmware-20250224-2.4cb13e0.fc42.aarch64,brcmfmac-firmware-20250311-1.fc42.noarch,cirrus-audio-firmware-20250311-1.fc42.noarch,crust-firmware-0.6-4.fc42.noarch,dvb-firmware-20250311-1.fc42.noarch,intel-audio-firmware-20250311-1.fc42.noarch,intel-gpu-firmware-20250311-1.fc42.noarch,intel-vsc-firmware-20250311-1.fc42.noarch,iwlegacy-firmware-20250311-1.fc42.noarch,iwlwifi-dvm-firmware-20250311-1.fc42.noarch,iwlwifi-mvm-firmware-20250311-1.fc42.noarch,libertas-firmware-20250311-1.fc42.noarch,linux-firmware-20250311-1.fc42.noarch,mt7xxx-firmware-20250311-1.fc42.noarch,nvidia-gpu-firmware-20250311-1.fc42.noarch,nxpwireless-firmware-20250311-1.fc42.noarch,qed-firmware-20250311-1.fc42.noarch,realtek-firmware-20250311-1.fc42.noarch,tiwilink-firmware-20250311-1.fc42.noarch,zd1211-firmware-1.5-18.fc42.noarch

and you can double-check what got installed by looking for lines starting "Install " that have the string "firmware" in them, I guess:

[adamw@toolbx tmp]$ grep "^Install.*firmware" pylorax.log 
Install linux-firmware-whence-20250311-1.fc42.noarch
Install atheros-firmware-20250311-1.fc42.noarch
Install linux-firmware-20250311-1.fc42.noarch
Install bcm2711-firmware-20250224-2.4cb13e0.fc42.aarch64
Install bcm2835-firmware-20250224-2.4cb13e0.fc42.aarch64
Install bcm283x-firmware-20250224-2.4cb13e0.fc42.aarch64
Install qcom-firmware-20250311-1.fc42.noarch
Install amd-gpu-firmware-20250311-1.fc42.noarch
Install amd-ucode-firmware-20250311-1.fc42.noarch
Install brcmfmac-firmware-20250311-1.fc42.noarch
Install cirrus-audio-firmware-20250311-1.fc42.noarch
Install dvb-firmware-20250311-1.fc42.noarch
Install intel-audio-firmware-20250311-1.fc42.noarch
Install intel-gpu-firmware-20250311-1.fc42.noarch
Install intel-vsc-firmware-20250311-1.fc42.noarch
Install iwlegacy-firmware-20250311-1.fc42.noarch
Install iwlwifi-dvm-firmware-20250311-1.fc42.noarch
Install iwlwifi-mvm-firmware-20250311-1.fc42.noarch
Install libertas-firmware-20250311-1.fc42.noarch
Install mt7xxx-firmware-20250311-1.fc42.noarch
Install nvidia-gpu-firmware-20250311-1.fc42.noarch
Install nxpwireless-firmware-20250311-1.fc42.noarch
Install qed-firmware-20250311-1.fc42.noarch
Install realtek-firmware-20250311-1.fc42.noarch
Install tiwilink-firmware-20250311-1.fc42.noarch
Install zd1211-firmware-1.5-18.fc42.noarch
Install crust-firmware-0.6-4.fc42.noarch
Install atmel-firmware-1.3-34.fc42.noarch

> Install linux-firmware-whence-20250311-1.fc42.noarch
> Install atheros-firmware-20250311-1.fc42.noarch
> Install linux-firmware-20250311-1.fc42.noarch
> Install bcm2711-firmware-20250224-2.4cb13e0.fc42.aarch64
> Install bcm2835-firmware-20250224-2.4cb13e0.fc42.aarch64
> Install bcm283x-firmware-20250224-2.4cb13e0.fc42.aarch64
> Install qcom-firmware-20250311-1.fc42.noarch
> Install amd-gpu-firmware-20250311-1.fc42.noarch
> Install brcmfmac-firmware-20250311-1.fc42.noarch
> Install iwlwifi-mvm-firmware-20250311-1.fc42.noarch
> Install mt7xxx-firmware-20250311-1.fc42.noarch
> Install nvidia-gpu-firmware-20250311-1.fc42.noarch
> Install realtek-firmware-20250311-1.fc42.noarch

This shouldn't be in netinst/standard install (it's actually wrapped into U-Boot firmwares):
> Install crust-firmware-0.6-4.fc42.noarch

These shouldn't be installed by default (on any arch) as they're mostly legacy HW and isn't generally needed for install:
> Install dvb-firmware-20250311-1.fc42.noarch
> Install atmel-firmware-1.3-34.fc42.noarch
> Install tiwilink-firmware-20250311-1.fc42.noarch
> Install zd1211-firmware-1.5-18.fc42.noarch
> Install libertas-firmware-20250311-1.fc42.noarch
> Install qed-firmware-20250311-1.fc42.noarch

These are x86 only (there's an argument intel-gpu might one day be wanted on aarch64):
> Install intel-vsc-firmware-20250311-1.fc42.noarch
> Install amd-ucode-firmware-20250311-1.fc42.noarch
> Install cirrus-audio-firmware-20250311-1.fc42.noarch
> Install intel-audio-firmware-20250311-1.fc42.noarch
> Install intel-gpu-firmware-20250311-1.fc42.noarch

It's possible these maybe used but they're so old I've never seen any used on aarch64:
> Install iwlegacy-firmware-20250311-1.fc42.noarch
> Install iwlwifi-dvm-firmware-20250311-1.fc42.noarch

> These shouldn't be installed by default (on any arch) as they're mostly
> legacy HW and isn't generally needed for install:

To clarfiy, it shouldn't be in initrd/netinst, likely shouldn't be installed by default (and I thought we'd stopped installing a bunch of these by default in comps).

Last time I asked I think someone said Intel GPU on aarch64 CPU is at least in theory possible?

We install wifi firmwares in case you need to use wifi during install. that's atmel-firmware, tiwilink-firmware, zd1211-firmware, libertas-firmware. If we're *very* sure you couldn't possibly have one of those on an aarch64 system I guess we could drop it (though at least some of them exist as USB sticks so it seems like we couldn't drop those).

qed-firmware says it covers "iSCSI, iWARP, FCoE and ethernet" - iSCSI, FCoE and ethernet are things you might need in the installer.

I agree we can probably ditch dvb-firmware (maybe on all arches, though I wonder why we didn't do that one before? It seems so obvious I wonder if there's a gotcha), intel-vsc-firmware, amd-ucode-firmware, cirrus-audio-firmware and intel-audio-firmware...let's CC bcl and see what he thinks.

(In reply to Adam Williamson from comment #15)
> Last time I asked I think someone said Intel GPU on aarch64 CPU is at least
> in theory possible?

Correct, it's in theory possible, I did mention that above if you look.

> We install wifi firmwares in case you need to use wifi during install.
> that's atmel-firmware, tiwilink-firmware, zd1211-firmware,
> libertas-firmware. If we're *very* sure you couldn't possibly have one of
> those on an aarch64 system I guess we could drop it (though at least some of
> them exist as USB sticks so it seems like we couldn't drop those).

Yes, again in theory they could exist, in reality the vast majority of them barely exist on *any* 64 bit platforms, they're largely 32 bit platforms. In the case they *may* exist they're more than likely to be using a raw image not the installer. We can always update if in the extremely unlikely hood we get a bug report. Check what we do on the comps, I don't think most of them are even installed by default anymore and I've had zero bug reports about them being missing on installs against linux-firmware RHBZ.

> qed-firmware says it covers "iSCSI, iWARP, FCoE and ethernet" - iSCSI, FCoE
> and ethernet are things you might need in the installer.

Correct, but on CNA adapters (which is what this is) they need further configuring before they're useful, in enterprise platforms they usually have a "management interface" which is a low end 1G NIC (intel/broadcom/etc) which is used for network install and then when the OS is installed and running you then use tools to configure the CNA on the host before then being able to attach storage.

> I agree we can probably ditch dvb-firmware (maybe on all arches, though I
> wonder why we didn't do that one before?

Probably because it's small, I don't think we even enable dvb adapters on !x86 platforms.

> intel-vsc-firmware, amd-ucode-firmware, cirrus-audio-firmware and intel-audio-firmware

They're all x86 only.

Ultimately I don't care what you do, I am just providing information to assist you in making a decision.

Server boot aarch64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-42-20250326.n.0/compose/Server/aarch64/iso/Fedora-Server-netinst-aarch64-42-20250326.n.0.iso from compose Fedora-42-20250326.n.0 is 1029879808 bytes, exceeding the maximum size 1000000000.

Thanks for the info Peter. I don't really have a good idea of what's going to break if we drop some of these so all I'd say is remove things in rawhide only.

As for the compressed size, firmware can be the cause since most of it is ARM code which the bcj x86 filter does't help with. At one time I tried playing with mixing the compression options to reduce it, but never made enough progress for it to be worthwhile.

I can test a change to drop crust-firmware , dvb-firmware , intel-vsc-firmware, amd-ucode-firmware, cirrus-audio-firmware and intel-audio-firmware at least...those really do seem pretty safe. if those get us under the limit it might be worth it?

Sent https://github.com/weldr/lorax/pull/1463 against master branch for now, I'll test an image build with it soon I hope.

So I tested that with openQA. Without that patch I get an image of 1020372992 bytes, exactly the same size as today's official compose. With the patch I get an image of 1006499840 bytes. so close!

Server boot aarch64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-42-20250329.n.0/compose/Server/aarch64/iso/Fedora-Server-netinst-aarch64-42-20250329.n.0.iso from compose Fedora-42-20250329.n.0 is 1029871616 bytes, exceeding the maximum size 1000000000.

Server boot aarch64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-42-20250402.n.0/compose/Server/aarch64/iso/Fedora-Server-netinst-aarch64-42-20250402.n.0.iso from compose Fedora-42-20250402.n.0 is 1029924864 bytes, exceeding the maximum size 1000000000.

We talked this over at the last couple of Server meetings, and I touched based with Adam during the last blocker review meeting. I have submitted a PR to increase the blocking size from 1 GB to 1.2 GB. 

PR:  https://pagure.io/fedora-pgm/pgm_docs/pull-request/83
Meeting minutes where I took the action item:  https://meetbot.fedoraproject.org/meeting_matrix_fedoraproject-org/2025-04-02/fedora-server.2025-04-02-15.00.html

Please let me know if further action or information is required. Thank you all for looking at this.

Note today's nightly, built with lorax with my recent firmware trim PRs, is 1060418948 bytes - much closer to the old target. We *might* be able to find another 60MB to trim somewhere.

The limit is now 1.2G , so this is fine, current images are well below that size.

Server boot aarch64 image https://kojipkgs.fedoraproject.org/compose/branched/Fedora-42-20250405.n.1/compose/Server/aarch64/iso/Fedora-Server-netinst-aarch64-42-20250405.n.1.iso from compose Fedora-42-20250405.n.1 is 1016061952 bytes, exceeding the maximum size 1000000000.

grr, I updated the metadata, but obviously got it wrong somehow. go away silly bot