We want to have Fedora CoreOS ship qemu-user-static by default but currently it is too large and pulls in too many dependencies. Could it be broken into qemu-user-static-arm64, qemu-user-static-amd65 ... And potentially eliminate the requirement on python. This would make it much smaller, and we could install just the arches that we want to support for multi-arch container builds.
https://github.com/coreos/fedora-coreos-tracker/issues/1088
Any comment on this?
I don't think it's worth reducing the number of architectures, but what part of it is using Python?
AIUI the proposal isn't to reduce the number of architectures, but to split them into multiple subpackages. The package as a whole is 150 MB, and most systems probably don't need e.g. qemu-m68k-static. /usr/bin/qemu-trace-stap-static is a Python script.
(In reply to Benjamin Gilbert from comment #4) > AIUI the proposal isn't to reduce the number of architectures, but to split > them into multiple subpackages. The package as a whole is 150 MB, and most > systems probably don't need e.g. qemu-m68k-static. > > /usr/bin/qemu-trace-stap-static is a Python script. That's a lot of subpackages: ngompa@Belldandy-Slimbook ~> ls -hal /usr/bin/qemu-* -rwxr-xr-x. 1 root root 5.9M Feb 10 16:12 /usr/bin/qemu-aarch64_be-static* -rwxr-xr-x. 1 root root 5.9M Feb 10 16:12 /usr/bin/qemu-aarch64-static* -rwxr-xr-x. 1 root root 3.4M Feb 10 16:12 /usr/bin/qemu-alpha-static* -rwxr-xr-x. 1 root root 4.6M Feb 10 16:12 /usr/bin/qemu-armeb-static* -rwxr-xr-x. 1 root root 4.6M Feb 10 16:12 /usr/bin/qemu-arm-static* -rwxr-xr-x. 1 root root 3.4M Feb 10 16:12 /usr/bin/qemu-cris-static* -rwxr-xr-x. 1 root root 5.5M Feb 10 16:12 /usr/bin/qemu-hexagon-static* -rwxr-xr-x. 1 root root 3.5M Feb 10 16:12 /usr/bin/qemu-hppa-static* -rwxr-xr-x. 1 root root 3.9M Feb 10 16:12 /usr/bin/qemu-i386-static* -rwxr-xr-x. 1 root root 3.7M Feb 10 16:12 /usr/bin/qemu-m68k-static* -rwxr-xr-x. 1 root root 3.5M Feb 10 16:12 /usr/bin/qemu-microblazeel-static* -rwxr-xr-x. 1 root root 3.5M Feb 10 16:12 /usr/bin/qemu-microblaze-static* -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mips64el-static* -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mips64-static* -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mipsel-static* -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mipsn32el-static* -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mipsn32-static* -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mips-static* -rwxr-xr-x. 1 root root 3.4M Feb 10 16:12 /usr/bin/qemu-nios2-static* -rwxr-xr-x. 1 root root 3.4M Feb 10 16:12 /usr/bin/qemu-or1k-static* -rwxr-xr-x. 1 root root 4.5M Feb 10 16:12 /usr/bin/qemu-ppc64le-static* -rwxr-xr-x. 1 root root 4.5M Feb 10 16:12 /usr/bin/qemu-ppc64-static* -rwxr-xr-x. 1 root root 4.4M Feb 10 16:12 /usr/bin/qemu-ppc-static* -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-riscv32-static* -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-riscv64-static* -rwxr-xr-x. 1 root root 3.8M Feb 10 16:12 /usr/bin/qemu-s390x-static* -rwxr-xr-x. 1 root root 3.5M Feb 10 16:12 /usr/bin/qemu-sh4eb-static* -rwxr-xr-x. 1 root root 3.5M Feb 10 16:12 /usr/bin/qemu-sh4-static* -rwxr-xr-x. 1 root root 3.6M Feb 10 16:12 /usr/bin/qemu-sparc32plus-static* -rwxr-xr-x. 1 root root 3.6M Feb 10 16:12 /usr/bin/qemu-sparc64-static* -rwxr-xr-x. 1 root root 3.5M Feb 10 16:12 /usr/bin/qemu-sparc-static* -rwxr-xr-x. 1 root root 5.5K Dec 14 15:42 /usr/bin/qemu-trace-stap-static* -rwxr-xr-x. 1 root root 3.9M Feb 10 16:12 /usr/bin/qemu-x86_64-static* -rwxr-xr-x. 1 root root 6.4M Feb 10 16:12 /usr/bin/qemu-xtensaeb-static* -rwxr-xr-x. 1 root root 6.4M Feb 10 16:12 /usr/bin/qemu-xtensa-static*
(In reply to Neal Gompa from comment #5) > (In reply to Benjamin Gilbert from comment #4) > > AIUI the proposal isn't to reduce the number of architectures, but to split > > them into multiple subpackages. The package as a whole is 150 MB, and most > > systems probably don't need e.g. qemu-m68k-static. > > That's a lot of subpackages: > > ngompa@Belldandy-Slimbook ~> ls -hal /usr/bin/qemu-* > -rwxr-xr-x. 1 root root 5.9M Feb 10 16:12 /usr/bin/qemu-aarch64_be-static* ...snip... That's the reason the system emulators are not fully split up one per target, but rather grouped into logically related archives. eg all the arm variants in the same package. Footprint is slightly larger but avoids a ridiculous number of RPMs. eg > -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mips64el-static* > -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mips64-static* > -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mipsel-static* > -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mipsn32el-static* > -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mipsn32-static* > -rwxr-xr-x. 1 root root 4.2M Feb 10 16:12 /usr/bin/qemu-mips-static* Could be in a 'qemu-user-mips' sub-RPM, instead of 6 separate sub-RPMs
(In reply to Benjamin Gilbert from comment #4) > /usr/bin/qemu-trace-stap-static is a Python script. This looks like a silly mistake In the RPM spec we have a loop: # Rename all QEMU user emulators to have a -static suffix for src in %{static_buildroot}%{_bindir}/qemu-*; do mv $src %{buildroot}%{_bindir}/$(basename $src)-static; done which was adding the 'static' suffix to user emulators, but this wildcard accidentally matches 'qemu-trace-stap'
I like Daniel Berrange's idea to package them by arch. I would even recommend a "fake" rpm package that installs them all. This would preserve the origin behavior and allow arches to be installed only.
Sure qumu-user-static would install all of the subpackages. But allow us to install qemu-user-x86-static and qemu-user-arm-static on Fedora CoreOS by default.
(In reply to Daniel Walsh from comment #9) > Sure qumu-user-static would install all of the subpackages. > > But allow us to install qemu-user-x86-static and qemu-user-arm-static on > Fedora CoreOS by default. You'd want more than that. You'd also want qemu-user-power-static and qemu-user-s390-static because containers exist for those types too.
Bogus python dep dropped in: commit d8c4df3d29c5354c28ea5fe51546adaa3ad0f7b2 Author: Daniel P. Berrangé <berrange> Date: Tue May 3 18:58:58 2022 +0100 Drop redundant qemu-trace-stap copy from qemu-user-static (rhbz#2061584) The static build of QEMU installs a copy of 'qemu-trace-stap' python script, which gets renamed to 'qemu-trace-stap-static' by an overly enthusiastic wildcard. This ends up adding a python dependency to the qemu-user-static RPM, which is unhelpful. Anyone who wants to trace QEMU user binaries with the stap helper can easily install qemu-common as desired. I've not triggered a full rawhide build just for that change, just checked it via a scratch-build https://koji.fedoraproject.org/koji/taskinfo?taskID=86593853 to validate the python dep is gone
(In reply to Neal Gompa from comment #10) > (In reply to Daniel Walsh from comment #9) > > Sure qumu-user-static would install all of the subpackages. > > > > But allow us to install qemu-user-x86-static and qemu-user-arm-static on > > Fedora CoreOS by default. > > You'd want more than that. You'd also want qemu-user-power-static and > qemu-user-s390-static because containers exist for those types too. With the bundling per arch, that would end up meaning you'd have 40 MB, down from 140 MB: $ du -h -c -s /usr/bin/qemu-*static | tail -1 139M total $ du -h -s -c /usr/bin/qemu-{i386,arm*,aarch64,s390x,x86_64,ppc*}-static | tail 3.9M /usr/bin/qemu-i386-static 4.4M /usr/bin/qemu-armeb-static 4.4M /usr/bin/qemu-arm-static 5.6M /usr/bin/qemu-aarch64-static 3.8M /usr/bin/qemu-s390x-static 3.9M /usr/bin/qemu-x86_64-static 4.4M /usr/bin/qemu-ppc64le-static 4.5M /usr/bin/qemu-ppc64-static 4.4M /usr/bin/qemu-ppc-static 39M total
(In reply to Daniel Berrangé from comment #11) > Bogus python dep dropped in: snip > I've not triggered a full rawhide build just for that change, just checked > it via a scratch-build > https://koji.fedoraproject.org/koji/taskinfo?taskID=86593853 to validate the > python dep is gone Oh, but ultimately that doesn't help you're needs, because 'qemu-user-static' still depends on 'qemu-common' and that will pull in python already. That dep on 'qemu-common' is arguably also somewhat bogus, or at least overkill, for qemu-user / qemu-user-static. I think the only thing in qemu-common that's actually relevant are the license file texts. Even the translations aren't needed as they're only used by the GTK frontend in system emulators
(In reply to Daniel Berrangé from comment #12) > $ du -h -s -c /usr/bin/qemu-{i386,arm*,aarch64,s390x,x86_64,ppc*}-static | > tail > 3.9M /usr/bin/qemu-i386-static > 4.4M /usr/bin/qemu-armeb-static > 4.4M /usr/bin/qemu-arm-static > 5.6M /usr/bin/qemu-aarch64-static > 3.8M /usr/bin/qemu-s390x-static > 3.9M /usr/bin/qemu-x86_64-static > 4.4M /usr/bin/qemu-ppc64le-static > 4.5M /usr/bin/qemu-ppc64-static > 4.4M /usr/bin/qemu-ppc-static > 39M total Nice.. I know Neal mentioned we would need s390x and ppc64le but I doubt we would actually add those in practice until some user demand existed for it. If we stated we only cared about x86_64 and aarch64 couldn't we get down to just qemu-x86_64-static and qemu-aarch64-static? (In reply to Daniel Berrangé from comment #13) > > Oh, but ultimately that doesn't help you're needs, because > 'qemu-user-static' still depends on 'qemu-common' and that will pull in > python already. > > That dep on 'qemu-common' is arguably also somewhat bogus, or at least > overkill, for qemu-user / qemu-user-static. I think the only thing in > qemu-common that's actually relevant are the license file texts. Even the > translations aren't needed as they're only used by the GTK frontend in > system emulators I know this is asking a lot, but is there a logical breakdown that could make sense for breaking out into yet another subpackage?
(In reply to Dusty Mabe from comment #14) > (In reply to Daniel Berrangé from comment #12) > > $ du -h -s -c /usr/bin/qemu-{i386,arm*,aarch64,s390x,x86_64,ppc*}-static | > > tail > > 3.9M /usr/bin/qemu-i386-static > > 4.4M /usr/bin/qemu-armeb-static > > 4.4M /usr/bin/qemu-arm-static > > 5.6M /usr/bin/qemu-aarch64-static > > 3.8M /usr/bin/qemu-s390x-static > > 3.9M /usr/bin/qemu-x86_64-static > > 4.4M /usr/bin/qemu-ppc64le-static > > 4.5M /usr/bin/qemu-ppc64-static > > 4.4M /usr/bin/qemu-ppc-static > > 39M total > > > Nice.. I know Neal mentioned we would need s390x and ppc64le but I doubt we > would actually add those in practice until some user demand existed for it. > If we stated we only cared about x86_64 and aarch64 couldn't we get down to > just qemu-x86_64-static and qemu-aarch64-static? > Since Fedora makes containers for those architectures, I would personally consider it problematic if you didn't ship them. And 40MB isn't going to hurt you. > > (In reply to Daniel Berrangé from comment #13) > > > > Oh, but ultimately that doesn't help you're needs, because > > 'qemu-user-static' still depends on 'qemu-common' and that will pull in > > python already. > > > > That dep on 'qemu-common' is arguably also somewhat bogus, or at least > > overkill, for qemu-user / qemu-user-static. I think the only thing in > > qemu-common that's actually relevant are the license file texts. Even the > > translations aren't needed as they're only used by the GTK frontend in > > system emulators > > I know this is asking a lot, but is there a logical breakdown that could > make sense for breaking out into yet another subpackage? There's already too many splits as it is... :(
FEDORA-2022-c93cfd49e8 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-c93cfd49e8
FEDORA-2022-c93cfd49e8 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.
Did you break up the qemu-user-static into subpackages?
I don't think so: https://src.fedoraproject.org/rpms/qemu/commits/rawhide
Sigh, stupid auto-closer. I merely removed the extra deps on python.
(In reply to Daniel Berrangé from comment #20) > Sigh, stupid auto-closer. I merely removed the extra deps on python. :) - nevertheless, thank you for doing that part ^^
If someone takes a stab at it and submits a PR I will help with review and any fine tuning to get it in. Most of the work should be grunt copy+paste stuff I think
Created attachment 1883168 [details] This patch breaks qemu-user-static into three packages. qemu-user-static package requires qemu-user-static-supported and qemu-user-static-extra. Not sure about the names for the two packages, but this is what I would envision for the split.
I very much don't like that kind of split because the notion of what is "supported" vs "extra" is very much in the eye of the beholder. If we're going to do any split, then it should mirror the way we split the system emulator packages, where we group by related arch.
Created attachment 1883363 [details] This patch breaks qemu-user-static into multiple arch packages. Daniel, I have broken out the packages by arch, PTAL.
(In reply to Daniel Walsh from comment #25) > Created attachment 1883363 [details] > This patch breaks qemu-user-static into multiple arch packages. > > Daniel, I have broken out the packages by arch, PTAL. The attachment is private.
Fixed, I don't know why this is defaulting to private.
Regardless of package management and "splitting" please bear in mind any and all corporate / federal restrictive environments in which downloading anything from beyond the trusted security enclave can prove rto be problematic. As a consequence this also means that you must be cognizant of additional trusted X.509 certificates in a "connected" environment. Reference my comment: https://github.com/containers/podman/issues/11458#issuecomment-1137858452
(In reply to Daniel Walsh from comment #27) > Fixed, I don't know why this is defaulting to private. You should check your Bugzilla account settings. They may still have legacy settings that default to "Red Hat Confidential"/private for tickets and comments by default. Most accounts shouldn't be affected by this anymore, but I'm aware that it was a problem at one point.
(In reply to Christian Polizzi from comment #28) > Regardless of package management and "splitting" please bear in mind any and > all corporate / federal restrictive environments in which downloading > anything from beyond the trusted security enclave can prove rto be > problematic. As a consequence this also means that you must be cognizant of > additional trusted X.509 certificates in a "connected" environment. > Reference my comment: > > https://github.com/containers/podman/issues/11458#issuecomment-1137858452 This comment is completely irrelevant in this context: * You already permit Fedora repositories through, so you must be able to do overlays. * This ticket is about doing extra package splitting so the FCOS team would be satisfied to ship qemu-user-static on the images rather than requiring overlays. * X.509 certificates are not used for Fedora content *at all* beyond standard CA certificates for TLS communication to Fedora MirrorManager. * Fedora CoreOS is not designed for regulated environments. There is no corporate/federal security concern for this request. Furthermore, this feature cannot be replicated in RHEL (which is what most of those environments use) because qemu-user is completely unavailable in RHEL. Thus foreign architecture support for multi-arch container builds in RHEL is functionally impossible in a reasonably supportable manner.
Perhaps you misunderstood. Good that you are working on packaging qemu-user-static into the FCOS image builds. Because without it, and if in a restrictive environment that requires an proxy and one that is SSL inspecting (e.g., a typical approach in such environments and generally considered to be an intentional SSL man-in-the-middle attack) then one does indeed need to provision into the podman VM that SSL inspection certificate chain as provided by this type of proxy. Without it, in a scenario such as this, installing the qemu-user-static package (and its dependencies) would be next to impossible. There are also even more restrictive environments in which one would not be able pull packages at all due to more stringent network ACLs and information security policies. RHEL is not even in scope for anything I stated. FCOS, podman and QEMU emulation of other CPU architectures. (In reply to Neal Gompa from comment #30) > * You already permit Fedora repositories through, so you must be able to do > overlays. > * This ticket is about doing extra package splitting so the FCOS team would > be satisfied to ship qemu-user-static on the images rather than requiring > overlays. > * X.509 certificates are not used for Fedora content *at all* beyond > standard CA certificates for TLS communication to Fedora MirrorManager. > * Fedora CoreOS is not designed for regulated environments. > > There is no corporate/federal security concern for this request. > > Furthermore, this feature cannot be replicated in RHEL (which is what most > of those environments use) because qemu-user is completely unavailable in > RHEL. Thus foreign architecture support for multi-arch container builds in > RHEL is functionally impossible in a reasonably supportable manner.
(In reply to Christian Polizzi from comment #31) > Perhaps you misunderstood. > > Good that you are working on packaging qemu-user-static into the FCOS image > builds. Because without it, and if in a restrictive environment that > requires an proxy and one that is SSL inspecting (e.g., a typical approach > in such environments and generally considered to be an intentional SSL > man-in-the-middle attack) then one does indeed need to provision into the > podman VM that SSL inspection certificate chain as provided by this type of > proxy. Without it, in a scenario such as this, installing the > qemu-user-static package (and its dependencies) would be next to impossible. > There are also even more restrictive environments in which one would not be > able pull packages at all due to more stringent network ACLs and information > security policies. As Neal already said, this is irrelevant to this bug. On this ticket we are *only* concerned with the packaging layout of QEMU user binaries. Please take any other discussions related to podman elsewhere.
(In reply to Daniel Walsh from comment #25) > Created attachment 1883363 [details] > This patch breaks qemu-user-static into multiple arch packages. > > Daniel, I have broken out the packages by arch, PTAL. I've not had time todo test scratch build, but the proposed spec changes look good to me now.
Hey Daniel, did you ever get a chance to look at this? Is there anything else I need to do to move this forward?
In rawhide now, I needed to add a few fixes on top since the generated binfmt file list is different depending on host arch. This bug is against f35. dwalsh do you want this for stable releases too?
I would love to get it in F36 if possible.
FEDORA-2022-7a6b58990b has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-7a6b58990b
FEDORA-2022-7a6b58990b has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-7a6b58990b` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-7a6b58990b See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-0142d562ca has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-0142d562ca` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-0142d562ca See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-0142d562ca has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.