Created attachment 1907794 [details] Patch I applied to the fedora package CentOS9/RHEL9 are built for the x86_64-v2 microarchitecture, which is not supported by the default qemu64 CPU used by qemu-x86_64-static. In particular, this means that one cannot use qemu together with binfmt to run these x86_64 binaries transparently on an aarch64 fedora install. The error message is "Fatal glibc error: CPU does not support x86-64-v2" This can be worked around by manually using a --cpu argument, for example 'qemu-x86_64-static --cpu max', possibly through a wrapper script if one wants to use this together with binfmt. However, there is one use case when this is not possible, which is when trying to use qemu-x86_64-static + binfmt through podman, the full report/investigation is in https://github.com/containers/podman/issues/15456 More details are in this comment https://github.com/containers/podman/issues/15456#issuecomment-1228210973: > When starting a container, [podman] enters a mount namespace corresponding to the filesystem of the container image, and then it's exec'ing /bin/bash or whatever the user asks it to run in the container. If it's a native binary, all is fine, if it's a non-native binary (x86_64 on aarch64 for example), then the kernel binfmt support kicks in. > qemu-user uses the 'F' flag in its binfmt configuration (see https://www.kernel.org/doc/html/v5.19/admin-guide/binfmt-misc.html ), which means /usr/bin/qemu-x86_64-static will be found regardless of the currently active mount namespace/changeroot/... However, this binary will be running in the mount namespace context, which means it will only be able to access the files available in the container image. This in particular means that we can't use a helper process which will then invoke qemu-x86_64-static with the right flags, as the latter is not going to be present in the container image filesystem. > https://www.kernel.org/doc/html/v5.19/admin-guide/binfmt-misc.html is also quite clear that it's not possible to pass additional arguments to the binfmt interpreter [without using a helper script/wrapper] > At this point, the best course of action will be to try to change the default CPU used by qemu-user-static-x86 so that it has x86_64-v2 support. I made a scratch QEMU build at https://koji.fedoraproject.org/koji/taskinfo?taskID=91274481 with the attached patch applied, and this allows `podman run -it --rm --arch x86_64 registry.access.redhat.com/ubi9 echo "hello world"` to be successfully run. This also impacts podman-machine/crc which are used to run containers on macOS M1 machines as they both run a f36 VM on macOS, and then run podman/its containers inside this VM. This means running x86_64 ubi9 images on the M1 is currently not working.
Since you're using TCG + -cpu max, please be aware of https://bugzilla.redhat.com/show_bug.cgi?id=2082806 https://gitlab.com/qemu-project/qemu/-/issues/1023 which causes intermittent crashes. You need to use `-cpu max,la57=off'
Hmm, I think there's a strong case to be made that all the qemu-XXXX userspace emulators should default to '-cpu max' (for targets where this exists). We only stick with qemu64 for machine emulators, since ABI is important for VMs. There are few, if any, guest ABI concerns for the userspace emulators, so switching to the best CPU is likely without significant downside. I'll raise this question upstream, as it makes more sense todo that everything, rather than distro-by-distro.
Fwiw, docker is already carrying a patch to default to '-cpu max' for their userspace emulators: https://github.com/tonistiigi/binfmt/blob/master/patches/cpu-max/0001-default-to-cpu-max-on-x86-and-arm.patch I agree it's better if this can be changed upstream.
https://lists.gnu.org/archive/html/qemu-devel/2022-08/msg04066.html(In reply to Daniel Berrangé from comment #2) > Hmm, I think there's a strong case to be made that all the qemu-XXXX > userspace emulators should default to '-cpu max' (for targets where this > exists). We only stick with qemu64 for machine emulators, since ABI is > important for VMs. There are few, if any, guest ABI concerns for the > userspace emulators, so switching to the best CPU is likely without > significant downside. Turns out most targets already pick the best CPU model, it was just x86 as the outlier. (In reply to Christophe Fergeau from comment #3) > Fwiw, docker is already carrying a patch to default to '-cpu max' for their > userspace emulators: > https://github.com/tonistiigi/binfmt/blob/master/patches/cpu-max/0001- > default-to-cpu-max-on-x86-and-arm.patch > I agree it's better if this can be changed upstream. The arm changes there are no-ops, since 'any' gets remapped to 'max' in arm #ifdef CONFIG_USER_ONLY /* For backwards compatibility usermode emulation allows "-cpu any", * which has the same semantics as "-cpu max". */ if (!strcmp(cpunamestr, "any")) { cpunamestr = "max"; } #endif so the only bit that really needed fixing is i386 and x86_64. I've posted https://lists.gnu.org/archive/html/qemu-devel/2022-08/msg04066.html
(In reply to Daniel Berrangé from comment #4) > I've posted https://lists.gnu.org/archive/html/qemu-devel/2022-08/msg04066.html Thanks! One comment regarding the commit log (not subscribed to qemu-devel, so adding it here), > There's no arg to podman that can be used to change the qemu-x86_64 args, and a non-root user of podman can not change binfmt rules without elevating privileges It's not possible to pass arguments to the binary you use through binfmt, https://www.kernel.org/doc/html/v5.19/admin-guide/binfmt-misc.html says an intermediate wrapper is needed to add arguments "If you want to pass special arguments to your interpreter, you can write a wrapper script for it". Such an intermediate wrapper is not an option for podman: when the helper is running, it is already restricted to a mount namespace (with the container filesystem) where qemu-x86_64-static won't be available.
I don't think the upstream patch was ever merged? Should this be added to the fedora packages for now?
The patches are sent for merge now, so we should be good to follow in Fedora at this point https://lists.gnu.org/archive/html/qemu-devel/2022-09/msg04787.html
Created attachment 1926383 [details] Patch for the rawhide package This adds the patch which went upstream to the rawhide package.
FEDORA-2023-c8a60f6f80 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2023-c8a60f6f80
FEDORA-2023-c8a60f6f80 has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-c8a60f6f80` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-c8a60f6f80 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-c8a60f6f80 has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.
Reopening as the f37 package will also need the patch. f36 works with qemu-6.2.0-17.fc36, f38 works because qemu 7.2.0 upstream has the patch. f37 still has this bug.
I created https://src.fedoraproject.org/rpms/qemu/pull-request/30 for this.
FEDORA-2023-cf93567517 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-cf93567517
FEDORA-2023-cf93567517 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-cf93567517` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-cf93567517 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-cf93567517 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.