Bug 1470397

What is the real world problem you're seeing. Dnf can happily resolve the few deps that are needed, so I'm not really finding the idea of dropping them appealing

In the example I gave you, we are building a root that has only one x86 package (qemu-user-static) and all other packages are armv7hl.

The real world problem is conflicts with cross-architecture roots. qemu-user-static doesn't *really* have any dependencies. But the ones it has, pull in a bunch of packages. But these packages conflict with the same packages from the cross-architecture.

Here is more backdrop:
https://npmccallum.gitlab.io/post/cross-architecture-roots-with-dnf/

We also have a related problem, which is that qemu-user-binfmt and qemu-user-static both install *different* versions of the binfmt conf files. So the example I gave in the blog doesn't actually work (I just discovered) because the host and container have different binaries. We should split apart the binfmt conf files into a separate package and have qemu-user and qemu-user-static have the same binary names and conflict. This way the host can install qemu-user-binfmt and qemu-user while the container can install just qemu-user-static.

Specifically, the problem is this (derived from the previous example):

$ sudo rpm -ivh --root=/tmp/F25ARM qemu-user-static-*.x86_64.rpm
warning: qemu-user-static-2.7.1-6.fc25.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fdb19c98: NOKEY
error: Failed dependencies:
	qemu-common = 2:2.7.1-6.fc25 is needed by qemu-user-static-2:2.7.1-6.fc25.x86_64
	systemd-units is needed by qemu-user-static-2:2.7.1-6.fc25.x86_64


First, we should have a qemu-user-binfmt package (as we currently do). This should depend on either qemu-user or qemu-user-static (it currently depends only on qemu-user). It would be best if we could figure out a way to make dnf prefer qemu-user.

Second, qemu-user-static should have its binaries renamed to remove the -static suffix. It should conflict with qemu-user.

Third, qemu-user-static should drop all its bundled binfmt .conf files.

Fourth, qemu-user-static should drop its qemu-common dependency. I don't believe qemu-common is needed at all.

Users who do "dnf upgrade" and have qemu-user installed should gain qemu-user-binfmt.

After more research, I think I have a simpler solution.

qemu-user-static uses the 'F' flag in its binfmt, but qemu-user-binfmt doesn't. We need to add the 'F' binfmt flag to qemu-user-binfmt and then we can remove qemu-user-static altogether. It isn't needed in the chroot at all. This will dramatically simplify qemu.spec.

I also don't see any reason to keep the split between qemu-user and qemu-user-binfmt. So (I think) these can be consolidated.

Here is my first stab at this simplification:

https://koji.fedoraproject.org/koji/taskinfo?taskID=20507784

(I'm posting it here for public record. It is building now...)

Here's a successful build:

https://koji.fedoraproject.org/koji/taskinfo?taskID=20511080

If I've done everything correctly, upgrades to this package should replace both qemu-user-static and qemu-user-binfmt with qemu-user. Cross-architecture roots should work fine without any modifications (qemu-user does NOT need to be installed).

BTW, qemu builds are broken on rawhide. But that isn't my fault.

Created attachment 1298074 [details]
Patch implementing the proposed updates

The attached patch implements my proposed updates.

However, qemu-common could still be cleaned up. The qemu-user subpackage depends on it but doesn't actually require anything from it. Nevertheless, this dependency bloat no longer blocks me. So I have not done anything about it.

Daniel, you should have everything needed to review these packages changes. Do you know when you'll be able to review them?

Created attachment 1300545 [details]
Patch implementing the proposed updates

I've rebased the patch given Daniel's new commit to rawhide today.

I built a new QEMU version for Fedora 26 with this patch applied, but AFAICT, it does not work.

I start off with  qemu-user-static installed on my x86_64 host, so i have that setup for binfmt:

# cd /proc/sys/fs/binfmt_misc
# cat qemu-arm
enabled
interpreter /usr/bin/qemu-arm-static
flags: F
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff000000000000000000feffffff


With this, I can chroot into an ARM tree and run commands:

# chroot  /home/berrange/arm-test/vroot /bin/busybox ls
bin   etc   init  lib   root  sbin  tmp   usr


Now, I unregister the binfmt handlers and install the QEMU build with this patch applied

# for i in qemu-* ; do echo -1 > $i ; done

# rpm -U /home/berrange/src/fedora/qemu/x86_64/qemu*-2.9.0-7.fc26.x86_64.rpm


Check that qemu-user-static got replaced by qemu-user:

# rpm -q qemu-user-static
package qemu-user-static is not installed
# rpm -q qemu-user
qemu-user-2.9.0-7.fc26.x86_64



See the binfmt now points to 'qemu-arm' with the "F" flag present:

# cat qemu-arm
enabled
interpreter /usr/bin/qemu-arm
flags: F
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff000000000000000000feffffff


Attempting to run commands in the ARM chroot now fails:

# chroot  /home/berrange/arm-test/vroot /bin/busybox ls
chroot: failed to run command ‘/bin/busybox’: No such file or directory

I suspect this is because of lazy linking (I haven't been able to confirm this due to network issues this morning). We can do LDFLAGS="-z now", though I'm not sure we want that for the entirety of qemu. Do we want to pursue retiring qemu-user-static in favor of non-lazy linking? Or should we just retain qemu-user-static?

QEMU is already linking using  -z,now becaue that's passed in from the RPM specs

eg RPM is calling configure with

 --extra-ldflags="-Wl,--build-id -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -pie -Wl,-z,relro -Wl,-z,now"

and i see that passed to the qemu-arm binary

c++ ....snip... -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g  -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g  -Wl,--build-id -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -pie -Wl,-z,relro -Wl,-z,now  -o qemu-arm ...snip...

From what I can see the original kernel patch[0] only works with static binaries. This is because while the executable file is loaded during configuration time, the loading of libraries is delayed until execution time - which obviously can't work.

I'm looking into our options. I'm hoping this is as easy as moving search_binary_handler() into bm_register_write().

In the meantime, I think we can merge qemu-user-binfmt back into qemu-user. The initial package split was never done correctly (qemu-user still depends on systemd and restarts the binfmt configuration even though it doesn't have binfmt config). Further, there are no packages which depend on qemu-user-binfmt. So, AFAICS, we should just ship the binfmts as part of qemu-user.

Longer term, we can retire qemu-user-static once the kernel supports dynamic binaries with the F flag.

[0]: https://github.com/torvalds/linux/commit/948b701a607f123df92ed29084413e5dd8cda2ed

The point of having qemu-user-binfmt separately is that it allows installation of both qemu-user and qemu-user-static at the same time. This is useful because you may have apps / scripts written to assume existance of the long term existing qemu-user package, but none the less need to have qemu-user-static for chroots. This mirrors the way Debian has split their qemu user packages.

The problem you mention with %post scripts is simply a bug we can easily fix.

Another option would be consolidating all the packages and simply building the qemu userspace emulation as static until we have a kernel fix. Once the kernel is fixed, then we can internally switch to dynamic linking. While this has the disadvantage of having static binaries, it has the advantage of exposing the package name changes earlier rather than later.

I'm happy with either option as a first step. Please let me know which you prefer.

I would just keep both variants, and once there's no more need for the static one, then we retire that variant.

I lean this direction as well. Daniel, do you mind if I use  my proven packager abilities to fix the aforementioned bug with the %post scripts and dependencies?

Then we can leave this bug open for a rework later.

I don't want to see us get rid of the dynamic linked binaries - users who have no need for the static linked version are better served by having the dynamic linked one installed, as they pick up security updates to any linked libraries immediately.

Feel free to fix the %post script mistake though.

I have fixed the %post scripts and dependencies in rawhide. It is building now. I spoke with crobinso about backporting to F26, which he'll do next week.

https://koji.fedoraproject.org/koji/buildinfo?buildID=920701

Once all this lands, our main blocker is kernel support for non-static binaries. If we can make that happen, we can consolidate the no-longer-necessary subpackages.

This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

Sounds like there isn't much to do here any more until there's kernel support for dynamic + binfmt 'F' ? Is there a bug for that?

Note, in rawhide I switched binfmt generation to use qemu's qemu-binfmt-conf.sh script which basically does all the same generation we were doing in the spec file. the mask values it uses are more restrictive but I think the net result is the same. Please be on the lookout for regressions

I think we can close this and reopen or file a new bug when there's more to do