Bug 1995353

Summary: qemu: Request for EPEL-8 build
Product: [Fedora] Fedora EPEL Reporter: Ben Kircher <bkircher>
Component: qemuAssignee: Lubomir Rintel <lkundrak>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: epel8CC: berrange, carl, cfergeau, crobinso, davide, hello, lkundrak, ngompa13, pbonzini, philmd, quent.haas, rjones, tdawson, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2125333    
Bug Blocks: 1914423, 2203445    

Description Ben Kircher 2021-08-18 21:53:18 UTC
I'd like to officially request an epel8 branch for qemu.

I see that there is already some preliminary (and hard) work from Cole Robinson. I'd be happy to help create and maintain such a branch. My FAS username is bkircher

Comment 1 Ben Kircher 2021-08-18 22:00:10 UTC
I'm interested to see a version of epel8 based on 6.0.0 (commit 6bb4bb5).

I would be also willing to maintain missing packages in epel8 that would be needed for qemu runtime or build, e.g., capstone or so.

ANy thoughts?

Comment 2 Ben Kircher 2021-08-20 09:41:23 UTC
Just ran a scratch build of 6bb4bb5. Looks good.

Comment 3 Richard W.M. Jones 2021-08-20 10:11:29 UTC
Just wondering - is there any reason not to use the qemu-kvm package
which is in RHEL / CentOS?  (Note you may need to enable a module
to get a recent version).

For instance, is there some specific feature you need?

Comment 4 Ben Kircher 2021-08-20 11:48:44 UTC
> is there any reason not to use the qemu-kvm package
which is in RHEL / CentOS?

Yes. To be honest, they are just too old and too different from upstream for us.

    $ dnf module info virt|grep -P 'qemu-kvm-\d'
                 : qemu-kvm-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.src
                 : qemu-kvm-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64
                 : qemu-kvm-15:4.2.0-51.module_el8.5.0+821+97472045.src
                 : qemu-kvm-15:4.2.0-51.module_el8.5.0+821+97472045.x86_64
                 : qemu-kvm-15:4.2.0-52.module_el8.5.0+853+a4d5519d.src
                 : qemu-kvm-15:4.2.0-52.module_el8.5.0+853+a4d5519d.x86_64

We use a qemu and libvirt that is really close to upstream. It has the same versioned machine types, for instance. We also closely look at what the upstream community is doing and whenever enough interesting features or improvements accumulate, we build a new package, test it, and then we interact with upstream to get issues resolved. Having the "same" qemu as upstream has been fruitful to us.

Another point I see in a RHEL/CentOS provided qemu-kvm is, that it leaves enough room for a community provided full-fledged "vanilla" qemu package, since it wouldn't clash (at least the package name). Although, this last argument might be reverse psychology ;-)

Does the answer make sense?

Comment 5 Richard W.M. Jones 2021-08-20 12:08:04 UTC
I think you're looking at only the RHEL AppStream module.  The
Advanced Virt (AV) module has:
qemu-kvm-5.2.0-16.module+el8.4.0+12221+7bf85bbe.6

RHEL AV 8.5 version will be at least:
qemu-kvm-6.0.0-28.module+el8.5.0+12271+fffa967b

RHEL 8.6 will unify the module streams, and RHEL 9 gets rid of the
modules altogether.

(Having said that, I've no idea if CentOS 8 actually rebuilds
the AV module or how to enable it).

Comment 6 Richard W.M. Jones 2021-08-20 12:11:03 UTC
Apparently it is built, and tagged as "virt8-advanced-virtualization-release",
eg:
https://cbs.centos.org/koji/buildinfo?buildID=33079

Comment 7 Daniel Berrangé 2021-08-20 12:28:44 UTC
I think building this in EPEL8 is going to create trouble for users.

Even though the main binary in the RHEL package is /usr/libexec/qemu-kvm, other binaries are still going to clash - qemu-img, qemu-nbd, qemu-io. This will prevent parallel installation, though not a big deal. The more troublesome issue is that applications with RPM deps  "Requires: /usr/bin/qemu-img" are now going to have non-deterministic results in EPEL is enabled - possibly getting either the EPEL or RHEL versions of the RPM. This is a example of why the general EPEL policy is to not duplicate packages that are already shipped in RHEL.  People will enable EPEL for many reasons unrelated to QEMU, so this side effect is undesirable.

I appreciate that the RHEL version of QEMU is more cut down in terms of features than a full QEMU build.

I think that using COPR repos for custom versions of QEMU RPMs for RHEL/CentOS is likely to be a better approach.

Comment 8 Ben Kircher 2021-08-20 12:57:25 UTC
Daniel, 

> EPEL policy is to not duplicate packages that are already shipped in RHEL.  People will enable EPEL for many reasons unrelated to QEMU, so this side effect is undesirable.

That absolutely makes sense.


> I appreciate that the RHEL version of QEMU is more cut down in terms of features than a full QEMU build.

I totally see the reason for a more stripped down, slower moving, lot's of patches applied version of qemu-kvm package. It just does not fit our use case :-)


> I think that using COPR repos for custom versions of QEMU RPMs for RHEL/CentOS is likely to be a better approach.

True. And to be honest we're doing that. But I was hoping by bringing our packaging efforts back to CentOS, now with Stream, we would have a wider audience which we then would in return benefit from.

Comment 9 Ben Kircher 2021-08-20 13:06:14 UTC
Richard, 

> RHEL 8.6 will unify the module streams, and RHEL 9 gets rid of the
modules altogether.

Thanks for digging and taking the time. 

Is this for modules in general or for AV in particular? Are there any discussions regarding QEMU future packaging for RHEL 9 / cs9 I can follow?

Comment 10 Daniel Berrangé 2021-08-20 13:09:41 UTC

(In reply to Benjamin Kircher from comment #9)
> Is this for modules in general or for AV in particular? Are there any
> discussions regarding QEMU future packaging for RHEL 9 / cs9 I can follow?

I'm afraid there's not really any public discussion of the plans, but in general looking at c9s is a good way to see what's happening with latest RHEL QEMU packaging. Note currently c9s is targetting RHEL-9 Beta content. it is likely to rebase again to newer QEMU before 9.0 is formally released.(In reply to Benjamin Kircher from comment #8)

> > I appreciate that the RHEL version of QEMU is more cut down in terms of features than a full QEMU build.
> 
> I totally see the reason for a more stripped down, slower moving, lot's of
> patches applied version of qemu-kvm package. It just does not fit our use
> case :-)

If you look at the advanced virtualization stream, you'll see the QEMU versions are actually not that slow moving - they're rebasing to latest QEMU upstream in every release, so they're not very far behind latest.

Comment 11 Richard W.M. Jones 2021-08-20 13:23:17 UTC
AV only, there will still be modules in general in RHEL 9.

Comment 12 Davide Cavalca 2021-08-20 20:31:09 UTC
> EPEL policy is to not duplicate packages that are already shipped in RHEL.  People will enable EPEL for many reasons unrelated to QEMU, so this side effect is undesirable.

fwiw, my understanding is that this specific policy doesn't apply to QEMU, as the RHEL package is called qemu-kvm while the Fedora one is called qemu. As long as it is possible to build an EPEL package without any file conflicts, it should be ok policy-wise. Adding Carl and Troy in cc as this was discussed at the last EPEL meeting.

Comment 13 Troy Dawson 2021-08-20 20:58:42 UTC
Davide is correct.
As long as the package in EPEL doesn't contain any of the binaries that come from RHEL, it does not violate the EPEL policies.

If we compare the Fedora qemu versus the RHEL 9 qemu-kvm, there is a 72 package difference.
RHEL9 - qemu-kvm has 15 binaries built  (on x86_64)
Fedora - qemu has 87 binaries built     (on x86_64)

That is quite alot of functionality that having an EPEL qemu can give.

This was given the EPEL Steering Committee vote of approval, but I do have three things to note.

Note 1 - Beyond the package NAMES, you need to make sure that you don't step on any PROVIDES.
In other words, the EPEL qemu packages should not override the RHEL packages, even if they have different package names.

Note 2 - My opinion, the EPEL qemu package should keep in step with the version of the RHEL qemu-kvm.
This will require some coordination.  But since the RHEL qemu-kvm should come out in Stream in plenty of time, this shouldn't be too hard.

Note 3 - Keep doing it.  Don't let it lapse after 6 months or so.
This sorta goes with Note 2.  But qemu isn't a simple library.  You can't just build it once and let it sit for the life of the RHEL release.  If you, or your group, feel that you need to drop it, go through the orphaning phase and let people know.  See if there is another person or group to take it up.  And if there isn't, retire it.

Comment 14 Richard W.M. Jones 2021-08-21 07:34:34 UTC
(In reply to Troy Dawson from comment #13)
> Note 3 - Keep doing it.  Don't let it lapse after 6 months or so.
> This sorta goes with Note 2.  But qemu isn't a simple library.  You can't
> just build it once and let it sit for the life of the RHEL release.  If you,
> or your group, feel that you need to drop it, go through the orphaning phase
> and let people know.  See if there is another person or group to take it up.
> And if there isn't, retire it.

In particular the EPEL 7 branch of qemu hasn't had a CVE-related
update since 2015.  I would not recommend anyone use that version.

https://src.fedoraproject.org/rpms/qemu/commits/epel7

Comment 15 Istiak Ferdous 2022-01-25 19:15:49 UTC
Any chance there will be EPEL9 branch at all?

Comment 16 Daniel Berrangé 2022-09-08 17:02:58 UTC
(In reply to Troy Dawson from comment #13)
> Davide is correct.
> As long as the package in EPEL doesn't contain any of the binaries that come
> from RHEL, it does not violate the EPEL policies.
> 
> If we compare the Fedora qemu versus the RHEL 9 qemu-kvm, there is a 72
> package difference.
> RHEL9 - qemu-kvm has 15 binaries built  (on x86_64)
> Fedora - qemu has 87 binaries built     (on x86_64)
> 
> That is quite alot of functionality that having an EPEL qemu can give.
> 
> This was given the EPEL Steering Committee vote of approval, but I do have
> three things to note.
> 
> Note 1 - Beyond the package NAMES, you need to make sure that you don't step
> on any PROVIDES.
> In other words, the EPEL qemu packages should not override the RHEL
> packages, even if they have different package names.

Even this is going to have some non-trivial complications/implications. 

For example, the qemu-system-aarch64 emulator wants the edk2-aarch64 firmware. This is already distributed by RHEL, but restricted to only be shipped on the aarch64 host arch, but the EPEL build will need it on all arches.

If the EPEL package names are all different, then they will not satisfy the dependencies from any libvirt packages. 

IIUC EPEL supports ppc64, however, RHEL does not ship libvirt builds for ppc64 any more so there won't be any virt mgmt toolstack present. 

RHEL ships a 'qemu-img' binary, but with features restricted, there's no scope for parallel installing.

> Note 2 - My opinion, the EPEL qemu package should keep in step with the
> version of the RHEL qemu-kvm.
> This will require some coordination.  But since the RHEL qemu-kvm should
> come out in Stream in plenty of time, this shouldn't be too hard.

Keeping versions in sync is likely to be mandatory if the EPEL builds want to be able to use firmware builds shipped by RHEL, as we can't guarantee the firmware build options will be compatible with older or newer QEMUs than what is in RHEL.

Comment 17 Daniel Berrangé 2022-09-14 08:03:51 UTC
*** Bug 1917536 has been marked as a duplicate of this bug. ***